* sched: softlockups in multi_cpu_stop
@ 2015-03-02 7:45 Sasha Levin
[not found] ` <CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com>
2015-03-06 11:27 ` Sasha Levin
0 siblings, 2 replies; 44+ messages in thread
From: Sasha Levin @ 2015-03-02 7:45 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar; +Cc: LKML, Dave Jones
Hi all,
I'm seeing the following lockup pretty often while fuzzing with trinity:
[ 880.960250] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 447s! [migration/1:14]
[ 880.960700] Modules linked in:
[ 880.960700] irq event stamp: 380954
[ 880.960700] hardirqs last enabled at (380953): restore_args (arch/x86/kernel/entry_64.S:780)
[ 880.960700] hardirqs last disabled at (380954): apic_timer_interrupt (arch/x86/kernel/entry_64.S:920)
[ 880.960700] softirqs last enabled at (380952): __do_softirq (./arch/x86/include/asm/preempt.h:22 kernel/softirq.c:300)
[ 880.960700] softirqs last disabled at (380947): irq_exit (kernel/softirq.c:350 kernel/softirq.c:391)
[ 880.960700] CPU: 1 PID: 14 Comm: migration/1 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880052cc8000 ti: ffff880052cd0000 task.ti: ffff880052cd0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff880052cd7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: 0000000042506841 RCX: 1ffff1000a5992d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff880052cd7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9315e211
[ 880.960700] R13: ffff880052cd7ae8 R14: ffffffff93220213 R15: ffff880052cd7ad8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880053200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002d88fd8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8805d4f67728 0000000000000282
[ 880.960700] ffff880000000000 00ff880052cc8000 ffff880052cd7bd8 dffffc0000000000
[ 880.960700] ffff880053218440 ffff8805d4f67778 ffff8805d4f67718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] sending NMI to other CPUs:
[ 881.290044] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.336 msecs
[ 881.292380] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.083 msecs
[ 881.297603] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 7.423 msecs
[ 880.960700] NMI backtrace for cpu 0
[ 880.960700] CPU: 0 PID: 11 Comm: migration/0 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88000d6e3000 ti: ffff88000d6f0000 task.ti: ffff88000d6f0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff88000d6f7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0007c21ee5 RCX: 1ffff10001adc8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff88000d6f7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88003e10f738 R14: 0000000000000001 R15: ffff88003e10f718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880029000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002d31fb0 CR3: 000000002082c000 CR4: 00000000000007b0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88003e10f728 0000000000000282
[ 880.960700] ffff880000000000 00ff88000d6e3000 ffff88000d6f7bd8 dffffc0000000000
[ 880.960700] ffff880029018440 ffff88003e10f778 ffff88003e10f718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 2
[ 880.960700] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802ccca0000 ti: ffff88000dee8000 task.ti: ffff88000dee8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000deefd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000deefd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000deeffd8 R14: 0000000000000000 R15: ffff88000deeffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007f6baacd10f8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000deefd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000deeffd8 0000000000000000 ffff88000deefd78 ffffffff931629ef
[ 880.960700] ffff88000deefe68 ffffffff934018e0 ffff88000deefde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 3
[ 880.960700] CPU: 3 PID: 32 Comm: migration/3 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8800a6cc0000 ti: ffff8800a6cc8000 task.ti: ffff8800a6cc8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8800a6ccfb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0067e09ee5 RCX: 1ffff10014d982d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8800a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88033f04f738 R14: 0000000000000001 R15: ffff88033f04f718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8800a7200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000001490fe8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88033f04f728 0000000000000282
[ 880.960700] ffff880000000000 00ff8800a6cc0000 ffff8800a6ccfbd8 dffffc0000000000
[ 880.960700] ffff8800a7218440 ffff88033f04f778 ffff88033f04f718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 4
[ 880.960700] CPU: 4 PID: 41 Comm: migration/4 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8800cacb3000 ti: ffff8800cacc0000 task.ti: ffff8800cacc0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8800cacc7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed006e615ee5 RCX: 1ffff100195968d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8800cacc7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8803730af738 R14: 0000000000000001 R15: ffff8803730af718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8800cf200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 000000000189efd8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803730af728 0000000000000282
[ 880.960700] ffff880000000000 00ff8800cacb3000 ffff8800cacc7bd8 dffffc0000000000
[ 880.960700] ffff8800cf218440 ffff8803730af778 ffff8803730af718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 5
[ 880.960700] CPU: 5 PID: 50 Comm: migration/5 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880128c6b000 ti: ffff880128c88000 task.ti: ffff880128c88000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff880128c8fb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed008c0bbee5 RCX: 1ffff1002518d8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff880128c8fbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8804605df738 R14: 0000000000000001 R15: ffff8804605df718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880129000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002bf8ff0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8804605df728 0000000000000282
[ 880.960700] ffff880100000000 00ff880128c6b000 ffff880128c8fbd8 dffffc0000000000
[ 880.960700] ffff880129018440 ffff8804605df778 ffff8804605df718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 6
[ 880.960700] CPU: 6 PID: 59 Comm: migration/6 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880152cb3000 ti: ffff880152cc0000 task.ti: ffff880152cc0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff880152cc7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0063d8cee5 RCX: 1ffff1002a5968d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff880152cc7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88031ec67738 R14: 0000000000000001 R15: ffff88031ec67718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880153200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002b54fa0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88031ec67728 0000000000000282
[ 880.960700] ffff880100000000 00ff880152cb3000 ffff880152cc7bd8 dffffc0000000000
[ 880.960700] ffff880153218440 ffff88031ec67778 ffff88031ec67718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 7
[ 880.960700] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8808dd0b0000 ti: ffff88000e2d0000 task.ti: ffff88000e2d0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0000:ffff88000e2d7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000e2d7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000e2d7fd8 R14: 0000000000000000 R15: ffff88000e2d7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88017d200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000000000000 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000e2d7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000e2d7fd8 0000000000000000 ffff88000e2d7d78 ffffffff931629ef
[ 880.960700] ffff88000e2d7e68 ffffffff934018e0 0000000000000000 0000000000000000
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 8
[ 880.960700] CPU: 8 PID: 77 Comm: migration/8 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8801a6cc0000 ti: ffff8801a6cc8000 task.ti: ffff8801a6cc8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8801a6ccfb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed003abaaee5 RCX: 1ffff10034d982d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8801a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8801d5d57738 R14: 0000000000000001 R15: ffff8801d5d57718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8801a7200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000000e69fc8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8801d5d57728 0000000000000282
[ 880.960700] ffff880100000000 00ff8801a6cc0000 ffff8801a6ccfbd8 dffffc0000000000
[ 880.960700] ffff8801a7218440 ffff8801d5d57778 ffff8801d5d57718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 9
[ 880.960700] CPU: 9 PID: 86 Comm: migration/9 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8801d0c88000 ti: ffff8801d0c90000 task.ti: ffff8801d0c90000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8801d0c97b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed007245bee5 RCX: 1ffff1003a1912d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8801d0c97bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8803922df738 R14: 0000000000000001 R15: ffff8803922df718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8801d1000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00000000029a8fb0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803922df728 0000000000000282
[ 880.960700] ffff880100000000 00ff8801d0c88000 ffff8801d0c97bd8 dffffc0000000000
[ 880.960700] ffff8801d1018440 ffff8803922df778 ffff8803922df718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 10
[ 880.960700] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880128c68000 ti: ffff88000e6c0000 task.ti: ffff88000e6c0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000e6c7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000e6c7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000e6c7fd8 R14: 0000000000000000 R15: ffff88000e6c7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8801fb200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000000636668 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000e6c7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000e6c7fd8 0000000000000000 ffff88000e6c7d78 ffffffff931629ef
[ 880.960700] ffff88000e6c7e68 ffffffff934018e0 0000000000000000 0000000000000000
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 11
[ 880.960700] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8801a6ca8000 ti: ffff88000e6c8000 task.ti: ffff88000e6c8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000e6cfd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000e6cfd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000e6cffd8 R14: 0000000000000000 R15: ffff88000e6cffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880225200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00000000030db000 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000e6cfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000e6cffd8 0000000000000000 ffff88000e6cfd78 ffffffff931629ef
[ 880.960700] ffff88000e6cfe68 ffffffff934018e0 ffff88000e6cfde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 12
[ 880.960700] CPU: 12 PID: 113 Comm: migration/12 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88024ecd0000 ti: ffff88024ecd8000 task.ti: ffff88024ecd8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff88024ecdfb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0073a32ee5 RCX: 1ffff10049d9a2d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff88024ecdfbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88039d197738 R14: 0000000000000001 R15: ffff88039d197718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88024f200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002648fe0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039d197728 0000000000000282
[ 880.960700] ffff880200000000 00ff88024ecd0000 ffff88024ecdfbd8 dffffc0000000000
[ 880.960700] ffff88024f218440 ffff88039d197778 ffff88039d197718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 13
[ 880.960700] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802f6cc0000 ti: ffff88000e6d8000 task.ti: ffff88000e6d8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000e6dfd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000e6dfd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000e6dffd8 R14: 0000000000000000 R15: ffff88000e6dffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880279000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 000000000063a69c CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000e6dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000e6dffd8 0000000000000000 ffff88000e6dfd78 ffffffff931629ef
[ 880.960700] ffff88000e6dfe68 ffffffff934018e0 ffff88000e6dfde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 14
[ 880.960700] CPU: 14 PID: 131 Comm: migration/14 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802a2cdb000 ti: ffff8802a2ce8000 task.ti: ffff8802a2ce8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8802a2cefb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0059614ee5 RCX: 1ffff1005459b8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8802a2cefbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8802cb0a7738 R14: 0000000000000001 R15: ffff8802cb0a7718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8802a3200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002cc8ff8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802cb0a7728 0000000000000282
[ 880.960700] ffff880200000000 00ff8802a2cdb000 ffff8802a2cefbd8 dffffc0000000000
[ 880.960700] ffff8802a3218440 ffff8802cb0a7778 ffff8802cb0a7718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 15
[ 880.960700] CPU: 15 PID: 140 Comm: migration/15 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802ccca3000 ti: ffff8802cccd0000 task.ti: ffff8802cccd0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8802cccd7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0023834ee5 RCX: 1ffff100599948d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8802cccd7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88011c1a7738 R14: 0000000000000001 R15: ffff88011c1a7718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8802cd200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000001ad0a08 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88011c1a7728 0000000000000282
[ 880.960700] ffff880200000000 00ff8802ccca3000 ffff8802cccd7bd8 dffffc0000000000
[ 880.960700] ffff8802cd218440 ffff88011c1a7778 ffff88011c1a7718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 16
[ 880.960700] CPU: 16 PID: 149 Comm: migration/16 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802f6ce8000 ti: ffff8802f6cf0000 task.ti: ffff8802f6cf0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8802f6cf7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed00149d5ee5 RCX: 1ffff1005ed9d2d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8802f6cf7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8800a4eaf738 R14: 0000000000000001 R15: ffff8800a4eaf718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8802f7200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000001f6afe8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4eaf728 0000000000000282
[ 880.960700] ffff880200000000 00ff8802f6ce8000 ffff8802f6cf7bd8 dffffc0000000000
[ 880.960700] ffff8802f7218440 ffff8800a4eaf778 ffff8800a4eaf718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 17
[ 880.960700] CPU: 17 PID: 158 Comm: migration/17 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880320c7b000 ti: ffff880320c88000 task.ti: ffff880320c88000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff880320c8fb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed000b451ee5 RCX: 1ffff1006418f8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff880320c8fbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88005a28f738 R14: 0000000000000001 R15: ffff88005a28f718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880321000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 000000000161bff8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88005a28f728 0000000000000282
[ 880.960700] ffff880300000000 00ff880320c7b000 ffff880320c8fbd8 dffffc0000000000
[ 880.960700] ffff880321018440 ffff88005a28f778 ffff88005a28f718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 18
[ 880.960700] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8806ad0b0000 ti: ffff88000eac0000 task.ti: ffff88000eac0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eac7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eac7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eac7fd8 R14: 0000000000000000 R15: ffff88000eac7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88034b200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000001692ac0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eac7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eac7fd8 0000000000000000 ffff88000eac7d78 ffffffff931629ef
[ 880.960700] ffff88000eac7e68 ffffffff934018e0 ffff88000eac7de8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 19
[ 880.960700] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8807ed0c8000 ti: ffff88000eac8000 task.ti: ffff88000eac8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eacfd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eacfd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eacffd8 R14: 0000000000000000 R15: ffff88000eacffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880375200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007ffe1ae26d00 CR3: 000000000660a000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eacfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eacffd8 0000000000000000 ffff88000eacfd78 ffffffff931629ef
[ 880.960700] ffff88000eacfe68 ffffffff934018e0 ffff88000eacfde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 20
[ 880.960700] CPU: 20 PID: 185 Comm: migration/20 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88039ece0000 ti: ffff88039ece8000 task.ti: ffff88039ece8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff88039ecefb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed008ddccee5 RCX: 1ffff10073d9c2d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff88039ecefbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88046ee67738 R14: 0000000000000001 R15: ffff88046ee67718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88039f200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00000000029aeff8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ee67728 0000000000000282
[ 880.960700] ffff880300000000 00ff88039ece0000 ffff88039ecefbd8 dffffc0000000000
[ 880.960700] ffff88039f218440 ffff88046ee67778 ffff88046ee67718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 21
[ 880.960700] CPU: 21 PID: 194 Comm: migration/21 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8803c8ca8000 ti: ffff8803c8cb0000 task.ti: ffff8803c8cb0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8803c8cb7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed00149caee5 RCX: 1ffff100791952d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8803c8cb7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8800a4e57738 R14: 0000000000000001 R15: ffff8800a4e57718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8803c9000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002914fb0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4e57728 0000000000000282
[ 880.960700] ffff880300000000 00ff8803c8ca8000 ffff8803c8cb7bd8 dffffc0000000000
[ 880.960700] ffff8803c9018440 ffff8800a4e57778 ffff8800a4e57718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 22
[ 880.960700] CPU: 22 PID: 203 Comm: migration/22 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8803f2cdb000 ti: ffff8803f2ce8000 task.ti: ffff8803f2ce8000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8803f2cefb78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed008dde3ee5 RCX: 1ffff1007e59b8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8803f2cefbd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88046ef1f738 R14: 0000000000000001 R15: ffff88046ef1f718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8803f3200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000002e38b80 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ef1f728 0000000000000282
[ 880.960700] ffff880300000000 00ff8803f2cdb000 ffff8803f2cefbd8 dffffc0000000000
[ 880.960700] ffff8803f3218440 ffff88046ef1f778 ffff88046ef1f718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 23
[ 880.960700] CPU: 23 PID: 212 Comm: migration/23 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88041cc7b000 ti: ffff88041cce0000 task.ti: ffff88041cce0000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff88041cce7b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed006926eee5 RCX: 1ffff1008398f8d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff88041cce7bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff880349377738 R14: 0000000000000001 R15: ffff880349377718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88041d200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007f9c055599d0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff880349377728 0000000000000282
[ 880.960700] ffff880400000000 00ff88041cc7b000 ffff88041cce7bd8 dffffc0000000000
[ 880.960700] ffff88041d218440 ffff880349377778 ffff880349377718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 24
[ 880.960700] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8801d0c68000 ti: ffff88000eed0000 task.ti: ffff88000eed0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eed7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eed7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eed7fd8 R14: 0000000000000000 R15: ffff88000eed7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880447200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00000000029e4220 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eed7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eed7fd8 0000000000000000 ffff88000eed7d78 ffffffff931629ef
[ 880.960700] ffff88000eed7e68 ffffffff934018e0 ffff88000eed7de8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 25
[ 880.960700] CPU: 25 PID: 12438 Comm: trinity-c68 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8802cb318000 ti: ffff8802cb3a8000 task.ti: ffff8802cb3a8000
[ 880.960700] RIP: rwsem_down_write_failed (./arch/x86/include/asm/processor.h:658 ./arch/x86/include/asm/processor.h:663 kernel/locking/rwsem-xadd.c:413 kernel/locking/rwsem-xadd.c:442)
[ 880.960700] RSP: 0018:ffff8802cb3afc08 EFLAGS: 00000286
[ 880.960700] RAX: ffffffff00000001 RBX: ffffffffa18e12a8 RCX: ffff8802cb318000
[ 880.960700] RDX: fffffffe00000001 RSI: fffffffeffffffff RDI: 0000000100000000
[ 880.960700] RBP: ffff8802cb3afd98 R08: 0000000000000000 R09: ffffffff00000001
[ 880.960700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803f10c3000
[ 880.960700] R13: ffffffffa18e12a0 R14: ffff8802cb3afd68 R15: ffffffffa18e1250
[ 880.960700] FS: 00007f9c05559700(0000) GS:ffff880471000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 880.960700] CR2: 00007f9c0549b614 CR3: 00000002cb385000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffff9dbf3ced ffff8802cb318d00 0000000000000000 ffff8802cb318000
[ 880.960700] 00000000000004ff ffffffffa3ebf6b0 1ffff10059675f89 1ffff10059675f8d
[ 880.960700] 0000000041b58ab3 ffff8802cb3a8010 1ffff10059675002 ffff8802cb3a8000
[ 880.960700] Call Trace:
[ 880.960700] ? rwsem_down_write_failed (include/linux/rcupdate.h:912 kernel/locking/rwsem-xadd.c:306 kernel/locking/rwsem-xadd.c:381 kernel/locking/rwsem-xadd.c:442)
[ 880.960700] ? rwsem_down_read_failed (kernel/locking/rwsem-xadd.c:433)
[ 880.960700] ? trace_hardirqs_off (kernel/locking/lockdep.c:2647)
[ 880.960700] ? __acct_update_integrals (kernel/tsacct.c:125)
[ 880.960700] ? taskstats_exit (kernel/tsacct.c:125)
[ 880.960700] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:104)
[ 880.960700] ? down_write (kernel/locking/rwsem.h:4 kernel/locking/rwsem.c:52)
[ 880.960700] ? ipcget (ipc/util.c:349 ipc/util.c:646)
[ 880.960700] ipcget (ipc/util.c:349 ipc/util.c:646)
[ 880.960700] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
[ 880.960700] ? syscall_trace_enter_phase1 (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1486)
[ 880.960700] SyS_semget (ipc/sem.c:591 ipc/sem.c:572)
[ 880.960700] ? sem_exit_ns (ipc/sem.c:572)
[ 880.960700] ? trace_hardirqs_on_thunk (arch/x86/lib/thunk_64.S:42)
[ 880.960700] tracesys_phase2 (arch/x86/kernel/entry_64.S:347)
[ 880.960700] Code: 8d 0c 0a 48 89 d0 f0 49 0f b1 0f 48 39 c2 0f 84 2b 04 00 00 48 89 c2 48 8d 04 3a 48 85 f0 74 df 4d 85 e4 0f 84 50 04 00 00 f3 90 <e9> 57 ff ff ff e8 02 62 87 f5 84 c0 0f 85 f5 fd ff ff 48 c7 c2
All code
========
0: 8d 0c 0a lea (%rdx,%rcx,1),%ecx
3: 48 89 d0 mov %rdx,%rax
6: f0 49 0f b1 0f lock cmpxchg %rcx,(%r15)
b: 48 39 c2 cmp %rax,%rdx
e: 0f 84 2b 04 00 00 je 0x43f
14: 48 89 c2 mov %rax,%rdx
17: 48 8d 04 3a lea (%rdx,%rdi,1),%rax
1b: 48 85 f0 test %rsi,%rax
1e: 74 df je 0xffffffffffffffff
20: 4d 85 e4 test %r12,%r12
23: 0f 84 50 04 00 00 je 0x479
29: f3 90 pause
2b:* e9 57 ff ff ff jmpq 0xffffffffffffff87 <-- trapping instruction
30: e8 02 62 87 f5 callq 0xfffffffff5876237
35: 84 c0 test %al,%al
37: 0f 85 f5 fd ff ff jne 0xfffffffffffffe32
3d: 48 rex.W
3e: c7 .byte 0xc7
3f: c2 .byte 0xc2
...
Code starting with the faulting instruction
===========================================
0: e9 57 ff ff ff jmpq 0xffffffffffffff5c
5: e8 02 62 87 f5 callq 0xfffffffff587620c
a: 84 c0 test %al,%al
c: 0f 85 f5 fd ff ff jne 0xfffffffffffffe07
12: 48 rex.W
13: c7 .byte 0xc7
14: c2 .byte 0xc2
...
[ 880.960700] NMI backtrace for cpu 26
[ 880.960700] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8803c8c78000 ti: ffff88000eee0000 task.ti: ffff88000eee0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eee7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eee7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eee7fd8 R14: 0000000000000000 R15: ffff88000eee7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88049b200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007f0f2bc1d5a8 CR3: 000000000678f000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eee7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eee7fd8 0000000000000000 ffff88000eee7d78 ffffffff931629ef
[ 880.960700] ffff88000eee7e68 ffffffff934018e0 ffff88000eee7de8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 27
[ 880.960700] CPU: 27 PID: 248 Comm: migration/27 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff8804c4cf8000 ti: ffff8804c3b00000 task.ti: ffff8804c3b00000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff8804c3b07b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed0072423ee5 RCX: 1ffff1009899f2d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff8804c3b07bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff88039211f738 R14: 0000000000000001 R15: ffff88039211f718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8804c5200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00000000029c4ff8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039211f728 0000000000000282
[ 880.960700] ffff880400000000 00ff8804c4cf8000 ffff8804c3b07bd8 dffffc0000000000
[ 880.960700] ffff8804c5218440 ffff88039211f778 ffff88039211f718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 28
[ 880.960700] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88065d0c0000 ti: ffff88000eef0000 task.ti: ffff88000eef0000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eef7d38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eef7d38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eef7fd8 R14: 0000000000000000 R15: ffff88000eef7fd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff8804ef200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 0000000000b34fb8 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eef7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eef7fd8 0000000000000000 ffff88000eef7d78 ffffffff931629ef
[ 880.960700] ffff88000eef7e68 ffffffff934018e0 0000000000000000 0000000000000000
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 29
[ 880.960700] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88079d0b0000 ti: ffff88000eef8000 task.ti: ffff88000eef8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0018:ffff88000eeffd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000eeffd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000eefffd8 R14: 0000000000000000 R15: ffff88000eefffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880519000000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007f0f2b8e5210 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000eeffd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000eefffd8 0000000000000000 ffff88000eeffd78 ffffffff931629ef
[ 880.960700] ffff88000eeffe68 ffffffff934018e0 ffff88000eeffde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
[ 880.960700] NMI backtrace for cpu 30
[ 880.960700] CPU: 30 PID: 269 Comm: migration/30 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff880542cf8000 ti: ffff880541b00000 task.ti: ffff880541b00000
[ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
[ 880.960700] RSP: 0000:ffff880541b07b78 EFLAGS: 00000246
[ 880.960700] RAX: 0000000000000000 RBX: ffffed005ea54ee5 RCX: 1ffff100a859f2d5
[ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
[ 880.960700] RBP: ffff880541b07bd8 R08: 0000000000000001 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
[ 880.960700] R13: ffff8802f52a7738 R14: 0000000000000001 R15: ffff8802f52a7718
[ 880.960700] FS: 0000000000000000(0000) GS:ffff880543200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007f42552112c0 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802f52a7728 0000000000000282
[ 880.960700] ffff880500000000 00ff880542cf8000 ffff880541b07bd8 dffffc0000000000
[ 880.960700] ffff880543218440 ffff8802f52a7778 ffff8802f52a7718 ffffffff935335b0
[ 880.960700] Call Trace:
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
[ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
[ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
[ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
[ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] ? sort_range (kernel/smpboot.c:105)
[ 880.960700] kthread (kernel/kthread.c:207)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
[ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
[ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
All code
========
0: eb 27 jmp 0x29
2: 0f 1f 00 nopl (%rax)
5: 41 83 fe 03 cmp $0x3,%r14d
9: 75 0a jne 0x15
b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
f: 0f 85 50 01 00 00 jne 0x165
15: f0 41 ff 4f 24 lock decl 0x24(%r15)
1a: 74 71 je 0x8d
1c: 41 83 fe 04 cmp $0x4,%r14d
20: 0f 84 d7 00 00 00 je 0xfd
26: 45 89 f0 mov %r14d,%r8d
29: f3 90 pause
2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
2e: 48 c1 e8 03 shr $0x3,%rax
32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
37: 84 c0 test %al,%al
39: 74 08 je 0x43
3b: 3c 03 cmp $0x3,%al
3d: 0f .byte 0xf
3e: 8e f5 mov %ebp,%?
...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax
3: 48 c1 e8 03 shr $0x3,%rax
7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
c: 84 c0 test %al,%al
e: 74 08 je 0x18
10: 3c 03 cmp $0x3,%al
12: 0f .byte 0xf
13: 8e f5 mov %ebp,%?
...
[ 880.960700] NMI backtrace for cpu 31
[ 880.960700] CPU: 31 PID: 0 Comm: swapper/31 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
[ 880.960700] task: ffff88000f2d0000 ti: ffff88000f2d8000 task.ti: ffff88000f2d8000
[ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
[ 880.960700] RSP: 0000:ffff88000f2dfd38 EFLAGS: 00000282
[ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
[ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
[ 880.960700] RBP: ffff88000f2dfd38 R08: 0000000000000000 R09: 0000000000000000
[ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
[ 880.960700] R13: ffff88000f2dffd8 R14: 0000000000000000 R15: ffff88000f2dffd8
[ 880.960700] FS: 0000000000000000(0000) GS:ffff88056d200000(0000) knlGS:0000000000000000
[ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 880.960700] CR2: 00007fff144e2980 CR3: 000000002082c000 CR4: 00000000000007a0
[ 880.960700] Stack:
[ 880.960700] ffff88000f2dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
[ 880.960700] ffff88000f2dffd8 0000000000000000 ffff88000f2dfd78 ffffffff931629ef
[ 880.960700] ffff88000f2dfe68 ffffffff934018e0 ffff88000f2dfde8 ffffffff9dbf7ec5
[ 880.960700] Call Trace:
[ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
[ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
[ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
[ 880.960700] ? arch_cpu_idle_prepare (??:?)
[ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
[ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
[ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
[ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
[ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
All code
========
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 00 55 48 add %dl,0x48(%rbp)
7: 89 e5 mov %esp,%ebp
9: fa cli
a: 5d pop %rbp
b: c3 retq
c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
13: 00 00
15: 55 push %rbp
16: 48 89 e5 mov %rsp,%rbp
19: fb sti
1a: 5d pop %rbp
1b: c3 retq
1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
23: 00 00
25: 55 push %rbp
26: 48 89 e5 mov %rsp,%rbp
29: fb sti
2a: f4 hlt
2b:* 5d pop %rbp <-- trapping instruction
2c: c3 retq
2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
34: 00
35: 55 push %rbp
36: 48 89 e5 mov %rsp,%rbp
39: f4 hlt
3a: 5d pop %rbp
3b: c3 retq
3c: 66 data16
3d: 0f .byte 0xf
3e: 1f (bad)
3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp
b: 48 89 e5 mov %rsp,%rbp
e: f4 hlt
f: 5d pop %rbp
10: c3 retq
11: 66 data16
12: 0f .byte 0xf
13: 1f (bad)
14: 84 00 test %al,(%rax)
Thanks,
Sasha
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
[not found] ` <CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com>
@ 2015-03-04 5:44 ` Rafael David Tinoco
0 siblings, 0 replies; 44+ messages in thread
From: Rafael David Tinoco @ 2015-03-04 5:44 UTC (permalink / raw)
To: Sasha Levin; +Cc: Peter Zijlstra, Ingo Molnar, LKML, Dave Jones
Some more info:
multi_cpu_stop seems to be spinning inside do { ... } while (curstate
!= MULTI_STOP_EXIT);
So, multi_cpu_stop is an offload ([migration]) for: migrate_swap ->
stop_two_cpus -> wait_for_completion() sequence... for cross-migrating
2 tasks.
Based on task structs from callers stacks:
PID 14990 CPU 05 -> PID 14996 CPU 00
PID 14991 CPU 30 -> PID 14998 CPU 01 (30 -> 1, different curr, same )
PID 14992 CPU 30 -> PID 14998 CPU 01 (30 -> 1, different curr)
PID 14996 CPU 00 -> PID 14992 CPU 30
PID 14998 CPU 01 -> PID 14990 CPU 05
RUNNING migration threads (cpu_stopper_thread -> multi_cpu_stop):
PID 118 = LAST CPU 09, CPU 09
PID 102 = LAST CPU 06, CPU 06
PID 143 = LAST CPU 14, CPU 14
PID 148 = LAST CPU 15, CPU 15
PID 153 = LAST CPU 16, CPU 16
### backtraces and task structs from stack
PID: 14990 TASK: ffff883e59e717f0 CPU: 5 COMMAND: "beam.smp"
#5 [ffff883e607edbc8] migrate_swap at ffffffff810987fa
ffff883e607edbd0: ffff883e59e717f0 ffff883e59e82fe0
ffff883e607edbe0: 000000100000000f ffff883e607edcc0
ffff883e607edbf0: ffffffff810a0827
ffff883e59e717f0 = task_struct -> pid 14990 (last cpu = 5)
ffff883e59e82fe0 = task_struct -> pid 14996 (last cpu = 0)
PID: 14991 TASK: ffff883e59e72fe0 CPU: 30 COMMAND: "beam.smp"
#5 [ffff883e59e0bbc8] migrate_swap at ffffffff810987fa
ffff883e59e0bbd0: ffff883e59e72fe0 ffff883e59e85fc0
ffff883e59e0bbe0: 000000060000000e ffff883e59e0bcc0
ffff883e59e0bbf0: ffffffff810a0827
ffff883e59e72fe0 = task_struct -> pid 14991 (last cpu = 30)
ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1)
PID: 14992 TASK: ffff883e59e747d0 CPU: 30 COMMAND: "beam.smp"
#5 [ffff883e59cadbc8] migrate_swap at ffffffff810987fa
ffff883e59cadbd0: ffff883e59e747d0 ffff883e59e85fc0
ffff883e59cadbe0: 0000000600000009 ffff883e59cadcc0
ffff883e59cadbf0: ffffffff810a0827
ffff883e59e747d0 = task_struct -> pid 14992 (last cpu = 30)
ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1)
PID: 14996 TASK: ffff883e59e82fe0 CPU: 0 COMMAND: "beam.smp"
#5 [ffff883f55d01bc8] migrate_swap at ffffffff810987fa
ffff883f55d01bd0: ffff883e59e82fe0 ffff883e59e747d0
ffff883f55d01be0: 0000000900000010 ffff883f55d01cc0
ffff883f55d01bf0: ffffffff810a0827
ffff883e59e82fe0 = task_struct -> pid 14996 (last cpu = 0)
ffff883e59e747d0 = task_struct -> pid 14992 (last cpu = 30)
PID: 14998 TASK: ffff883e59e85fc0 CPU: 1 COMMAND: "beam.smp"
#5 [ffff883e59e05bc8] migrate_swap at ffffffff810987fa
ffff883e59e05bd0: ffff883e59e85fc0 ffff883e59e717f0
ffff883e59e05be0: 0000000f00000006 ffff883e59e05cc0
ffff883e59e05bf0: ffffffff810a0827
ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1)
ffff883e59e717f0 = task_struct -> pid 14990 (last cpu = 5)
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-02 7:45 sched: softlockups in multi_cpu_stop Sasha Levin
[not found] ` <CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com>
@ 2015-03-06 11:27 ` Sasha Levin
2015-03-06 12:32 ` Ingo Molnar
1 sibling, 1 reply; 44+ messages in thread
From: Sasha Levin @ 2015-03-06 11:27 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar; +Cc: LKML, Dave Jones, Davidlohr Bueso, jason.low2
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
Thanks,
Sasha
On 03/02/2015 02:45 AM, Sasha Levin wrote:
> Hi all,
>
> I'm seeing the following lockup pretty often while fuzzing with trinity:
>
> [ 880.960250] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 447s! [migration/1:14]
> [ 880.960700] Modules linked in:
> [ 880.960700] irq event stamp: 380954
> [ 880.960700] hardirqs last enabled at (380953): restore_args (arch/x86/kernel/entry_64.S:780)
> [ 880.960700] hardirqs last disabled at (380954): apic_timer_interrupt (arch/x86/kernel/entry_64.S:920)
> [ 880.960700] softirqs last enabled at (380952): __do_softirq (./arch/x86/include/asm/preempt.h:22 kernel/softirq.c:300)
> [ 880.960700] softirqs last disabled at (380947): irq_exit (kernel/softirq.c:350 kernel/softirq.c:391)
> [ 880.960700] CPU: 1 PID: 14 Comm: migration/1 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880052cc8000 ti: ffff880052cd0000 task.ti: ffff880052cd0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880052cd7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: 0000000042506841 RCX: 1ffff1000a5992d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880052cd7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9315e211
> [ 880.960700] R13: ffff880052cd7ae8 R14: ffffffff93220213 R15: ffff880052cd7ad8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880053200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002d88fd8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8805d4f67728 0000000000000282
> [ 880.960700] ffff880000000000 00ff880052cc8000 ffff880052cd7bd8 dffffc0000000000
> [ 880.960700] ffff880053218440 ffff8805d4f67778 ffff8805d4f67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] sending NMI to other CPUs:
> [ 881.290044] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.336 msecs
> [ 881.292380] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.083 msecs
> [ 881.297603] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 7.423 msecs
> [ 880.960700] NMI backtrace for cpu 0
> [ 880.960700] CPU: 0 PID: 11 Comm: migration/0 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88000d6e3000 ti: ffff88000d6f0000 task.ti: ffff88000d6f0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88000d6f7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0007c21ee5 RCX: 1ffff10001adc8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88000d6f7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88003e10f738 R14: 0000000000000001 R15: ffff88003e10f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880029000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002d31fb0 CR3: 000000002082c000 CR4: 00000000000007b0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88003e10f728 0000000000000282
> [ 880.960700] ffff880000000000 00ff88000d6e3000 ffff88000d6f7bd8 dffffc0000000000
> [ 880.960700] ffff880029018440 ffff88003e10f778 ffff88003e10f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 2
> [ 880.960700] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802ccca0000 ti: ffff88000dee8000 task.ti: ffff88000dee8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000deefd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000deefd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000deeffd8 R14: 0000000000000000 R15: ffff88000deeffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f6baacd10f8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000deefd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000deeffd8 0000000000000000 ffff88000deefd78 ffffffff931629ef
> [ 880.960700] ffff88000deefe68 ffffffff934018e0 ffff88000deefde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 3
> [ 880.960700] CPU: 3 PID: 32 Comm: migration/3 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8800a6cc0000 ti: ffff8800a6cc8000 task.ti: ffff8800a6cc8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8800a6ccfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0067e09ee5 RCX: 1ffff10014d982d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8800a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88033f04f738 R14: 0000000000000001 R15: ffff88033f04f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800a7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001490fe8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88033f04f728 0000000000000282
> [ 880.960700] ffff880000000000 00ff8800a6cc0000 ffff8800a6ccfbd8 dffffc0000000000
> [ 880.960700] ffff8800a7218440 ffff88033f04f778 ffff88033f04f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 4
> [ 880.960700] CPU: 4 PID: 41 Comm: migration/4 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8800cacb3000 ti: ffff8800cacc0000 task.ti: ffff8800cacc0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8800cacc7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed006e615ee5 RCX: 1ffff100195968d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8800cacc7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8803730af738 R14: 0000000000000001 R15: ffff8803730af718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800cf200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000189efd8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803730af728 0000000000000282
> [ 880.960700] ffff880000000000 00ff8800cacb3000 ffff8800cacc7bd8 dffffc0000000000
> [ 880.960700] ffff8800cf218440 ffff8803730af778 ffff8803730af718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 5
> [ 880.960700] CPU: 5 PID: 50 Comm: migration/5 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880128c6b000 ti: ffff880128c88000 task.ti: ffff880128c88000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880128c8fb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008c0bbee5 RCX: 1ffff1002518d8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880128c8fbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8804605df738 R14: 0000000000000001 R15: ffff8804605df718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880129000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002bf8ff0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8804605df728 0000000000000282
> [ 880.960700] ffff880100000000 00ff880128c6b000 ffff880128c8fbd8 dffffc0000000000
> [ 880.960700] ffff880129018440 ffff8804605df778 ffff8804605df718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 6
> [ 880.960700] CPU: 6 PID: 59 Comm: migration/6 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880152cb3000 ti: ffff880152cc0000 task.ti: ffff880152cc0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880152cc7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0063d8cee5 RCX: 1ffff1002a5968d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880152cc7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88031ec67738 R14: 0000000000000001 R15: ffff88031ec67718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880153200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002b54fa0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88031ec67728 0000000000000282
> [ 880.960700] ffff880100000000 00ff880152cb3000 ffff880152cc7bd8 dffffc0000000000
> [ 880.960700] ffff880153218440 ffff88031ec67778 ffff88031ec67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 7
> [ 880.960700] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8808dd0b0000 ti: ffff88000e2d0000 task.ti: ffff88000e2d0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0000:ffff88000e2d7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e2d7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e2d7fd8 R14: 0000000000000000 R15: ffff88000e2d7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88017d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000000000 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e2d7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e2d7fd8 0000000000000000 ffff88000e2d7d78 ffffffff931629ef
> [ 880.960700] ffff88000e2d7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 8
> [ 880.960700] CPU: 8 PID: 77 Comm: migration/8 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801a6cc0000 ti: ffff8801a6cc8000 task.ti: ffff8801a6cc8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8801a6ccfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed003abaaee5 RCX: 1ffff10034d982d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8801a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8801d5d57738 R14: 0000000000000001 R15: ffff8801d5d57718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801a7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000e69fc8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8801d5d57728 0000000000000282
> [ 880.960700] ffff880100000000 00ff8801a6cc0000 ffff8801a6ccfbd8 dffffc0000000000
> [ 880.960700] ffff8801a7218440 ffff8801d5d57778 ffff8801d5d57718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 9
> [ 880.960700] CPU: 9 PID: 86 Comm: migration/9 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801d0c88000 ti: ffff8801d0c90000 task.ti: ffff8801d0c90000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8801d0c97b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed007245bee5 RCX: 1ffff1003a1912d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8801d0c97bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8803922df738 R14: 0000000000000001 R15: ffff8803922df718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801d1000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029a8fb0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803922df728 0000000000000282
> [ 880.960700] ffff880100000000 00ff8801d0c88000 ffff8801d0c97bd8 dffffc0000000000
> [ 880.960700] ffff8801d1018440 ffff8803922df778 ffff8803922df718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 10
> [ 880.960700] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880128c68000 ti: ffff88000e6c0000 task.ti: ffff88000e6c0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6c7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6c7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6c7fd8 R14: 0000000000000000 R15: ffff88000e6c7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801fb200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000636668 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6c7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6c7fd8 0000000000000000 ffff88000e6c7d78 ffffffff931629ef
> [ 880.960700] ffff88000e6c7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 11
> [ 880.960700] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801a6ca8000 ti: ffff88000e6c8000 task.ti: ffff88000e6c8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6cfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6cfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6cffd8 R14: 0000000000000000 R15: ffff88000e6cffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880225200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000030db000 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6cfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6cffd8 0000000000000000 ffff88000e6cfd78 ffffffff931629ef
> [ 880.960700] ffff88000e6cfe68 ffffffff934018e0 ffff88000e6cfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 12
> [ 880.960700] CPU: 12 PID: 113 Comm: migration/12 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88024ecd0000 ti: ffff88024ecd8000 task.ti: ffff88024ecd8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88024ecdfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0073a32ee5 RCX: 1ffff10049d9a2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88024ecdfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88039d197738 R14: 0000000000000001 R15: ffff88039d197718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88024f200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002648fe0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039d197728 0000000000000282
> [ 880.960700] ffff880200000000 00ff88024ecd0000 ffff88024ecdfbd8 dffffc0000000000
> [ 880.960700] ffff88024f218440 ffff88039d197778 ffff88039d197718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 13
> [ 880.960700] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802f6cc0000 ti: ffff88000e6d8000 task.ti: ffff88000e6d8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6dfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6dfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6dffd8 R14: 0000000000000000 R15: ffff88000e6dffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880279000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000063a69c CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6dffd8 0000000000000000 ffff88000e6dfd78 ffffffff931629ef
> [ 880.960700] ffff88000e6dfe68 ffffffff934018e0 ffff88000e6dfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 14
> [ 880.960700] CPU: 14 PID: 131 Comm: migration/14 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802a2cdb000 ti: ffff8802a2ce8000 task.ti: ffff8802a2ce8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802a2cefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0059614ee5 RCX: 1ffff1005459b8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802a2cefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8802cb0a7738 R14: 0000000000000001 R15: ffff8802cb0a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802a3200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002cc8ff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802cb0a7728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802a2cdb000 ffff8802a2cefbd8 dffffc0000000000
> [ 880.960700] ffff8802a3218440 ffff8802cb0a7778 ffff8802cb0a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 15
> [ 880.960700] CPU: 15 PID: 140 Comm: migration/15 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802ccca3000 ti: ffff8802cccd0000 task.ti: ffff8802cccd0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802cccd7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0023834ee5 RCX: 1ffff100599948d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802cccd7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88011c1a7738 R14: 0000000000000001 R15: ffff88011c1a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802cd200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001ad0a08 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88011c1a7728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802ccca3000 ffff8802cccd7bd8 dffffc0000000000
> [ 880.960700] ffff8802cd218440 ffff88011c1a7778 ffff88011c1a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 16
> [ 880.960700] CPU: 16 PID: 149 Comm: migration/16 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802f6ce8000 ti: ffff8802f6cf0000 task.ti: ffff8802f6cf0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802f6cf7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149d5ee5 RCX: 1ffff1005ed9d2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802f6cf7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8800a4eaf738 R14: 0000000000000001 R15: ffff8800a4eaf718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802f7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001f6afe8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4eaf728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802f6ce8000 ffff8802f6cf7bd8 dffffc0000000000
> [ 880.960700] ffff8802f7218440 ffff8800a4eaf778 ffff8800a4eaf718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 17
> [ 880.960700] CPU: 17 PID: 158 Comm: migration/17 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880320c7b000 ti: ffff880320c88000 task.ti: ffff880320c88000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880320c8fb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed000b451ee5 RCX: 1ffff1006418f8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880320c8fbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88005a28f738 R14: 0000000000000001 R15: ffff88005a28f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880321000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000161bff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88005a28f728 0000000000000282
> [ 880.960700] ffff880300000000 00ff880320c7b000 ffff880320c8fbd8 dffffc0000000000
> [ 880.960700] ffff880321018440 ffff88005a28f778 ffff88005a28f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 18
> [ 880.960700] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8806ad0b0000 ti: ffff88000eac0000 task.ti: ffff88000eac0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eac7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eac7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eac7fd8 R14: 0000000000000000 R15: ffff88000eac7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88034b200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001692ac0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eac7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eac7fd8 0000000000000000 ffff88000eac7d78 ffffffff931629ef
> [ 880.960700] ffff88000eac7e68 ffffffff934018e0 ffff88000eac7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 19
> [ 880.960700] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8807ed0c8000 ti: ffff88000eac8000 task.ti: ffff88000eac8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eacfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eacfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eacffd8 R14: 0000000000000000 R15: ffff88000eacffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880375200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007ffe1ae26d00 CR3: 000000000660a000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eacfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eacffd8 0000000000000000 ffff88000eacfd78 ffffffff931629ef
> [ 880.960700] ffff88000eacfe68 ffffffff934018e0 ffff88000eacfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 20
> [ 880.960700] CPU: 20 PID: 185 Comm: migration/20 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88039ece0000 ti: ffff88039ece8000 task.ti: ffff88039ece8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88039ecefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008ddccee5 RCX: 1ffff10073d9c2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88039ecefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88046ee67738 R14: 0000000000000001 R15: ffff88046ee67718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88039f200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029aeff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ee67728 0000000000000282
> [ 880.960700] ffff880300000000 00ff88039ece0000 ffff88039ecefbd8 dffffc0000000000
> [ 880.960700] ffff88039f218440 ffff88046ee67778 ffff88046ee67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 21
> [ 880.960700] CPU: 21 PID: 194 Comm: migration/21 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803c8ca8000 ti: ffff8803c8cb0000 task.ti: ffff8803c8cb0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8803c8cb7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149caee5 RCX: 1ffff100791952d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8803c8cb7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8800a4e57738 R14: 0000000000000001 R15: ffff8800a4e57718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803c9000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002914fb0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4e57728 0000000000000282
> [ 880.960700] ffff880300000000 00ff8803c8ca8000 ffff8803c8cb7bd8 dffffc0000000000
> [ 880.960700] ffff8803c9018440 ffff8800a4e57778 ffff8800a4e57718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 22
> [ 880.960700] CPU: 22 PID: 203 Comm: migration/22 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803f2cdb000 ti: ffff8803f2ce8000 task.ti: ffff8803f2ce8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8803f2cefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008dde3ee5 RCX: 1ffff1007e59b8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8803f2cefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88046ef1f738 R14: 0000000000000001 R15: ffff88046ef1f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803f3200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002e38b80 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ef1f728 0000000000000282
> [ 880.960700] ffff880300000000 00ff8803f2cdb000 ffff8803f2cefbd8 dffffc0000000000
> [ 880.960700] ffff8803f3218440 ffff88046ef1f778 ffff88046ef1f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 23
> [ 880.960700] CPU: 23 PID: 212 Comm: migration/23 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88041cc7b000 ti: ffff88041cce0000 task.ti: ffff88041cce0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88041cce7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed006926eee5 RCX: 1ffff1008398f8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88041cce7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff880349377738 R14: 0000000000000001 R15: ffff880349377718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88041d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f9c055599d0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff880349377728 0000000000000282
> [ 880.960700] ffff880400000000 00ff88041cc7b000 ffff88041cce7bd8 dffffc0000000000
> [ 880.960700] ffff88041d218440 ffff880349377778 ffff880349377718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 24
> [ 880.960700] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801d0c68000 ti: ffff88000eed0000 task.ti: ffff88000eed0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eed7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eed7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eed7fd8 R14: 0000000000000000 R15: ffff88000eed7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880447200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029e4220 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eed7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eed7fd8 0000000000000000 ffff88000eed7d78 ffffffff931629ef
> [ 880.960700] ffff88000eed7e68 ffffffff934018e0 ffff88000eed7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 25
> [ 880.960700] CPU: 25 PID: 12438 Comm: trinity-c68 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802cb318000 ti: ffff8802cb3a8000 task.ti: ffff8802cb3a8000
> [ 880.960700] RIP: rwsem_down_write_failed (./arch/x86/include/asm/processor.h:658 ./arch/x86/include/asm/processor.h:663 kernel/locking/rwsem-xadd.c:413 kernel/locking/rwsem-xadd.c:442)
> [ 880.960700] RSP: 0018:ffff8802cb3afc08 EFLAGS: 00000286
> [ 880.960700] RAX: ffffffff00000001 RBX: ffffffffa18e12a8 RCX: ffff8802cb318000
> [ 880.960700] RDX: fffffffe00000001 RSI: fffffffeffffffff RDI: 0000000100000000
> [ 880.960700] RBP: ffff8802cb3afd98 R08: 0000000000000000 R09: ffffffff00000001
> [ 880.960700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803f10c3000
> [ 880.960700] R13: ffffffffa18e12a0 R14: ffff8802cb3afd68 R15: ffffffffa18e1250
> [ 880.960700] FS: 00007f9c05559700(0000) GS:ffff880471000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 880.960700] CR2: 00007f9c0549b614 CR3: 00000002cb385000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffff9dbf3ced ffff8802cb318d00 0000000000000000 ffff8802cb318000
> [ 880.960700] 00000000000004ff ffffffffa3ebf6b0 1ffff10059675f89 1ffff10059675f8d
> [ 880.960700] 0000000041b58ab3 ffff8802cb3a8010 1ffff10059675002 ffff8802cb3a8000
> [ 880.960700] Call Trace:
> [ 880.960700] ? rwsem_down_write_failed (include/linux/rcupdate.h:912 kernel/locking/rwsem-xadd.c:306 kernel/locking/rwsem-xadd.c:381 kernel/locking/rwsem-xadd.c:442)
> [ 880.960700] ? rwsem_down_read_failed (kernel/locking/rwsem-xadd.c:433)
> [ 880.960700] ? trace_hardirqs_off (kernel/locking/lockdep.c:2647)
> [ 880.960700] ? __acct_update_integrals (kernel/tsacct.c:125)
> [ 880.960700] ? taskstats_exit (kernel/tsacct.c:125)
> [ 880.960700] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:104)
> [ 880.960700] ? down_write (kernel/locking/rwsem.h:4 kernel/locking/rwsem.c:52)
> [ 880.960700] ? ipcget (ipc/util.c:349 ipc/util.c:646)
> [ 880.960700] ipcget (ipc/util.c:349 ipc/util.c:646)
> [ 880.960700] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
> [ 880.960700] ? syscall_trace_enter_phase1 (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1486)
> [ 880.960700] SyS_semget (ipc/sem.c:591 ipc/sem.c:572)
> [ 880.960700] ? sem_exit_ns (ipc/sem.c:572)
> [ 880.960700] ? trace_hardirqs_on_thunk (arch/x86/lib/thunk_64.S:42)
> [ 880.960700] tracesys_phase2 (arch/x86/kernel/entry_64.S:347)
> [ 880.960700] Code: 8d 0c 0a 48 89 d0 f0 49 0f b1 0f 48 39 c2 0f 84 2b 04 00 00 48 89 c2 48 8d 04 3a 48 85 f0 74 df 4d 85 e4 0f 84 50 04 00 00 f3 90 <e9> 57 ff ff ff e8 02 62 87 f5 84 c0 0f 85 f5 fd ff ff 48 c7 c2
> All code
> ========
> 0: 8d 0c 0a lea (%rdx,%rcx,1),%ecx
> 3: 48 89 d0 mov %rdx,%rax
> 6: f0 49 0f b1 0f lock cmpxchg %rcx,(%r15)
> b: 48 39 c2 cmp %rax,%rdx
> e: 0f 84 2b 04 00 00 je 0x43f
> 14: 48 89 c2 mov %rax,%rdx
> 17: 48 8d 04 3a lea (%rdx,%rdi,1),%rax
> 1b: 48 85 f0 test %rsi,%rax
> 1e: 74 df je 0xffffffffffffffff
> 20: 4d 85 e4 test %r12,%r12
> 23: 0f 84 50 04 00 00 je 0x479
> 29: f3 90 pause
> 2b:* e9 57 ff ff ff jmpq 0xffffffffffffff87 <-- trapping instruction
> 30: e8 02 62 87 f5 callq 0xfffffffff5876237
> 35: 84 c0 test %al,%al
> 37: 0f 85 f5 fd ff ff jne 0xfffffffffffffe32
> 3d: 48 rex.W
> 3e: c7 .byte 0xc7
> 3f: c2 .byte 0xc2
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: e9 57 ff ff ff jmpq 0xffffffffffffff5c
> 5: e8 02 62 87 f5 callq 0xfffffffff587620c
> a: 84 c0 test %al,%al
> c: 0f 85 f5 fd ff ff jne 0xfffffffffffffe07
> 12: 48 rex.W
> 13: c7 .byte 0xc7
> 14: c2 .byte 0xc2
> ...
> [ 880.960700] NMI backtrace for cpu 26
> [ 880.960700] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803c8c78000 ti: ffff88000eee0000 task.ti: ffff88000eee0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eee7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eee7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eee7fd8 R14: 0000000000000000 R15: ffff88000eee7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88049b200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f0f2bc1d5a8 CR3: 000000000678f000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eee7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eee7fd8 0000000000000000 ffff88000eee7d78 ffffffff931629ef
> [ 880.960700] ffff88000eee7e68 ffffffff934018e0 ffff88000eee7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 27
> [ 880.960700] CPU: 27 PID: 248 Comm: migration/27 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8804c4cf8000 ti: ffff8804c3b00000 task.ti: ffff8804c3b00000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8804c3b07b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0072423ee5 RCX: 1ffff1009899f2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8804c3b07bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88039211f738 R14: 0000000000000001 R15: ffff88039211f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804c5200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029c4ff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039211f728 0000000000000282
> [ 880.960700] ffff880400000000 00ff8804c4cf8000 ffff8804c3b07bd8 dffffc0000000000
> [ 880.960700] ffff8804c5218440 ffff88039211f778 ffff88039211f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 28
> [ 880.960700] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88065d0c0000 ti: ffff88000eef0000 task.ti: ffff88000eef0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eef7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eef7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eef7fd8 R14: 0000000000000000 R15: ffff88000eef7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804ef200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000b34fb8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eef7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eef7fd8 0000000000000000 ffff88000eef7d78 ffffffff931629ef
> [ 880.960700] ffff88000eef7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 29
> [ 880.960700] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88079d0b0000 ti: ffff88000eef8000 task.ti: ffff88000eef8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eeffd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eeffd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eefffd8 R14: 0000000000000000 R15: ffff88000eefffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880519000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f0f2b8e5210 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eeffd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eefffd8 0000000000000000 ffff88000eeffd78 ffffffff931629ef
> [ 880.960700] ffff88000eeffe68 ffffffff934018e0 ffff88000eeffde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 30
> [ 880.960700] CPU: 30 PID: 269 Comm: migration/30 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880542cf8000 ti: ffff880541b00000 task.ti: ffff880541b00000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880541b07b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed005ea54ee5 RCX: 1ffff100a859f2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880541b07bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8802f52a7738 R14: 0000000000000001 R15: ffff8802f52a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880543200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f42552112c0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802f52a7728 0000000000000282
> [ 880.960700] ffff880500000000 00ff880542cf8000 ffff880541b07bd8 dffffc0000000000
> [ 880.960700] ffff880543218440 ffff8802f52a7778 ffff8802f52a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 31
> [ 880.960700] CPU: 31 PID: 0 Comm: swapper/31 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88000f2d0000 ti: ffff88000f2d8000 task.ti: ffff88000f2d8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0000:ffff88000f2dfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000f2dfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000f2dffd8 R14: 0000000000000000 R15: ffff88000f2dffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88056d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007fff144e2980 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000f2dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000f2dffd8 0000000000000000 ffff88000f2dfd78 ffffffff931629ef
> [ 880.960700] ffff88000f2dfe68 ffffffff934018e0 ffff88000f2dfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
>
>
> Thanks,
> Sasha
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 11:27 ` Sasha Levin
@ 2015-03-06 12:32 ` Ingo Molnar
2015-03-06 14:34 ` Rafael David Tinoco
2015-03-06 17:19 ` Davidlohr Bueso
0 siblings, 2 replies; 44+ messages in thread
From: Ingo Molnar @ 2015-03-06 12:32 UTC (permalink / raw)
To: Sasha Levin
Cc: Peter Zijlstra, LKML, Dave Jones, Davidlohr Bueso, jason.low2,
Linus Torvalds
* Sasha Levin <sasha.levin@oracle.com> wrote:
> I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
That would be:
1a99367023f6 ("locking/rwsem: Check for active lock before bailing on spinning")
attached below.
Thanks,
Ingo
===========================>
>From 1a99367023f6ac664365a37fa508b059e31d0e88 Mon Sep 17 00:00:00 2001
From: Davidlohr Bueso <dave@stgolabs.net>
Date: Fri, 30 Jan 2015 01:14:27 -0800
Subject: [PATCH] locking/rwsem: Check for active lock before bailing on spinning
37e9562453b ("locking/rwsem: Allow conservative optimistic
spinning when readers have lock") forced the default for
optimistic spinning to be disabled if the lock owner was
nil, which makes much sense for readers. However, while
it is not our priority, we can make some optimizations
for write-mostly workloads. We can bail the spinning step
and still be conservative if there are any active tasks,
otherwise there's really no reason not to spin, as the
semaphore is most likely unlocked.
This patch recovers most of a Unixbench 'execl' benchmark
throughput by sleeping less and making better average system
usage:
before:
CPU %user %nice %system %iowait %steal %idle
all 0.60 0.00 8.02 0.00 0.00 91.38
after:
CPU %user %nice %system %iowait %steal %idle
all 1.22 0.00 70.18 0.00 0.00 28.60
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Jason Low <jason.low2@hp.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michel Lespinasse <walken@google.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Link: http://lkml.kernel.org/r/1422609267-15102-6-git-send-email-dave@stgolabs.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 1c0d11e8ce34..e4ad019e23f5 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
{
struct task_struct *owner;
- bool on_cpu = false;
+ bool ret = true;
if (need_resched())
return false;
rcu_read_lock();
owner = ACCESS_ONCE(sem->owner);
- if (owner)
- on_cpu = owner->on_cpu;
- rcu_read_unlock();
+ if (!owner) {
+ long count = ACCESS_ONCE(sem->count);
+ /*
+ * If sem->owner is not set, yet we have just recently entered the
+ * slowpath with the lock being active, then there is a possibility
+ * reader(s) may have the lock. To be safe, bail spinning in these
+ * situations.
+ */
+ if (count & RWSEM_ACTIVE_MASK)
+ ret = false;
+ goto done;
+ }
- /*
- * If sem->owner is not set, yet we have just recently entered the
- * slowpath, then there is a possibility reader(s) may have the lock.
- * To be safe, avoid spinning in these situations.
- */
- return on_cpu;
+ ret = owner->on_cpu;
+done:
+ rcu_read_unlock();
+ return ret;
}
static inline bool owner_running(struct rw_semaphore *sem,
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 12:32 ` Ingo Molnar
@ 2015-03-06 14:34 ` Rafael David Tinoco
2015-03-06 14:45 ` Sasha Levin
2015-03-06 17:19 ` Davidlohr Bueso
1 sibling, 1 reply; 44+ messages in thread
From: Rafael David Tinoco @ 2015-03-06 14:34 UTC (permalink / raw)
To: Ingo Molnar
Cc: Sasha Levin, Peter Zijlstra, LKML, Dave Jones, Davidlohr Bueso,
jason.low2, Linus Torvalds
Are you sure about this ? I have a core dump locked on the same place
(state machine for powering cpu down for the task swap) from a 3.13 (+
upstream patches) and this commit wasn't backported yet.
-> multi_cpu_stop -> do { } while (curstate != MULTI_STOP_EXIT);
In my case, curstate is WAY different from enum containing MULTI_STOP_EXIT (4).
Register totally messed up (probably after cpu_relax(), right where
you were trapped -> after the pause instruction).
my case:
PID: 118 TASK: ffff883fd28ec7d0 CPU: 9 COMMAND: "migration/9"
...
[exception RIP: multi_cpu_stop+0x64]
RIP: ffffffff810f5944 RSP: ffff883fd2907d98 RFLAGS: 00000246
RAX: 0000000000000010 RBX: 0000000000000010 RCX: 0000000000000246
RDX: ffff883fd2907d98 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffffff810f5944 R8: ffffffff810f5944 R9: 0000000000000000
R10: ffff883fd2907d98 R11: 0000000000000246 R12: ffffffffffffffff
R13: ffff883f55d01b48 R14: 0000000000000000 R15: 0000000000000001
ORIG_RAX: 0000000000000001 CS: 0010 SS: 0000
--- <NMI exception stack> ---
#4 [ffff883fd2907d98] multi_cpu_stop+0x64 at ffffffff810f5944
208 } while (curstate != MULTI_STOP_EXIT);
---> RIP
RIP 0xffffffff810f5944 <+100>: cmp $0x4,%edx
---> CHECKING FOR MULTI_STOP_EXIT
RDX: ffff883fd2907d98 -> does not make any sense
###
If i'm reading this right,
"""
CPU 05 - PID 14990
do_numa_page
task_numa_fault
numa_migrate_preferred
task_numa_migrate
migrate_swap (curr: 14990, task: 14996)
stop_two_cpus (cpu1=05(14996), cpu2=00(14990))
wait_for_completion
14990 - CPU05
14996 - CPU00
stop_two_cpus:
multi_stop_data (msdata->state = MULTI_STOP_PREPARE)
smp_call_function_single (min=cpu2=00, irq_cpu_stop_queue_work, wait=1)
smp_call_function_single (ran on lowest CPU, 00 for this case)
irq_cpu_stop_queue_work
cpu_stop_queue_work(cpu1=05(14996)) # add work
(multi_cpu_stop) to cpu 05 cpu_stopper queue
cpu_stop_queue_work(cpu2=00(14990)) # add work
(multi_cpu_stop) to cpu 00 cpu_stopper queue
wait_for_completion() --> HERE
"""
in my case, checking task structs for tasks scheduled when
"waiting_for_completion()":
PID 14990 CPU 05 -> PID 14996 CPU 00
PID 14991 CPU 30 -> PID 14998 CPU 01
PID 14992 CPU 30 -> PID 14998 CPU 01
PID 14996 CPU 00 -> PID 14992 CPU 30
PID 14998 CPU 01 -> PID 14990 CPU 05
AND
> 102 2 6 ffff881fd2ea97f0 RU 0.0 0 0 [migration/6]
> 118 2 9 ffff883fd28ec7d0 RU 0.0 0 0 [migration/9]
> 143 2 14 ffff883fd29d47d0 RU 0.0 0 0 [migration/14]
> 148 2 15 ffff883fd29fc7d0 RU 0.0 0 0 [migration/15]
> 153 2 16 ffff881fd2f517f0 RU 0.0 0 0 [migration/16]
THEN
I am still waiting for 5 cpu_stopper_thread -> multi_cpu_stop just
scheduled (probably in the per cpu's queue of cpus 0,1,5,30), not
running yet.
AND
I don't have any "wait_for_completion" for those "OLDER" migration
threads (6, 9, 14, 15 and 16)
Probably wait_for_completion signaled done.completion before racing.
Looks like something messed up with curstate in the "multi_cpu_stop"
state machine.
/* Simple state machine */
do {
/* Chill out and ensure we re-read multi_stop_state. */
cpu_relax();
cpu_relax maybe ?
--
Rafael Tinoco
On Fri, Mar 6, 2015 at 9:32 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Sasha Levin <sasha.levin@oracle.com> wrote:
>
>> I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
>
> That would be:
>
> 1a99367023f6 ("locking/rwsem: Check for active lock before bailing on spinning")
>
> attached below.
>
> Thanks,
>
> Ingo
>
> ===========================>
> From 1a99367023f6ac664365a37fa508b059e31d0e88 Mon Sep 17 00:00:00 2001
> From: Davidlohr Bueso <dave@stgolabs.net>
> Date: Fri, 30 Jan 2015 01:14:27 -0800
> Subject: [PATCH] locking/rwsem: Check for active lock before bailing on spinning
>
> 37e9562453b ("locking/rwsem: Allow conservative optimistic
> spinning when readers have lock") forced the default for
> optimistic spinning to be disabled if the lock owner was
> nil, which makes much sense for readers. However, while
> it is not our priority, we can make some optimizations
> for write-mostly workloads. We can bail the spinning step
> and still be conservative if there are any active tasks,
> otherwise there's really no reason not to spin, as the
> semaphore is most likely unlocked.
>
> This patch recovers most of a Unixbench 'execl' benchmark
> throughput by sleeping less and making better average system
> usage:
>
> before:
> CPU %user %nice %system %iowait %steal %idle
> all 0.60 0.00 8.02 0.00 0.00 91.38
>
> after:
> CPU %user %nice %system %iowait %steal %idle
> all 1.22 0.00 70.18 0.00 0.00 28.60
>
> Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Acked-by: Jason Low <jason.low2@hp.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: Michel Lespinasse <walken@google.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Tim Chen <tim.c.chen@linux.intel.com>
> Link: http://lkml.kernel.org/r/1422609267-15102-6-git-send-email-dave@stgolabs.net
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
> ---
> kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++----------
> 1 file changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index 1c0d11e8ce34..e4ad019e23f5 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
> static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
> {
> struct task_struct *owner;
> - bool on_cpu = false;
> + bool ret = true;
>
> if (need_resched())
> return false;
>
> rcu_read_lock();
> owner = ACCESS_ONCE(sem->owner);
> - if (owner)
> - on_cpu = owner->on_cpu;
> - rcu_read_unlock();
> + if (!owner) {
> + long count = ACCESS_ONCE(sem->count);
> + /*
> + * If sem->owner is not set, yet we have just recently entered the
> + * slowpath with the lock being active, then there is a possibility
> + * reader(s) may have the lock. To be safe, bail spinning in these
> + * situations.
> + */
> + if (count & RWSEM_ACTIVE_MASK)
> + ret = false;
> + goto done;
> + }
>
> - /*
> - * If sem->owner is not set, yet we have just recently entered the
> - * slowpath, then there is a possibility reader(s) may have the lock.
> - * To be safe, avoid spinning in these situations.
> - */
> - return on_cpu;
> + ret = owner->on_cpu;
> +done:
> + rcu_read_unlock();
> + return ret;
> }
>
> static inline bool owner_running(struct rw_semaphore *sem,
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 14:34 ` Rafael David Tinoco
@ 2015-03-06 14:45 ` Sasha Levin
2015-03-06 15:46 ` Sasha Levin
0 siblings, 1 reply; 44+ messages in thread
From: Sasha Levin @ 2015-03-06 14:45 UTC (permalink / raw)
To: Rafael David Tinoco, Ingo Molnar
Cc: Peter Zijlstra, LKML, Dave Jones, Davidlohr Bueso, jason.low2,
Linus Torvalds
On 03/06/2015 09:34 AM, Rafael David Tinoco wrote:
> Are you sure about this ? I have a core dump locked on the same place
> (state machine for powering cpu down for the task swap) from a 3.13 (+
> upstream patches) and this commit wasn't backported yet.
bisect took me to that same commit twice, and I'm running a -next with that
commit reverted and not seeing those hangs anymore.
I also didn't see the hang up until a month or so ago (which matches with that
commit date), and definitely not back in 3.13.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 14:45 ` Sasha Levin
@ 2015-03-06 15:46 ` Sasha Levin
0 siblings, 0 replies; 44+ messages in thread
From: Sasha Levin @ 2015-03-06 15:46 UTC (permalink / raw)
To: Rafael David Tinoco, Ingo Molnar
Cc: Peter Zijlstra, LKML, Dave Jones, Davidlohr Bueso, jason.low2,
Linus Torvalds
On 03/06/2015 09:45 AM, Sasha Levin wrote:
> On 03/06/2015 09:34 AM, Rafael David Tinoco wrote:
>> Are you sure about this ? I have a core dump locked on the same place
>> (state machine for powering cpu down for the task swap) from a 3.13 (+
>> upstream patches) and this commit wasn't backported yet.
>
> bisect took me to that same commit twice, and I'm running a -next with that
> commit reverted and not seeing those hangs anymore.
>
> I also didn't see the hang up until a month or so ago (which matches with that
> commit date), and definitely not back in 3.13.
I just remembered that I was seeing something similar back then, which was fixed:
http://lists.openwall.net/linux-kernel/2014/05/12/725
Thanks,
Sasha
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 12:32 ` Ingo Molnar
2015-03-06 14:34 ` Rafael David Tinoco
@ 2015-03-06 17:19 ` Davidlohr Bueso
2015-03-06 18:02 ` Sasha Levin
2015-03-06 18:57 ` Jason Low
1 sibling, 2 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 17:19 UTC (permalink / raw)
To: Ingo Molnar
Cc: Sasha Levin, Peter Zijlstra, LKML, Dave Jones, jason.low2,
Linus Torvalds
On Fri, 2015-03-06 at 13:32 +0100, Ingo Molnar wrote:
> * Sasha Levin <sasha.levin@oracle.com> wrote:
>
> > I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
>
> That would be:
>
> 1a99367023f6 ("locking/rwsem: Check for active lock before bailing on spinning")
>
> attached below.
[...]
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index 1c0d11e8ce34..e4ad019e23f5 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
> static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
> {
> struct task_struct *owner;
> - bool on_cpu = false;
> + bool ret = true;
>
> if (need_resched())
> return false;
>
> rcu_read_lock();
> owner = ACCESS_ONCE(sem->owner);
> - if (owner)
> - on_cpu = owner->on_cpu;
> - rcu_read_unlock();
> + if (!owner) {
> + long count = ACCESS_ONCE(sem->count);
> + /*
> + * If sem->owner is not set, yet we have just recently entered the
> + * slowpath with the lock being active, then there is a possibility
> + * reader(s) may have the lock. To be safe, bail spinning in these
> + * situations.
> + */
> + if (count & RWSEM_ACTIVE_MASK)
> + ret = false;
> + goto done;
Hmmm so the lockup would be due to this (when owner is non-nil the patch
has no effect), telling users to spin instead of sleep -- _except_ for
this condition. And when spinning we're always checking for need_resched
to be safe. So even if this function was completely bogus, we'd end up
needlessly spinning but I'm surprised about the lockup. Maybe coffee
will make things clearer.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 17:19 ` Davidlohr Bueso
@ 2015-03-06 18:02 ` Sasha Levin
2015-03-06 21:59 ` Sasha Levin
2015-03-06 18:57 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Sasha Levin @ 2015-03-06 18:02 UTC (permalink / raw)
To: Davidlohr Bueso, Ingo Molnar
Cc: Peter Zijlstra, LKML, Dave Jones, jason.low2, Linus Torvalds
On 03/06/2015 12:19 PM, Davidlohr Bueso wrote:
>> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
>> > index 1c0d11e8ce34..e4ad019e23f5 100644
>> > --- a/kernel/locking/rwsem-xadd.c
>> > +++ b/kernel/locking/rwsem-xadd.c
>> > @@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
>> > static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
>> > {
>> > struct task_struct *owner;
>> > - bool on_cpu = false;
>> > + bool ret = true;
>> >
>> > if (need_resched())
>> > return false;
>> >
>> > rcu_read_lock();
>> > owner = ACCESS_ONCE(sem->owner);
>> > - if (owner)
>> > - on_cpu = owner->on_cpu;
>> > - rcu_read_unlock();
>> > + if (!owner) {
>> > + long count = ACCESS_ONCE(sem->count);
>> > + /*
>> > + * If sem->owner is not set, yet we have just recently entered the
>> > + * slowpath with the lock being active, then there is a possibility
>> > + * reader(s) may have the lock. To be safe, bail spinning in these
>> > + * situations.
>> > + */
>> > + if (count & RWSEM_ACTIVE_MASK)
>> > + ret = false;
>> > + goto done;
> Hmmm so the lockup would be due to this (when owner is non-nil the patch
> has no effect), telling users to spin instead of sleep -- _except_ for
> this condition. And when spinning we're always checking for need_resched
> to be safe. So even if this function was completely bogus, we'd end up
> needlessly spinning but I'm surprised about the lockup. Maybe coffee
> will make things clearer.
There's always the possibility that bisect went wrong. I did it twice, but
since I don't have a sure way of reproducing it I was basing my good/bad
decisions on whether I saw it within a reasonable amount of time.
I can go redo that again if you suspect that that commit is not the cause.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 17:19 ` Davidlohr Bueso
2015-03-06 18:02 ` Sasha Levin
@ 2015-03-06 18:57 ` Jason Low
2015-03-06 19:05 ` Linus Torvalds
1 sibling, 1 reply; 44+ messages in thread
From: Jason Low @ 2015-03-06 18:57 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones,
Linus Torvalds, jason.low2
On Fri, 2015-03-06 at 09:19 -0800, Davidlohr Bueso wrote:
> On Fri, 2015-03-06 at 13:32 +0100, Ingo Molnar wrote:
> > * Sasha Levin <sasha.levin@oracle.com> wrote:
> >
> > > I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
> >
> > That would be:
> >
> > 1a99367023f6 ("locking/rwsem: Check for active lock before bailing on spinning")
>
> > diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> > index 1c0d11e8ce34..e4ad019e23f5 100644
> > --- a/kernel/locking/rwsem-xadd.c
> > +++ b/kernel/locking/rwsem-xadd.c
> > @@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
> > static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
> > {
> > struct task_struct *owner;
> > - bool on_cpu = false;
> > + bool ret = true;
> >
> > if (need_resched())
> > return false;
> >
> > rcu_read_lock();
> > owner = ACCESS_ONCE(sem->owner);
> > - if (owner)
> > - on_cpu = owner->on_cpu;
> > - rcu_read_unlock();
> > + if (!owner) {
> > + long count = ACCESS_ONCE(sem->count);
> > + /*
> > + * If sem->owner is not set, yet we have just recently entered the
> > + * slowpath with the lock being active, then there is a possibility
> > + * reader(s) may have the lock. To be safe, bail spinning in these
> > + * situations.
> > + */
> > + if (count & RWSEM_ACTIVE_MASK)
> > + ret = false;
> > + goto done;
>
> Hmmm so the lockup would be due to this (when owner is non-nil the patch
> has no effect), telling users to spin instead of sleep -- _except_ for
> this condition. And when spinning we're always checking for need_resched
> to be safe. So even if this function was completely bogus, we'd end up
> needlessly spinning but I'm surprised about the lockup. Maybe coffee
> will make things clearer.
Right, the can_spin_on_owner() was originally added to the mutex
spinning code for optimization purposes, particularly so that we can
avoid adding the spinner to the OSQ only to find that it doesn't need to
spin. This function needing to return a correct value should really only
affect performance, so yes, lockups due to this seems surprising.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 18:57 ` Jason Low
@ 2015-03-06 19:05 ` Linus Torvalds
2015-03-06 19:20 ` Davidlohr Bueso
2015-03-06 19:29 ` Jason Low
0 siblings, 2 replies; 44+ messages in thread
From: Linus Torvalds @ 2015-03-06 19:05 UTC (permalink / raw)
To: Jason Low
Cc: Davidlohr Bueso, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones
On Fri, Mar 6, 2015 at 10:57 AM, Jason Low <jason.low2@hp.com> wrote:
>
> Right, the can_spin_on_owner() was originally added to the mutex
> spinning code for optimization purposes, particularly so that we can
> avoid adding the spinner to the OSQ only to find that it doesn't need to
> spin. This function needing to return a correct value should really only
> affect performance, so yes, lockups due to this seems surprising.
Well, softlockups aren't about "correct behavior". They are about
certain things not happening in a timely manner.
Clearly the mutex code now tries to hold on to the CPU too aggressively.
At some point people need to admit that busy-looping isn't always a
good idea. Especially if
(a) we could idle the core instead
(b) the tuning has been done based on som especial-purpose benchmark
that is likely not realistic
(c) we get reports from people that it causes problems.
In other words: Let's just undo that excessive busy-looping. The
performance numbers were dubious to begin with. Real scalability comes
from fixing the locking, not from trying to play games with the locks
themselves. Particularly games that then cause problems.
Linus
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:05 ` Linus Torvalds
@ 2015-03-06 19:20 ` Davidlohr Bueso
2015-03-06 19:32 ` Linus Torvalds
2015-03-06 19:29 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 19:20 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, 2015-03-06 at 11:05 -0800, Linus Torvalds wrote:
> On Fri, Mar 6, 2015 at 10:57 AM, Jason Low <jason.low2@hp.com> wrote:
> >
> > Right, the can_spin_on_owner() was originally added to the mutex
> > spinning code for optimization purposes, particularly so that we can
> > avoid adding the spinner to the OSQ only to find that it doesn't need to
> > spin. This function needing to return a correct value should really only
> > affect performance, so yes, lockups due to this seems surprising.
>
> Well, softlockups aren't about "correct behavior". They are about
> certain things not happening in a timely manner.
>
> Clearly the mutex code now tries to hold on to the CPU too aggressively.
This patch was a performance "fix" for rwsems, where it works well
mutexes.
>
> At some point people need to admit that busy-looping isn't always a
> good idea. Especially if
>
> (a) we could idle the core instead
>
> (b) the tuning has been done based on som especial-purpose benchmark
> that is likely not realistic
>
> (c) we get reports from people that it causes problems.
>
> In other words: Let's just undo that excessive busy-looping. The
> performance numbers were dubious to begin with. Real scalability comes
> from fixing the locking, not from trying to play games with the locks
> themselves. Particularly games that then cause problems.
I obviously agree with all those points, however fyi most of the testing
on rwsems I do includes scaling address space ops stressing the
mmap_sem, which is a real world concern. So while it does include
microbenchmarks, it is not guided by them.
Thanks,
Davidlohr
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:05 ` Linus Torvalds
2015-03-06 19:20 ` Davidlohr Bueso
@ 2015-03-06 19:29 ` Jason Low
2015-03-06 21:12 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Jason Low @ 2015-03-06 19:29 UTC (permalink / raw)
To: Linus Torvalds
Cc: Davidlohr Bueso, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones, jason.low2
On Fri, 2015-03-06 at 11:05 -0800, Linus Torvalds wrote:
> On Fri, Mar 6, 2015 at 10:57 AM, Jason Low <jason.low2@hp.com> wrote:
> >
> > Right, the can_spin_on_owner() was originally added to the mutex
> > spinning code for optimization purposes, particularly so that we can
> > avoid adding the spinner to the OSQ only to find that it doesn't need to
> > spin. This function needing to return a correct value should really only
> > affect performance, so yes, lockups due to this seems surprising.
>
> Well, softlockups aren't about "correct behavior". They are about
> certain things not happening in a timely manner.
>
> Clearly the mutex code now tries to hold on to the CPU too aggressively.
>
> At some point people need to admit that busy-looping isn't always a
> good idea. Especially if
>
> (a) we could idle the core instead
>
> (b) the tuning has been done based on som especial-purpose benchmark
> that is likely not realistic
>
> (c) we get reports from people that it causes problems.
>
> In other words: Let's just undo that excessive busy-looping. The
> performance numbers were dubious to begin with. Real scalability comes
> from fixing the locking, not from trying to play games with the locks
> themselves. Particularly games that then cause problems.
Hi Linus,
Agreed, this is an issue we need to address, though we're just trying to
figure out if the change to rwsem_can_spin_on_owner() in "commit:
37e9562453b" is really the one that's causing the issue.
For example, it looks like Ming recently found another change in the
same patchset: commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving
lock spinners) to be causing lockups.
https://lkml.org/lkml/2015/3/6/521
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:20 ` Davidlohr Bueso
@ 2015-03-06 19:32 ` Linus Torvalds
2015-03-06 19:45 ` Davidlohr Bueso
2015-03-06 19:55 ` Davidlohr Bueso
0 siblings, 2 replies; 44+ messages in thread
From: Linus Torvalds @ 2015-03-06 19:32 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, Mar 6, 2015 at 11:20 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
>
> I obviously agree with all those points, however fyi most of the testing
> on rwsems I do includes scaling address space ops stressing the
> mmap_sem, which is a real world concern. So while it does include
> microbenchmarks, it is not guided by them.
So I agree that mmap_sem is problematic.
We probably still end up holding it over many actual IO operations,
for example. The whole "FAULT_RETRY" thing should have helped a lot,
in that hopefully at least a fair amount of the time we now end up
waiting for the IO without holding the semaphore, but I bet many other
cases remain.
And I also suspect that we could try to be even more aggressive, and
allow some entirely unlocked cases. For example, long long ago we used
to have a completely SMP-unsafe model where we would do things
optimistically - doing IO without holding any locks, and then before
we "committed" to it, we'd re-try. And I wonder if we might want to
re-introduce that for the cases where we hit in caches and could use
RCU.
IOW, I wonder if we could special-case the common non-IO
fault-handling path something along the lines of:
- look up the vma in the vma lookup cache
- look up the page in the page cache
- get the page table spinlock
- re-check the vma now (it ends up being stable if it can't be torn
down due to the page table spinlock)
because I suspect that page faults are the biggest users of that
mmap_sem, and we could probably handle a fairly large common case
(making it simpler by special-casing it and punting in any even
_slightly_ complicated situations) without even getting the semaphore
at all, since we have to serialize on the actual page table *anyway*.
Basically, to me, the whole "if a lock is so contended that we need to
play locking games, then we should look at why we *use* the lock,
rather than at the lock itself" is a religion.
Linus
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:32 ` Linus Torvalds
@ 2015-03-06 19:45 ` Davidlohr Bueso
2015-03-06 19:55 ` Davidlohr Bueso
1 sibling, 0 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 19:45 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, 2015-03-06 at 11:32 -0800, Linus Torvalds wrote:
> Basically, to me, the whole "if a lock is so contended that we need to
> play locking games, then we should look at why we *use* the lock,
> rather than at the lock itself" is a religion.
Oh absolutely, I'm only mentioning the locking primitive side of
mmap_sem, for which minor optimizations will _never_ solve the actual
scalability issues. But yes, point taken.
Thanks,
Davidlohr
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:32 ` Linus Torvalds
2015-03-06 19:45 ` Davidlohr Bueso
@ 2015-03-06 19:55 ` Davidlohr Bueso
2015-03-06 20:00 ` Davidlohr Bueso
2015-03-06 21:42 ` Linus Torvalds
1 sibling, 2 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 19:55 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, 2015-03-06 at 11:32 -0800, Linus Torvalds wrote:
> IOW, I wonder if we could special-case the common non-IO
> fault-handling path something along the lines of:
>
> - look up the vma in the vma lookup cache
But you'd still need mmap_sem there to at least get the VMA's first
value.
> - look up the page in the page cache
> - get the page table spinlock
> - re-check the vma now (it ends up being stable if it can't be torn
> down due to the page table spinlock)
>
> because I suspect that page faults are the biggest users of that
> mmap_sem, and we could probably handle a fairly large common case
> (making it simpler by special-casing it and punting in any even
> _slightly_ complicated situations) without even getting the semaphore
> at all, since we have to serialize on the actual page table *anyway*.
>
> Basically, to me, the whole "if a lock is so contended that we need to
> play locking games, then we should look at why we *use* the lock,
> rather than at the lock itself" is a religion.
>
> Linus
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:55 ` Davidlohr Bueso
@ 2015-03-06 20:00 ` Davidlohr Bueso
2015-03-06 21:42 ` Linus Torvalds
1 sibling, 0 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 20:00 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, 2015-03-06 at 11:55 -0800, Davidlohr Bueso wrote:
> On Fri, 2015-03-06 at 11:32 -0800, Linus Torvalds wrote:
>
> > IOW, I wonder if we could special-case the common non-IO
> > fault-handling path something along the lines of:
> >
> > - look up the vma in the vma lookup cache
>
> But you'd still need mmap_sem there to at least get the VMA's first
> value.
Incomplete msg... what I'm getting at is that one way or another
mmap_sem contention is easy to hit -- yes, avoiding it in IO paths is
obviously good for hold times. But I also think that it protects way too
much, ie making pthread intense workloads really suck as well.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 19:29 ` Jason Low
@ 2015-03-06 21:12 ` Jason Low
2015-03-06 21:24 ` Linus Torvalds
` (2 more replies)
0 siblings, 3 replies; 44+ messages in thread
From: Jason Low @ 2015-03-06 21:12 UTC (permalink / raw)
To: Linus Torvalds
Cc: Davidlohr Bueso, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones, jason.low2
On Fri, 2015-03-06 at 11:29 -0800, Jason Low wrote:
> Hi Linus,
>
> Agreed, this is an issue we need to address, though we're just trying to
> figure out if the change to rwsem_can_spin_on_owner() in "commit:
> 37e9562453b" is really the one that's causing the issue.
>
> For example, it looks like Ming recently found another change in the
> same patchset: commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving
> lock spinners) to be causing lockups.
>
> https://lkml.org/lkml/2015/3/6/521
So I think I may have spotted a problem in the tip commit:
Commit b3fd4f03ca0b995 (locking/rwsem: Avoid deceiving lock spinners).
In owner_running() there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However,
that patch continues spinning if there is a "new owner" but it does not
take into account that we may want to stop spinning if the owner is not
running (due to getting rescheduled).
So we we really want this right (not yet tested):
---
Subject: [PATCH] locking/rwsem: Avoid spinning when owner is not running
not-yet-Signed-off-by: Jason Low <jason.low2@hp.com>
---
kernel/locking/rwsem-xadd.c | 28 ++++++++--------------------
1 files changed, 8 insertions(+), 20 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 06e2214..e9379ee 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -324,32 +324,20 @@ done:
return ret;
}
-static inline bool owner_running(struct rw_semaphore *sem,
- struct task_struct *owner)
-{
- if (sem->owner != owner)
- return false;
-
- /*
- * Ensure we emit the owner->on_cpu, dereference _after_ checking
- * sem->owner still matches owner, if that fails, owner might
- * point to free()d memory, if it still matches, the rcu_read_lock()
- * ensures the memory stays valid.
- */
- barrier();
-
- return owner->on_cpu;
-}
-
static noinline
bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
{
long count;
rcu_read_lock();
- while (owner_running(sem, owner)) {
- /* abort spinning when need_resched */
- if (need_resched()) {
+ while (true) {
+ if (sem->owner != owner)
+ break;
+
+ barrier();
+
+ /* abort spinning when need_resched or owner is not running*/
+ if (!owner->on_cpu || need_resched()) {
rcu_read_unlock();
return false;
}
--
1.7.2.5
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 21:12 ` Jason Low
@ 2015-03-06 21:24 ` Linus Torvalds
2015-03-07 1:53 ` Jason Low
2015-03-06 22:15 ` Davidlohr Bueso
2015-03-07 4:31 ` Jason Low
2 siblings, 1 reply; 44+ messages in thread
From: Linus Torvalds @ 2015-03-06 21:24 UTC (permalink / raw)
To: Jason Low
Cc: Davidlohr Bueso, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones
On Fri, Mar 6, 2015 at 1:12 PM, Jason Low <jason.low2@hp.com> wrote:
>
> + while (true) {
> + if (sem->owner != owner)
> + break;
That looks *really* odd.
Why is this not
while (sem->owner == owner) {
Also, this "barrier()" now lost the comment:
> + barrier();
so it looks very odd indeed.
Linus
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 19:55 ` Davidlohr Bueso
2015-03-06 20:00 ` Davidlohr Bueso
@ 2015-03-06 21:42 ` Linus Torvalds
1 sibling, 0 replies; 44+ messages in thread
From: Linus Torvalds @ 2015-03-06 21:42 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Fri, Mar 6, 2015 at 11:55 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
>>
>> - look up the vma in the vma lookup cache
>
> But you'd still need mmap_sem there to at least get the VMA's first
> value.
So my theory was that the vma cache is such a trivial data structure
that we could trivially make it be rcu-protected.
The vma allocations are already SLAB_DESTROY_BY_RCU, because we play
games with the anon-vma stuff. Or something. i forget the exact
details.
So I think that vmacache_find() would *already* basically work under
just the RCU read lock, and we can look at the resulting vma without
having to worry about it getting free'd.
Yes, the actual field values may change (ie start/end offsets etc due
to vma merging etc), but again, that's not necessarily deadly if we
are careful and make use of the vmacache sequence number. We can
optimistically do things like page cache lookups (which is already RCU
safe), and then before we actually *use* the result, we do another
vmacache sequence number validation.
So I *think* we could do at least that limited "we hit in the vma
cache, and it's a nice normal simple vma with regular vma ops" with
just a RCU read lock, and skip the mmap_sem entirely. Of course, we'd
have to fall back on the mmap_sem if anything fails (not in the vma
cache, or the sequence number changes before we can actually insert
the result in the page tables etc).
The page cache is already RCU-safe, and the actual page table
operations are protected by another lock anyway (which should scale
better because it's a spinlock and held for shorter times, _and_ is
spread out by pte address).
Is it some trivial one-liner? No. But I suspect we could make a trial
"lockless page lookup for the simple cases that hit in the caches"
without a *lot* of effort.
Linus
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: sched: softlockups in multi_cpu_stop
2015-03-06 18:02 ` Sasha Levin
@ 2015-03-06 21:59 ` Sasha Levin
0 siblings, 0 replies; 44+ messages in thread
From: Sasha Levin @ 2015-03-06 21:59 UTC (permalink / raw)
To: Davidlohr Bueso, Ingo Molnar
Cc: Peter Zijlstra, LKML, Dave Jones, jason.low2, Linus Torvalds
On 03/06/2015 01:02 PM, Sasha Levin wrote:
> I can go redo that again if you suspect that that commit is not the cause.
I took a closer look at the logs, and I'm seeing hangs that begin this way
as well:
[ 2298.020237] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [trinity-c19:839]
[ 2298.020237] Modules linked in:
[ 2298.020237] CPU: 19 PID: 839 Comm: trinity-c19 Not tainted 4.0.0-rc2-next-20150306-sasha-00056-g61886e8 #2005
[ 2298.020237] task: ffff880278d62000 ti: ffff880254fe8000 task.ti: ffff880254fe8000
[ 2298.020237] RIP: 0010:[<ffffffffa442702f>] [<ffffffffa442702f>] __rcu_read_unlock+0x9f/0x130
[ 2298.020237] RSP: 0000:ffff880254fefbd8 EFLAGS: 00000207
[ 2298.020237] RAX: dffffc0000000000 RBX: ffff880254fe8000 RCX: 1ffff1004a9fd002
[ 2298.020237] RDX: 1ffff1004f1ac4e2 RSI: ffff8802c3ff6000 RDI: ffff880278d62714
[ 2298.020237] RBP: ffff880254fefbe8 R08: ffff880362b2e080 R09: ffffffff00000001
[ 2298.020237] R10: ffff880362b2e140 R11: ffffea000e253800 R12: 0000000000000a3e
[ 2298.020237] R13: ffff880278d62cb0 R14: ffffed014e1e4899 R15: 0034c1c55efd9eff
[ 2298.020237] FS: 00007f183b9c3700(0000) GS:ffff880375200000(0000) knlGS:0000000000000000
[ 2298.020237] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2298.020237] CR2: 0000000000bc8fe8 CR3: 0000000259a0a000 CR4: 00000000000007a0
[ 2298.020237] Stack:
[ 2298.020237] ffff8802c3ff6000 ffff880126cfdc28 ffff880254fefc38 ffffffffa43fcb75
[ 2298.020237] ffff8802c3ff6000 ffff8802c3ff6000 ffff880278d62714 ffff880126cfdc48
[ 2298.020237] ffff8802c3ff6000 ffff880126cfdc44 ffff880126cfdc28 ffff880254fefd78
[ 2298.020237] Call Trace:
[ 2298.020237] [<ffffffffa43fcb75>] rwsem_spin_on_owner+0x165/0x250
[ 2298.020237] [<ffffffffae92a67f>] rwsem_down_write_failed+0x22f/0x750
[ 2298.020237] [<ffffffffae92a450>] ? rwsem_down_read_failed+0x260/0x260
[ 2298.020237] [<ffffffffa438fc31>] ? get_parent_ip+0x11/0x50
[ 2298.020237] [<ffffffffa438fd76>] ? preempt_count_add+0x106/0x160
[ 2298.020237] [<ffffffffa5f77c77>] ? debug_smp_processor_id+0x17/0x20
[ 2298.020237] [<ffffffffa47128c0>] ? cmpxchg_double_slab.isra.25+0x210/0x240
[ 2298.020237] [<ffffffffa47119df>] ? free_debug_processing+0x19f/0x320
[ 2298.020237] [<ffffffffa5f4da33>] call_rwsem_down_write_failed+0x13/0x20
[ 2298.020237] [<ffffffffae9296a9>] ? down_write+0x29/0x70
[ 2298.020237] [<ffffffffa46acd32>] validate_mm+0xa2/0x910
[ 2298.020237] [<ffffffffa46b5fd1>] do_munmap+0x421/0xf50
[ 2298.020237] [<ffffffffa41627d0>] ? send_sigtrap+0x1e0/0x1e0
[ 2298.020237] [<ffffffffa46b6b5f>] vm_munmap+0x5f/0x80
[ 2298.020237] [<ffffffffa46b9562>] SyS_munmap+0x22/0x30
[ 2298.020237] [<ffffffffae92e60d>] system_call_fastpath+0x16/0x1b
[ 2298.020237] Code: 02 84 c0 74 04 3c 03 7e 7c c7 83 10 07 00 00 00 00 00 80 48 8d bb 14 07 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 14 02 48 89 f8 83 e0 07 83 c0 01 38 d0 7c 04 84 d2 75 52
So it seems that we end up spinning for quite a while?
Thanks,
Sasha
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 21:12 ` Jason Low
2015-03-06 21:24 ` Linus Torvalds
@ 2015-03-06 22:15 ` Davidlohr Bueso
2015-03-07 1:55 ` Ming Lei
2015-03-07 1:58 ` Jason Low
2015-03-07 4:31 ` Jason Low
2 siblings, 2 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-06 22:15 UTC (permalink / raw)
To: Jason Low
Cc: Linus Torvalds, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones, Ming Lei
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> In owner_running() there are 2 conditions that would make it return
> false: if the owner changed or if the owner is not running. However,
> that patch continues spinning if there is a "new owner" but it does not
> take into account that we may want to stop spinning if the owner is not
> running (due to getting rescheduled).
So you're rationale is that we're missing this need_resched:
while (owner_running(sem, owner)) {
/* abort spinning when need_resched */
if (need_resched()) {
rcu_read_unlock();
return false;
}
}
Because the owner_running() would return false, right? Yeah that makes
sense, as missing a resched is a bug, as opposed to our heuristics being
so painfully off.
Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
Thanks,
Davidlohr
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 21:24 ` Linus Torvalds
@ 2015-03-07 1:53 ` Jason Low
0 siblings, 0 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 1:53 UTC (permalink / raw)
To: Linus Torvalds
Cc: Davidlohr Bueso, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones, jason.low2
On Fri, 2015-03-06 at 13:24 -0800, Linus Torvalds wrote:
> On Fri, Mar 6, 2015 at 1:12 PM, Jason Low <jason.low2@hp.com> wrote:
> >
> > + while (true) {
> > + if (sem->owner != owner)
> > + break;
>
> That looks *really* odd.
>
> Why is this not
>
> while (sem->owner == owner) {
Yes, this looks more readable.
That while (true) thing was something we recently did for mutexes which
was why I originally had that.
> Also, this "barrier()" now lost the comment:
>
> > + barrier();
>
> so it looks very odd indeed.
Right, we should keep the comment for the barrier().
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 22:15 ` Davidlohr Bueso
@ 2015-03-07 1:55 ` Ming Lei
2015-03-07 2:07 ` Davidlohr Bueso
2015-03-07 1:58 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Ming Lei @ 2015-03-07 1:55 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Fri, 06 Mar 2015 14:15:37 -0800
Davidlohr Bueso <dave@stgolabs.net> wrote:
> On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> > In owner_running() there are 2 conditions that would make it return
> > false: if the owner changed or if the owner is not running. However,
> > that patch continues spinning if there is a "new owner" but it does not
> > take into account that we may want to stop spinning if the owner is not
> > running (due to getting rescheduled).
>
> So you're rationale is that we're missing this need_resched:
>
> while (owner_running(sem, owner)) {
> /* abort spinning when need_resched */
> if (need_resched()) {
> rcu_read_unlock();
> return false;
> }
> }
>
> Because the owner_running() would return false, right? Yeah that makes
> sense, as missing a resched is a bug, as opposed to our heuristics being
> so painfully off.
>
> Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since
the following simple change does fix the issue:
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 06e2214..5e08705 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
}
rcu_read_unlock();
- if (READ_ONCE(sem->owner))
- return true; /* new owner, continue spinning */
+ owner = READ_ONCE(sem->owner);
+ if (owner && owner->on_cpu)
+ return true;
/*
* When the owner is not set, the lock could be free or
Thanks,
Ming Lei
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 22:15 ` Davidlohr Bueso
2015-03-07 1:55 ` Ming Lei
@ 2015-03-07 1:58 ` Jason Low
1 sibling, 0 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 1:58 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Linus Torvalds, Ingo Molnar, Sasha Levin, Peter Zijlstra, LKML,
Dave Jones, Ming Lei, jason.low2
On Fri, 2015-03-06 at 14:15 -0800, Davidlohr Bueso wrote:
> On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> > In owner_running() there are 2 conditions that would make it return
> > false: if the owner changed or if the owner is not running. However,
> > that patch continues spinning if there is a "new owner" but it does not
> > take into account that we may want to stop spinning if the owner is not
> > running (due to getting rescheduled).
>
> So you're rationale is that we're missing this need_resched:
>
> while (owner_running(sem, owner)) {
> /* abort spinning when need_resched */
> if (need_resched()) {
> rcu_read_unlock();
> return false;
> }
> }
>
> Because the owner_running() would return false, right? Yeah that makes
> sense, as missing a resched is a bug, as opposed to our heuristics being
> so painfully off.
Actually, the rationale is that when the lock owner reschedules while
holding the lock, we'd want the spinners to stop spinning. The original
owner_running() check takes care of this since it returns false if
->on_cpu gets set to false and the sem->owner != NULL would be false
causing us to stop spinning . However, with the patch, when
owner_running returns false, we check sem->owner, which causes the
->on_cpu check to essentially get ignored.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 1:55 ` Ming Lei
@ 2015-03-07 2:07 ` Davidlohr Bueso
2015-03-07 2:10 ` Ming Lei
0 siblings, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 2:07 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> On Fri, 06 Mar 2015 14:15:37 -0800
> Davidlohr Bueso <dave@stgolabs.net> wrote:
>
> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> > > In owner_running() there are 2 conditions that would make it return
> > > false: if the owner changed or if the owner is not running. However,
> > > that patch continues spinning if there is a "new owner" but it does not
> > > take into account that we may want to stop spinning if the owner is not
> > > running (due to getting rescheduled).
> >
> > So you're rationale is that we're missing this need_resched:
> >
> > while (owner_running(sem, owner)) {
> > /* abort spinning when need_resched */
> > if (need_resched()) {
> > rcu_read_unlock();
> > return false;
> > }
> > }
> >
> > Because the owner_running() would return false, right? Yeah that makes
> > sense, as missing a resched is a bug, as opposed to our heuristics being
> > so painfully off.
> >
> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
>
> For the xfstest lockup, what matters is that the owner isn't running, since
> the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the
issue, as it includes the !owner->on_cpu stop condition to stop
spinning.
>
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index 06e2214..5e08705 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
> }
> rcu_read_unlock();
>
> - if (READ_ONCE(sem->owner))
> - return true; /* new owner, continue spinning */
> + owner = READ_ONCE(sem->owner);
> + if (owner && owner->on_cpu)
> + return true;
>
> /*
> * When the owner is not set, the lock could be free or
>
>
> Thanks,
> Ming Lei
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:07 ` Davidlohr Bueso
@ 2015-03-07 2:10 ` Ming Lei
2015-03-07 2:26 ` Davidlohr Bueso
2015-03-07 2:56 ` Jason Low
0 siblings, 2 replies; 44+ messages in thread
From: Ming Lei @ 2015-03-07 2:10 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
>> On Fri, 06 Mar 2015 14:15:37 -0800
>> Davidlohr Bueso <dave@stgolabs.net> wrote:
>>
>> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
>> > > In owner_running() there are 2 conditions that would make it return
>> > > false: if the owner changed or if the owner is not running. However,
>> > > that patch continues spinning if there is a "new owner" but it does not
>> > > take into account that we may want to stop spinning if the owner is not
>> > > running (due to getting rescheduled).
>> >
>> > So you're rationale is that we're missing this need_resched:
>> >
>> > while (owner_running(sem, owner)) {
>> > /* abort spinning when need_resched */
>> > if (need_resched()) {
>> > rcu_read_unlock();
>> > return false;
>> > }
>> > }
>> >
>> > Because the owner_running() would return false, right? Yeah that makes
>> > sense, as missing a resched is a bug, as opposed to our heuristics being
>> > so painfully off.
>> >
>> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
>>
>> For the xfstest lockup, what matters is that the owner isn't running, since
>> the following simple change does fix the issue:
>
> I much prefer Jason's approach, which should also take care of the
> issue, as it includes the !owner->on_cpu stop condition to stop
> spinning.
But the check on owner->on_cpu should be moved outside the loop
because new owner can be scheduled out too, right?
>>
>> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
>> index 06e2214..5e08705 100644
>> --- a/kernel/locking/rwsem-xadd.c
>> +++ b/kernel/locking/rwsem-xadd.c
>> @@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
>> }
>> rcu_read_unlock();
>>
>> - if (READ_ONCE(sem->owner))
>> - return true; /* new owner, continue spinning */
>> + owner = READ_ONCE(sem->owner);
>> + if (owner && owner->on_cpu)
>> + return true;
>>
>> /*
>> * When the owner is not set, the lock could be free or
>>
>>
>> Thanks,
>> Ming Lei
>
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:10 ` Ming Lei
@ 2015-03-07 2:26 ` Davidlohr Bueso
2015-03-07 2:29 ` Davidlohr Bueso
2015-03-07 2:56 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 2:26 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> >> On Fri, 06 Mar 2015 14:15:37 -0800
> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
> >>
> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >> > > In owner_running() there are 2 conditions that would make it return
> >> > > false: if the owner changed or if the owner is not running. However,
> >> > > that patch continues spinning if there is a "new owner" but it does not
> >> > > take into account that we may want to stop spinning if the owner is not
> >> > > running (due to getting rescheduled).
> >> >
> >> > So you're rationale is that we're missing this need_resched:
> >> >
> >> > while (owner_running(sem, owner)) {
> >> > /* abort spinning when need_resched */
> >> > if (need_resched()) {
> >> > rcu_read_unlock();
> >> > return false;
> >> > }
> >> > }
> >> >
> >> > Because the owner_running() would return false, right? Yeah that makes
> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
> >> > so painfully off.
> >> >
> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
> >>
> >> For the xfstest lockup, what matters is that the owner isn't running, since
> >> the following simple change does fix the issue:
> >
> > I much prefer Jason's approach, which should also take care of the
> > issue, as it includes the !owner->on_cpu stop condition to stop
> > spinning.
>
> But the check on owner->on_cpu should be moved outside the loop
> because new owner can be scheduled out too, right?
That's not what this is about. New lock _owners_ need to worry about
burning cycles trying to acquire the lock ;)
> >>
> >> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> >> index 06e2214..5e08705 100644
> >> --- a/kernel/locking/rwsem-xadd.c
> >> +++ b/kernel/locking/rwsem-xadd.c
> >> @@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
> >> }
> >> rcu_read_unlock();
> >>
> >> - if (READ_ONCE(sem->owner))
> >> - return true; /* new owner, continue spinning */
> >> + owner = READ_ONCE(sem->owner);
> >> + if (owner && owner->on_cpu)
> >> + return true;
So if I'm understanding this right, your patch works because you add
another on_cpu check and at this point we could very well have
sem->owner == owner -- such that owner_running return false for the same
reason in the first place! So Jason's patch takes on the issue directly
by never allowing ups to reach this point.
Thanks,
Davidlohr
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:26 ` Davidlohr Bueso
@ 2015-03-07 2:29 ` Davidlohr Bueso
2015-03-07 2:55 ` Ming Lei
0 siblings, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 2:29 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Fri, 2015-03-06 at 18:26 -0800, Davidlohr Bueso wrote:
> That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:29 ` Davidlohr Bueso
@ 2015-03-07 2:55 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
0 siblings, 1 reply; 44+ messages in thread
From: Ming Lei @ 2015-03-07 2:55 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 10:29 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> On Fri, 2015-03-06 at 18:26 -0800, Davidlohr Bueso wrote:
>> That's not what this is about. New lock _owners_ need to worry about
> ^^^ make that "need not"
Sorry, could you explain a bit why new owner can't be scheduled
out(on_cpu becomes zero)? If that is possible, it still can cause
soft lockup like current problem.
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:10 ` Ming Lei
2015-03-07 2:26 ` Davidlohr Bueso
@ 2015-03-07 2:56 ` Jason Low
2015-03-07 3:08 ` Ming Lei
1 sibling, 1 reply; 44+ messages in thread
From: Jason Low @ 2015-03-07 2:56 UTC (permalink / raw)
To: Ming Lei
Cc: Davidlohr Bueso, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, jason.low2
On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> >> On Fri, 06 Mar 2015 14:15:37 -0800
> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
> >>
> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >> > > In owner_running() there are 2 conditions that would make it return
> >> > > false: if the owner changed or if the owner is not running. However,
> >> > > that patch continues spinning if there is a "new owner" but it does not
> >> > > take into account that we may want to stop spinning if the owner is not
> >> > > running (due to getting rescheduled).
> >> >
> >> > So you're rationale is that we're missing this need_resched:
> >> >
> >> > while (owner_running(sem, owner)) {
> >> > /* abort spinning when need_resched */
> >> > if (need_resched()) {
> >> > rcu_read_unlock();
> >> > return false;
> >> > }
> >> > }
> >> >
> >> > Because the owner_running() would return false, right? Yeah that makes
> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
> >> > so painfully off.
> >> >
> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
> >>
> >> For the xfstest lockup, what matters is that the owner isn't running, since
> >> the following simple change does fix the issue:
> >
> > I much prefer Jason's approach, which should also take care of the
> > issue, as it includes the !owner->on_cpu stop condition to stop
> > spinning.
>
> But the check on owner->on_cpu should be moved outside the loop
> because new owner can be scheduled out too, right?
We should keep the owner->on_cpu check inside the loop, otherwise we
could continue spinning if the owner is not running.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:56 ` Jason Low
@ 2015-03-07 3:08 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:17 ` Jason Low
0 siblings, 2 replies; 44+ messages in thread
From: Ming Lei @ 2015-03-07 3:08 UTC (permalink / raw)
To: Jason Low
Cc: Davidlohr Bueso, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 10:56 AM, Jason Low <jason.low2@hp.com> wrote:
> On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
>> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
>> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
>> >> On Fri, 06 Mar 2015 14:15:37 -0800
>> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
>> >>
>> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
>> >> > > In owner_running() there are 2 conditions that would make it return
>> >> > > false: if the owner changed or if the owner is not running. However,
>> >> > > that patch continues spinning if there is a "new owner" but it does not
>> >> > > take into account that we may want to stop spinning if the owner is not
>> >> > > running (due to getting rescheduled).
>> >> >
>> >> > So you're rationale is that we're missing this need_resched:
>> >> >
>> >> > while (owner_running(sem, owner)) {
>> >> > /* abort spinning when need_resched */
>> >> > if (need_resched()) {
>> >> > rcu_read_unlock();
>> >> > return false;
>> >> > }
>> >> > }
>> >> >
>> >> > Because the owner_running() would return false, right? Yeah that makes
>> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
>> >> > so painfully off.
>> >> >
>> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
>> >>
>> >> For the xfstest lockup, what matters is that the owner isn't running, since
>> >> the following simple change does fix the issue:
>> >
>> > I much prefer Jason's approach, which should also take care of the
>> > issue, as it includes the !owner->on_cpu stop condition to stop
>> > spinning.
>>
>> But the check on owner->on_cpu should be moved outside the loop
>> because new owner can be scheduled out too, right?
>
> We should keep the owner->on_cpu check inside the loop, otherwise we
> could continue spinning if the owner is not running.
So how about checking in this way outside the loop for avoiding the spin?
if (owner)
return owner->on_cpu;
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 2:55 ` Ming Lei
@ 2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:19 ` Ming Lei
0 siblings, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 3:10 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, 2015-03-07 at 10:55 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 10:29 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> > On Fri, 2015-03-06 at 18:26 -0800, Davidlohr Bueso wrote:
> >> That's not what this is about. New lock _owners_ need to worry about
> > ^^^ make that "need not"
>
> Sorry, could you explain a bit why new owner can't be scheduled
> out(on_cpu becomes zero)? If that is possible, it still can cause
> soft lockup like current problem.
Oh its not that it can't be scheduled out. The point is we don't care
what happens with the lock owner itself (new or not). We care about, and
the point of this discussion, how _other_ threads handle themselves when
trying to take that lock (a lock having an owner implies the lock is not
free, of course). So if a lock owner gets scheduled out... so what?
That's already taken into account by spinners.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:08 ` Ming Lei
@ 2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:17 ` Jason Low
1 sibling, 0 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 3:10 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, 2015-03-07 at 11:08 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 10:56 AM, Jason Low <jason.low2@hp.com> wrote:
> > On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
> >> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> >> >> On Fri, 06 Mar 2015 14:15:37 -0800
> >> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> >>
> >> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >> >> > > In owner_running() there are 2 conditions that would make it return
> >> >> > > false: if the owner changed or if the owner is not running. However,
> >> >> > > that patch continues spinning if there is a "new owner" but it does not
> >> >> > > take into account that we may want to stop spinning if the owner is not
> >> >> > > running (due to getting rescheduled).
> >> >> >
> >> >> > So you're rationale is that we're missing this need_resched:
> >> >> >
> >> >> > while (owner_running(sem, owner)) {
> >> >> > /* abort spinning when need_resched */
> >> >> > if (need_resched()) {
> >> >> > rcu_read_unlock();
> >> >> > return false;
> >> >> > }
> >> >> > }
> >> >> >
> >> >> > Because the owner_running() would return false, right? Yeah that makes
> >> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
> >> >> > so painfully off.
> >> >> >
> >> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
> >> >>
> >> >> For the xfstest lockup, what matters is that the owner isn't running, since
> >> >> the following simple change does fix the issue:
> >> >
> >> > I much prefer Jason's approach, which should also take care of the
> >> > issue, as it includes the !owner->on_cpu stop condition to stop
> >> > spinning.
> >>
> >> But the check on owner->on_cpu should be moved outside the loop
> >> because new owner can be scheduled out too, right?
> >
> > We should keep the owner->on_cpu check inside the loop, otherwise we
> > could continue spinning if the owner is not running.
>
> So how about checking in this way outside the loop for avoiding the spin?
>
> if (owner)
> return owner->on_cpu;
Ming are you reading the thread?? Have you at least tried jason's
patch?? *sigh*
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:08 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
@ 2015-03-07 3:17 ` Jason Low
2015-03-07 3:39 ` Ming Lei
1 sibling, 1 reply; 44+ messages in thread
From: Jason Low @ 2015-03-07 3:17 UTC (permalink / raw)
To: Ming Lei
Cc: Davidlohr Bueso, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, jason.low2
On Sat, 2015-03-07 at 11:08 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 10:56 AM, Jason Low <jason.low2@hp.com> wrote:
> > On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
> >> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> >> >> On Fri, 06 Mar 2015 14:15:37 -0800
> >> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> >>
> >> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >> >> > > In owner_running() there are 2 conditions that would make it return
> >> >> > > false: if the owner changed or if the owner is not running. However,
> >> >> > > that patch continues spinning if there is a "new owner" but it does not
> >> >> > > take into account that we may want to stop spinning if the owner is not
> >> >> > > running (due to getting rescheduled).
> >> >> >
> >> >> > So you're rationale is that we're missing this need_resched:
> >> >> >
> >> >> > while (owner_running(sem, owner)) {
> >> >> > /* abort spinning when need_resched */
> >> >> > if (need_resched()) {
> >> >> > rcu_read_unlock();
> >> >> > return false;
> >> >> > }
> >> >> > }
> >> >> >
> >> >> > Because the owner_running() would return false, right? Yeah that makes
> >> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
> >> >> > so painfully off.
> >> >> >
> >> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
> >> >>
> >> >> For the xfstest lockup, what matters is that the owner isn't running, since
> >> >> the following simple change does fix the issue:
> >> >
> >> > I much prefer Jason's approach, which should also take care of the
> >> > issue, as it includes the !owner->on_cpu stop condition to stop
> >> > spinning.
> >>
> >> But the check on owner->on_cpu should be moved outside the loop
> >> because new owner can be scheduled out too, right?
> >
> > We should keep the owner->on_cpu check inside the loop, otherwise we
> > could continue spinning if the owner is not running.
>
> So how about checking in this way outside the loop for avoiding the spin?
>
> if (owner)
> return owner->on_cpu;
So these owner->on_cpu checks outside of the loop "fixes" the issue as
well, but I don't see the benefit of needing to guess why we break out
of the spin loop (which may make things less readable) and checking
owner->on_cpu duplicate times when one check is enough.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:10 ` Davidlohr Bueso
@ 2015-03-07 3:19 ` Ming Lei
2015-03-07 3:41 ` Davidlohr Bueso
0 siblings, 1 reply; 44+ messages in thread
From: Ming Lei @ 2015-03-07 3:19 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 11:10 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> On Sat, 2015-03-07 at 10:55 +0800, Ming Lei wrote:
>> On Sat, Mar 7, 2015 at 10:29 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
>> > On Fri, 2015-03-06 at 18:26 -0800, Davidlohr Bueso wrote:
>> >> That's not what this is about. New lock _owners_ need to worry about
>> > ^^^ make that "need not"
>>
>> Sorry, could you explain a bit why new owner can't be scheduled
>> out(on_cpu becomes zero)? If that is possible, it still can cause
>> soft lockup like current problem.
>
> Oh its not that it can't be scheduled out. The point is we don't care
> what happens with the lock owner itself (new or not). We care about, and
> the point of this discussion, how _other_ threads handle themselves when
> trying to take that lock (a lock having an owner implies the lock is not
> free, of course). So if a lock owner gets scheduled out... so what?
> That's already taken into account by spinners.
Not exactly, current problem is just in spinner because it
ignores scheduled out owner and continues to spin, then
cause lockup, isn't it?
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:17 ` Jason Low
@ 2015-03-07 3:39 ` Ming Lei
2015-03-07 3:53 ` Jason Low
0 siblings, 1 reply; 44+ messages in thread
From: Ming Lei @ 2015-03-07 3:39 UTC (permalink / raw)
To: Jason Low
Cc: Davidlohr Bueso, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 11:17 AM, Jason Low <jason.low2@hp.com> wrote:
> On Sat, 2015-03-07 at 11:08 +0800, Ming Lei wrote:
>> On Sat, Mar 7, 2015 at 10:56 AM, Jason Low <jason.low2@hp.com> wrote:
>> > On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
>> >> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
>> >> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
>> >> >> On Fri, 06 Mar 2015 14:15:37 -0800
>> >> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
>> >> >>
>> >> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
>> >> >> > > In owner_running() there are 2 conditions that would make it return
>> >> >> > > false: if the owner changed or if the owner is not running. However,
>> >> >> > > that patch continues spinning if there is a "new owner" but it does not
>> >> >> > > take into account that we may want to stop spinning if the owner is not
>> >> >> > > running (due to getting rescheduled).
>> >> >> >
>> >> >> > So you're rationale is that we're missing this need_resched:
>> >> >> >
>> >> >> > while (owner_running(sem, owner)) {
>> >> >> > /* abort spinning when need_resched */
>> >> >> > if (need_resched()) {
>> >> >> > rcu_read_unlock();
>> >> >> > return false;
>> >> >> > }
>> >> >> > }
>> >> >> >
>> >> >> > Because the owner_running() would return false, right? Yeah that makes
>> >> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
>> >> >> > so painfully off.
>> >> >> >
>> >> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
>> >> >>
>> >> >> For the xfstest lockup, what matters is that the owner isn't running, since
>> >> >> the following simple change does fix the issue:
>> >> >
>> >> > I much prefer Jason's approach, which should also take care of the
>> >> > issue, as it includes the !owner->on_cpu stop condition to stop
>> >> > spinning.
>> >>
>> >> But the check on owner->on_cpu should be moved outside the loop
>> >> because new owner can be scheduled out too, right?
>> >
>> > We should keep the owner->on_cpu check inside the loop, otherwise we
>> > could continue spinning if the owner is not running.
>>
>> So how about checking in this way outside the loop for avoiding the spin?
>>
>> if (owner)
>> return owner->on_cpu;
>
> So these owner->on_cpu checks outside of the loop "fixes" the issue as
> well, but I don't see the benefit of needing to guess why we break out
> of the spin loop (which may make things less readable) and checking
> owner->on_cpu duplicate times when one check is enough.
I mean moving the check on owner->on_cpu outside loop, so there is
only one check for both new and old owner. If it is inside loop,
the check is only on old owner.
That is correct to keep it inside loop if you guys are sure new
owner can't be scheduled out, but better to add comment why
it can't, looks no one explained yet.
Thanks,
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:19 ` Ming Lei
@ 2015-03-07 3:41 ` Davidlohr Bueso
0 siblings, 0 replies; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 3:41 UTC (permalink / raw)
To: Ming Lei
Cc: Jason Low, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones
On Sat, 2015-03-07 at 11:19 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 11:10 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> > On Sat, 2015-03-07 at 10:55 +0800, Ming Lei wrote:
> >> On Sat, Mar 7, 2015 at 10:29 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> > On Fri, 2015-03-06 at 18:26 -0800, Davidlohr Bueso wrote:
> >> >> That's not what this is about. New lock _owners_ need to worry about
> >> > ^^^ make that "need not"
> >>
> >> Sorry, could you explain a bit why new owner can't be scheduled
> >> out(on_cpu becomes zero)? If that is possible, it still can cause
> >> soft lockup like current problem.
> >
> > Oh its not that it can't be scheduled out. The point is we don't care
> > what happens with the lock owner itself (new or not). We care about, and
> > the point of this discussion, how _other_ threads handle themselves when
> > trying to take that lock (a lock having an owner implies the lock is not
> > free, of course). So if a lock owner gets scheduled out... so what?
> > That's already taken into account by spinners.
>
> Not exactly, current problem is just in spinner because it
> ignores scheduled out owner and continues to spin, then
> cause lockup, isn't it?
Exactly my point, Ming. It's the _spinner_ that has the problem, hence
the fix in the part of the code that must decide just that. By the time
we're doing this:
if (READ_ONCE(sem->owner))
return true; /* new owner, continue spinning */
We need to have already taken into account the owner->on_cpu situation.
We fix spinners, not lock owners.
I'm really running out of ways to explain this, and you are going in
circles, which is getting annoying given that you haven't even tried the
other patch.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 3:39 ` Ming Lei
@ 2015-03-07 3:53 ` Jason Low
0 siblings, 0 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 3:53 UTC (permalink / raw)
To: Ming Lei
Cc: Davidlohr Bueso, Linus Torvalds, Ingo Molnar, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, jason.low2
On Sat, 2015-03-07 at 11:39 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 11:17 AM, Jason Low <jason.low2@hp.com> wrote:
> > On Sat, 2015-03-07 at 11:08 +0800, Ming Lei wrote:
> >> On Sat, Mar 7, 2015 at 10:56 AM, Jason Low <jason.low2@hp.com> wrote:
> >> > On Sat, 2015-03-07 at 10:10 +0800, Ming Lei wrote:
> >> >> On Sat, Mar 7, 2015 at 10:07 AM, Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> >> > On Sat, 2015-03-07 at 09:55 +0800, Ming Lei wrote:
> >> >> >> On Fri, 06 Mar 2015 14:15:37 -0800
> >> >> >> Davidlohr Bueso <dave@stgolabs.net> wrote:
> >> >> >>
> >> >> >> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >> >> >> > > In owner_running() there are 2 conditions that would make it return
> >> >> >> > > false: if the owner changed or if the owner is not running. However,
> >> >> >> > > that patch continues spinning if there is a "new owner" but it does not
> >> >> >> > > take into account that we may want to stop spinning if the owner is not
> >> >> >> > > running (due to getting rescheduled).
> >> >> >> >
> >> >> >> > So you're rationale is that we're missing this need_resched:
> >> >> >> >
> >> >> >> > while (owner_running(sem, owner)) {
> >> >> >> > /* abort spinning when need_resched */
> >> >> >> > if (need_resched()) {
> >> >> >> > rcu_read_unlock();
> >> >> >> > return false;
> >> >> >> > }
> >> >> >> > }
> >> >> >> >
> >> >> >> > Because the owner_running() would return false, right? Yeah that makes
> >> >> >> > sense, as missing a resched is a bug, as opposed to our heuristics being
> >> >> >> > so painfully off.
> >> >> >> >
> >> >> >> > Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
> >> >> >>
> >> >> >> For the xfstest lockup, what matters is that the owner isn't running, since
> >> >> >> the following simple change does fix the issue:
> >> >> >
> >> >> > I much prefer Jason's approach, which should also take care of the
> >> >> > issue, as it includes the !owner->on_cpu stop condition to stop
> >> >> > spinning.
> >> >>
> >> >> But the check on owner->on_cpu should be moved outside the loop
> >> >> because new owner can be scheduled out too, right?
> >> >
> >> > We should keep the owner->on_cpu check inside the loop, otherwise we
> >> > could continue spinning if the owner is not running.
> >>
> >> So how about checking in this way outside the loop for avoiding the spin?
> >>
> >> if (owner)
> >> return owner->on_cpu;
> >
> > So these owner->on_cpu checks outside of the loop "fixes" the issue as
> > well, but I don't see the benefit of needing to guess why we break out
> > of the spin loop (which may make things less readable) and checking
> > owner->on_cpu duplicate times when one check is enough.
>
> I mean moving the check on owner->on_cpu outside loop, so there is
> only one check for both new and old owner. If it is inside loop,
> the check is only on old owner.
>
> That is correct to keep it inside loop if you guys are sure new
> owner can't be scheduled out, but better to add comment why
> it can't, looks no one explained yet.
The new owner can get rescheduled.
And if there's a new owner, then the spinner goes to
rwsem_spin_on_owner() again and checks the new owner's on_cpu.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-06 21:12 ` Jason Low
2015-03-06 21:24 ` Linus Torvalds
2015-03-06 22:15 ` Davidlohr Bueso
@ 2015-03-07 4:31 ` Jason Low
2015-03-07 4:44 ` Davidlohr Bueso
2015-03-07 5:54 ` Ming Lei
2 siblings, 2 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 4:31 UTC (permalink / raw)
To: Linus Torvalds
Cc: Davidlohr Bueso, Ingo Molnar, tim.c.chen, paulmck, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, Ming Lei, jason.low2
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Just in case, here's the updated patch which addresses Linus's comments
and with a changelog.
Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving
lock spinners), though I still haven't seen full confirmation that it
addresses all of the lockup reports.
------
Subject: [PATCH] rwsem: Avoid spinning when owner is not running
Fixes tip commmit b3fd4f03ca0b (locking/rwsem: Avoid deceiving lock spinners).
When doing optimistic spinning in rwsem, threads should stop spinning when
the lock owner is not running. While a thread is spinning on owner, if
the owner reschedules, owner->on_cpu returns false and we stop spinning.
However, commit b3fd4f03ca0b essentially caused the check to get ignored
because when we break out of the spin loop due to !on_cpu, we continue
spinning if sem->owner != NULL.
This patch fixes this by making sure we stop spinning if the owner is not
running. Furthermore, just like with mutexes, refactor the code such that
we don't have separate checks for owner_running(). This makes it more
straightforward in terms of why we exit the spin on owner loop and we
would also avoid needing to "guess" why we broke out of the loop to make
this more readable.
Cc: Ming Lei <ming.lei@canonical.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Signed-off-by: Jason Low <jason.low2@hp.com>
---
kernel/locking/rwsem-xadd.c | 31 +++++++++++--------------------
1 files changed, 11 insertions(+), 20 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 06e2214..3417d01 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -324,32 +324,23 @@ done:
return ret;
}
-static inline bool owner_running(struct rw_semaphore *sem,
- struct task_struct *owner)
-{
- if (sem->owner != owner)
- return false;
-
- /*
- * Ensure we emit the owner->on_cpu, dereference _after_ checking
- * sem->owner still matches owner, if that fails, owner might
- * point to free()d memory, if it still matches, the rcu_read_lock()
- * ensures the memory stays valid.
- */
- barrier();
-
- return owner->on_cpu;
-}
-
static noinline
bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
{
long count;
rcu_read_lock();
- while (owner_running(sem, owner)) {
- /* abort spinning when need_resched */
- if (need_resched()) {
+ while (sem->owner == owner) {
+ /*
+ * Ensure we emit the owner->on_cpu, dereference _after_
+ * checking sem->owner still matches owner, if that fails,
+ * owner might point to free()d memory, if it still matches,
+ * the rcu_read_lock() ensures the memory stays valid.
+ */
+ barrier();
+
+ /* abort spinning when need_resched or owner is not running */
+ if (!owner->on_cpu || need_resched()) {
rcu_read_unlock();
return false;
}
--
1.7.2.5
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 4:31 ` Jason Low
@ 2015-03-07 4:44 ` Davidlohr Bueso
2015-03-07 6:45 ` Jason Low
2015-03-07 5:54 ` Ming Lei
1 sibling, 1 reply; 44+ messages in thread
From: Davidlohr Bueso @ 2015-03-07 4:44 UTC (permalink / raw)
To: Jason Low
Cc: Linus Torvalds, Ingo Molnar, tim.c.chen, paulmck, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, Ming Lei
On Fri, 2015-03-06 at 20:31 -0800, Jason Low wrote:
> On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
>
> Just in case, here's the updated patch which addresses Linus's comments
> and with a changelog.
>
> Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving
> lock spinners), though I still haven't seen full confirmation that it
> addresses all of the lockup reports.
>
> ------
> Subject: [PATCH] rwsem: Avoid spinning when owner is not running
>
> Fixes tip commmit b3fd4f03ca0b (locking/rwsem: Avoid deceiving lock spinners).
>
> When doing optimistic spinning in rwsem, threads should stop spinning when
> the lock owner is not running. While a thread is spinning on owner, if
> the owner reschedules, owner->on_cpu returns false and we stop spinning.
>
> However, commit b3fd4f03ca0b essentially caused the check to get ignored
> because when we break out of the spin loop due to !on_cpu, we continue
> spinning if sem->owner != NULL.
I would mention the actual effects of the bug, either just a "lockup"
and/or a fragment of the trace. But ultimately this comes down to
missing a need_resched() condition.
>
> This patch fixes this by making sure we stop spinning if the owner is not
> running. Furthermore, just like with mutexes, refactor the code such that
> we don't have separate checks for owner_running(). This makes it more
> straightforward in terms of why we exit the spin on owner loop and we
> would also avoid needing to "guess" why we broke out of the loop to make
> this more readable.
>
> Cc: Ming Lei <ming.lei@canonical.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
Acked-by: Davidlohr Bueso <dave@stgolabs.net>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 4:31 ` Jason Low
2015-03-07 4:44 ` Davidlohr Bueso
@ 2015-03-07 5:54 ` Ming Lei
2015-03-07 6:57 ` Jason Low
1 sibling, 1 reply; 44+ messages in thread
From: Ming Lei @ 2015-03-07 5:54 UTC (permalink / raw)
To: Jason Low
Cc: Linus Torvalds, Davidlohr Bueso, Ingo Molnar, Tim Chen,
Paul McKenney, Sasha Levin, Peter Zijlstra, LKML, Dave Jones
On Sat, Mar 7, 2015 at 12:31 PM, Jason Low <jason.low2@hp.com> wrote:
> On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
>
> Just in case, here's the updated patch which addresses Linus's comments
> and with a changelog.
>
> Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving
> lock spinners), though I still haven't seen full confirmation that it
> addresses all of the lockup reports.
>
> ------
> Subject: [PATCH] rwsem: Avoid spinning when owner is not running
>
> Fixes tip commmit b3fd4f03ca0b (locking/rwsem: Avoid deceiving lock spinners).
>
> When doing optimistic spinning in rwsem, threads should stop spinning when
> the lock owner is not running. While a thread is spinning on owner, if
> the owner reschedules, owner->on_cpu returns false and we stop spinning.
>
> However, commit b3fd4f03ca0b essentially caused the check to get ignored
> because when we break out of the spin loop due to !on_cpu, we continue
> spinning if sem->owner != NULL.
>
> This patch fixes this by making sure we stop spinning if the owner is not
> running. Furthermore, just like with mutexes, refactor the code such that
> we don't have separate checks for owner_running(). This makes it more
> straightforward in terms of why we exit the spin on owner loop and we
> would also avoid needing to "guess" why we broke out of the loop to make
> this more readable.
>
> Cc: Ming Lei <ming.lei@canonical.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Signed-off-by: Jason Low <jason.low2@hp.com>
Reported-and-tested-by: Ming Lei <ming.lei@canonical.com>
> ---
> kernel/locking/rwsem-xadd.c | 31 +++++++++++--------------------
> 1 files changed, 11 insertions(+), 20 deletions(-)
>
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index 06e2214..3417d01 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -324,32 +324,23 @@ done:
> return ret;
> }
>
> -static inline bool owner_running(struct rw_semaphore *sem,
> - struct task_struct *owner)
> -{
> - if (sem->owner != owner)
> - return false;
> -
> - /*
> - * Ensure we emit the owner->on_cpu, dereference _after_ checking
> - * sem->owner still matches owner, if that fails, owner might
> - * point to free()d memory, if it still matches, the rcu_read_lock()
> - * ensures the memory stays valid.
> - */
> - barrier();
> -
> - return owner->on_cpu;
> -}
> -
> static noinline
> bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
> {
> long count;
>
> rcu_read_lock();
> - while (owner_running(sem, owner)) {
> - /* abort spinning when need_resched */
> - if (need_resched()) {
> + while (sem->owner == owner) {
> + /*
> + * Ensure we emit the owner->on_cpu, dereference _after_
> + * checking sem->owner still matches owner, if that fails,
> + * owner might point to free()d memory, if it still matches,
> + * the rcu_read_lock() ensures the memory stays valid.
> + */
> + barrier();
> +
> + /* abort spinning when need_resched or owner is not running */
> + if (!owner->on_cpu || need_resched()) {
BTW, could the need_resched() be handled in loop of
rwsem_optimistic_spin() directly? Then code may get
simplified a bit.
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 4:44 ` Davidlohr Bueso
@ 2015-03-07 6:45 ` Jason Low
0 siblings, 0 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 6:45 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: Linus Torvalds, Ingo Molnar, tim.c.chen, paulmck, Sasha Levin,
Peter Zijlstra, LKML, Dave Jones, Ming Lei, jason.low2
On Fri, 2015-03-06 at 20:44 -0800, Davidlohr Bueso wrote:
> On Fri, 2015-03-06 at 20:31 -0800, Jason Low wrote:
> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> >
> > Just in case, here's the updated patch which addresses Linus's comments
> > and with a changelog.
> >
> > Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving
> > lock spinners), though I still haven't seen full confirmation that it
> > addresses all of the lockup reports.
> >
> > ------
> > Subject: [PATCH] rwsem: Avoid spinning when owner is not running
> >
> > Fixes tip commmit b3fd4f03ca0b (locking/rwsem: Avoid deceiving lock spinners).
> >
> > When doing optimistic spinning in rwsem, threads should stop spinning when
> > the lock owner is not running. While a thread is spinning on owner, if
> > the owner reschedules, owner->on_cpu returns false and we stop spinning.
> >
> > However, commit b3fd4f03ca0b essentially caused the check to get ignored
> > because when we break out of the spin loop due to !on_cpu, we continue
> > spinning if sem->owner != NULL.
>
> I would mention the actual effects of the bug, either just a "lockup"
> and/or a fragment of the trace.
Right, we should mention about the lockup in the changelog.
> > Cc: Ming Lei <ming.lei@canonical.com>
> > Cc: Davidlohr Bueso <dave@stgolabs.net>
>
> Acked-by: Davidlohr Bueso <dave@stgolabs.net>
Thanks!
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: softlockups in multi_cpu_stop
2015-03-07 5:54 ` Ming Lei
@ 2015-03-07 6:57 ` Jason Low
0 siblings, 0 replies; 44+ messages in thread
From: Jason Low @ 2015-03-07 6:57 UTC (permalink / raw)
To: Ming Lei
Cc: Linus Torvalds, Davidlohr Bueso, Ingo Molnar, Tim Chen,
Paul McKenney, Sasha Levin, Peter Zijlstra, LKML, Dave Jones,
jason.low2
On Sat, 2015-03-07 at 13:54 +0800, Ming Lei wrote:
> On Sat, Mar 7, 2015 at 12:31 PM, Jason Low <jason.low2@hp.com> wrote:
> > On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
> > Cc: Ming Lei <ming.lei@canonical.com>
> > Cc: Davidlohr Bueso <dave@stgolabs.net>
> > Signed-off-by: Jason Low <jason.low2@hp.com>
>
> Reported-and-tested-by: Ming Lei <ming.lei@canonical.com>
Thanks!
> > static noinline
> > bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
> > {
> > long count;
> >
> > rcu_read_lock();
> > - while (owner_running(sem, owner)) {
> > - /* abort spinning when need_resched */
> > - if (need_resched()) {
> > + while (sem->owner == owner) {
> > + /*
> > + * Ensure we emit the owner->on_cpu, dereference _after_
> > + * checking sem->owner still matches owner, if that fails,
> > + * owner might point to free()d memory, if it still matches,
> > + * the rcu_read_lock() ensures the memory stays valid.
> > + */
> > + barrier();
> > +
> > + /* abort spinning when need_resched or owner is not running */
> > + if (!owner->on_cpu || need_resched()) {
>
> BTW, could the need_resched() be handled in loop of
> rwsem_optimistic_spin() directly? Then code may get
> simplified a bit.
We still need the need_resched() check here, since if the thread needs
to reschedule, it should immediately stop spinning for the lock.
Otherwise, it could potentially spin for a long time before it checks
for it needs to reschedule.
^ permalink raw reply [flat|nested] 44+ messages in thread
end of thread, other threads:[~2015-03-07 6:57 UTC | newest]
Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-03-02 7:45 sched: softlockups in multi_cpu_stop Sasha Levin
[not found] ` <CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com>
2015-03-04 5:44 ` Rafael David Tinoco
2015-03-06 11:27 ` Sasha Levin
2015-03-06 12:32 ` Ingo Molnar
2015-03-06 14:34 ` Rafael David Tinoco
2015-03-06 14:45 ` Sasha Levin
2015-03-06 15:46 ` Sasha Levin
2015-03-06 17:19 ` Davidlohr Bueso
2015-03-06 18:02 ` Sasha Levin
2015-03-06 21:59 ` Sasha Levin
2015-03-06 18:57 ` Jason Low
2015-03-06 19:05 ` Linus Torvalds
2015-03-06 19:20 ` Davidlohr Bueso
2015-03-06 19:32 ` Linus Torvalds
2015-03-06 19:45 ` Davidlohr Bueso
2015-03-06 19:55 ` Davidlohr Bueso
2015-03-06 20:00 ` Davidlohr Bueso
2015-03-06 21:42 ` Linus Torvalds
2015-03-06 19:29 ` Jason Low
2015-03-06 21:12 ` Jason Low
2015-03-06 21:24 ` Linus Torvalds
2015-03-07 1:53 ` Jason Low
2015-03-06 22:15 ` Davidlohr Bueso
2015-03-07 1:55 ` Ming Lei
2015-03-07 2:07 ` Davidlohr Bueso
2015-03-07 2:10 ` Ming Lei
2015-03-07 2:26 ` Davidlohr Bueso
2015-03-07 2:29 ` Davidlohr Bueso
2015-03-07 2:55 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:19 ` Ming Lei
2015-03-07 3:41 ` Davidlohr Bueso
2015-03-07 2:56 ` Jason Low
2015-03-07 3:08 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:17 ` Jason Low
2015-03-07 3:39 ` Ming Lei
2015-03-07 3:53 ` Jason Low
2015-03-07 1:58 ` Jason Low
2015-03-07 4:31 ` Jason Low
2015-03-07 4:44 ` Davidlohr Bueso
2015-03-07 6:45 ` Jason Low
2015-03-07 5:54 ` Ming Lei
2015-03-07 6:57 ` Jason Low
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.