* possible deadlock in aio_poll @ 2018-09-10 7:41 syzbot 2018-09-10 16:53 ` Christoph Hellwig 2018-10-27 6:16 ` syzbot 0 siblings, 2 replies; 7+ messages in thread From: syzbot @ 2018-09-10 7:41 UTC (permalink / raw) To: bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, viro Hello, syzbot found the following crash on: HEAD commit: f8f65382c98a Merge tag 'for-linus' of git://git.kernel.org.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=1587e266400000 kernel config: https://syzkaller.appspot.com/x/.config?x=8f59875069d721b6 dashboard link: https://syzkaller.appspot.com/bug?extid=5b1df0420c523b45a953 compiler: gcc (GCC) 8.0.1 20180413 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1753bdca400000 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+5b1df0420c523b45a953@syzkaller.appspotmail.com 8021q: adding VLAN 0 to HW filter on device team0 ===================================================== WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected 4.19.0-rc2+ #229 Not tainted ----------------------------------------------------- syz-executor2/9399 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: spin_lock include/linux/spinlock.h:329 [inline] 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: aio_poll+0x760/0x1420 fs/aio.c:1747 and this task is already holding: 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:354 [inline] 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 fs/aio.c:1746 which would create a new lock dependency: (&(&ctx->ctx_lock)->rlock){..-.} -> (&ctx->fd_wqh){+.+.} but this new dependency connects a SOFTIRQ-irq-safe lock: (&(&ctx->ctx_lock)->rlock){..-.} ... which became SOFTIRQ-irq-safe at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:284 [inline] percpu_ref_put include/linux/percpu-refcount.h:300 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x62c/0x820 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:236 [inline] rcu_do_batch kernel/rcu/tree.c:2576 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2880 [inline] __rcu_process_callbacks kernel/rcu/tree.c:2847 [inline] rcu_process_callbacks+0xf23/0x2670 kernel/rcu/tree.c:2864 __do_softirq+0x30b/0xad8 kernel/softirq.c:292 invoke_softirq kernel/softirq.c:372 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:412 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1056 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x3db/0x5b0 kernel/sched/idle.c:262 cpu_startup_entry+0x10c/0x120 kernel/sched/idle.c:368 start_secondary+0x523/0x750 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242 to a SOFTIRQ-irq-unsafe lock: (&ctx->fd_wqh){+.+.} ... which became SOFTIRQ-irq-unsafe at: ... lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 do_loop_readv_writev fs/read_write.c:700 [inline] do_iter_read+0x4a3/0x650 fs/read_write.c:924 vfs_readv+0x175/0x1c0 fs/read_write.c:986 do_readv+0x11a/0x310 fs/read_write.c:1019 __do_sys_readv fs/read_write.c:1106 [inline] __se_sys_readv fs/read_write.c:1103 [inline] __x64_sys_readv+0x75/0xb0 fs/read_write.c:1103 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&ctx->fd_wqh); local_irq_disable(); lock(&(&ctx->ctx_lock)->rlock); lock(&ctx->fd_wqh); <Interrupt> lock(&(&ctx->ctx_lock)->rlock); *** DEADLOCK *** 1 lock held by syz-executor2/9399: #0: 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:354 [inline] #0: 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 fs/aio.c:1746 the dependencies between SOFTIRQ-irq-safe lock and the holding lock: -> (&(&ctx->ctx_lock)->rlock){..-.} ops: 387 { IN-SOFTIRQ-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:284 [inline] percpu_ref_put include/linux/percpu-refcount.h:300 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x62c/0x820 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:236 [inline] rcu_do_batch kernel/rcu/tree.c:2576 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2880 [inline] __rcu_process_callbacks kernel/rcu/tree.c:2847 [inline] rcu_process_callbacks+0xf23/0x2670 kernel/rcu/tree.c:2864 __do_softirq+0x30b/0xad8 kernel/softirq.c:292 invoke_softirq kernel/softirq.c:372 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:412 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1056 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x3db/0x5b0 kernel/sched/idle.c:262 cpu_startup_entry+0x10c/0x120 kernel/sched/idle.c:368 start_secondary+0x523/0x750 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242 INITIAL USE at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:284 [inline] percpu_ref_put include/linux/percpu-refcount.h:300 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x62c/0x820 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:236 [inline] rcu_do_batch kernel/rcu/tree.c:2576 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2880 [inline] __rcu_process_callbacks kernel/rcu/tree.c:2847 [inline] rcu_process_callbacks+0xf23/0x2670 kernel/rcu/tree.c:2864 __do_softirq+0x30b/0xad8 kernel/softirq.c:292 invoke_softirq kernel/softirq.c:372 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:412 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1056 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x3db/0x5b0 kernel/sched/idle.c:262 cpu_startup_entry+0x10c/0x120 kernel/sched/idle.c:368 start_secondary+0x523/0x750 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242 } ... key at: [<ffffffff8b3dc960>] __key.50120+0x0/0x40 ... acquired at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xab8/0x1090 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b9/0x5d0 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: -> (&ctx->fd_wqh){+.+.} ops: 2209 { HARDIRQ-ON-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 do_loop_readv_writev fs/read_write.c:700 [inline] do_iter_read+0x4a3/0x650 fs/read_write.c:924 vfs_readv+0x175/0x1c0 fs/read_write.c:986 do_readv+0x11a/0x310 fs/read_write.c:1019 __do_sys_readv fs/read_write.c:1106 [inline] __se_sys_readv fs/read_write.c:1103 [inline] __x64_sys_readv+0x75/0xb0 fs/read_write.c:1103 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe SOFTIRQ-ON-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 do_loop_readv_writev fs/read_write.c:700 [inline] do_iter_read+0x4a3/0x650 fs/read_write.c:924 vfs_readv+0x175/0x1c0 fs/read_write.c:986 do_readv+0x11a/0x310 fs/read_write.c:1019 __do_sys_readv fs/read_write.c:1106 [inline] __se_sys_readv fs/read_write.c:1103 [inline] __x64_sys_readv+0x75/0xb0 fs/read_write.c:1103 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe INITIAL USE at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 do_loop_readv_writev fs/read_write.c:700 [inline] do_iter_read+0x4a3/0x650 fs/read_write.c:924 vfs_readv+0x175/0x1c0 fs/read_write.c:986 do_readv+0x11a/0x310 fs/read_write.c:1019 __do_sys_readv fs/read_write.c:1106 [inline] __se_sys_readv fs/read_write.c:1103 [inline] __x64_sys_readv+0x75/0xb0 fs/read_write.c:1103 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe } ... key at: [<ffffffff8b3dc6e0>] __key.43670+0x0/0x40 ... acquired at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xab8/0x1090 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b9/0x5d0 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe stack backtrace: CPU: 0 PID: 9399 Comm: syz-executor2 Not tainted 4.19.0-rc2+ #229 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113 print_bad_irq_dependency kernel/locking/lockdep.c:1569 [inline] check_usage.cold.58+0x6d5/0xad1 kernel/locking/lockdep.c:1601 check_irq_usage kernel/locking/lockdep.c:1657 [inline] check_prev_add_irq kernel/locking/lockdep_states.h:8 [inline] check_prev_add kernel/locking/lockdep.c:1867 [inline] check_prevs_add kernel/locking/lockdep.c:1975 [inline] validate_chain kernel/locking/lockdep.c:2416 [inline] __lock_acquire+0x2400/0x4ec0 kernel/locking/lockdep.c:3412 lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3901 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xab8/0x1090 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b9/0x5d0 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457099 Code: fd b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007fa4bd11bc78 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1 RAX: ffffffffffffffda RBX: 00007fa4bd11c6d4 RCX: 0000000000457099 RDX: 0000000020000b00 RSI: 0000000000000001 RDI: 00007fa4bd13e000 RBP: 00000000009301e0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 00000000004cd990 R14: 00000000004c40a7 R15: 0000000000000002 kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop1' (00000000fd2f09a6): kobject_uevent_env kobject: 'loop1' (00000000fd2f09a6): fill_kobj_path: path = '/devices/virtual/block/loop1' kobject: 'loop6' (00000000790743e9): kobject_uevent_env kobject: 'loop6' (00000000790743e9): fill_kobj_path: path = '/devices/virtual/block/loop6' kobject: 'loop7' (00000000b52cfd8a): kobject_uevent_env kobject: 'loop7' (00000000b52cfd8a): fill_kobj_path: path = '/devices/virtual/block/loop7' kobject: 'loop2' (000000002d051810): kobject_uevent_env kobject: 'loop2' (000000002d051810): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop5' (0000000044e01f3d): kobject_uevent_env kobject: 'loop5' (0000000044e01f3d): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop4' (000000003f6c580a): kobject_uevent_env kobject: 'loop4' (000000003f6c580a): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop3' (000000001d0a0601): kobject_uevent_env kobject: 'loop3' (000000001d0a0601): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop0' (000000005501af28): kobject_uevent_env kobject: 'loop0' (000000005501af28): fill_kobj_path: path = '/devices/virtual/block/loop0' --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. syzbot can test patches for this bug, for details see: https://goo.gl/tpsmEJ#testing-patches ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-10 7:41 possible deadlock in aio_poll syzbot @ 2018-09-10 16:53 ` Christoph Hellwig 2018-09-10 18:14 ` Miklos Szeredi 2018-10-17 23:55 ` Andrea Arcangeli 2018-10-27 6:16 ` syzbot 1 sibling, 2 replies; 7+ messages in thread From: Christoph Hellwig @ 2018-09-10 16:53 UTC (permalink / raw) To: syzbot Cc: bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, viro, Andrea Arcangeli, akpm On Mon, Sep 10, 2018 at 12:41:05AM -0700, syzbot wrote: > ===================================================== > WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected > 4.19.0-rc2+ #229 Not tainted > ----------------------------------------------------- > syz-executor2/9399 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: > 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: spin_lock > include/linux/spinlock.h:329 [inline] > 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: aio_poll+0x760/0x1420 > fs/aio.c:1747 > > and this task is already holding: > 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq > include/linux/spinlock.h:354 [inline] > 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 > fs/aio.c:1746 > which would create a new lock dependency: > (&(&ctx->ctx_lock)->rlock){..-.} -> (&ctx->fd_wqh){+.+.} ctx->fd_wqh seems to only exist in userfaultfd, which indeed seems to do strange open coded waitqueue locking, and seems to fail to disable irqs. Something like this should fix it: diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index bfa0ec69f924..356d2b8568c1 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1026,7 +1026,7 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait, struct userfaultfd_ctx *fork_nctx = NULL; /* always take the fd_wqh lock before the fault_pending_wqh lock */ - spin_lock(&ctx->fd_wqh.lock); + spin_lock_irq(&ctx->fd_wqh.lock); __add_wait_queue(&ctx->fd_wqh, &wait); for (;;) { set_current_state(TASK_INTERRUPTIBLE); @@ -1112,13 +1112,13 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait, ret = -EAGAIN; break; } - spin_unlock(&ctx->fd_wqh.lock); + spin_unlock_irq(&ctx->fd_wqh.lock); schedule(); - spin_lock(&ctx->fd_wqh.lock); + spin_lock_irq(&ctx->fd_wqh.lock); } __remove_wait_queue(&ctx->fd_wqh, &wait); __set_current_state(TASK_RUNNING); - spin_unlock(&ctx->fd_wqh.lock); + spin_unlock_irq(&ctx->fd_wqh.lock); if (!ret && msg->event == UFFD_EVENT_FORK) { ret = resolve_userfault_fork(ctx, fork_nctx, msg); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-10 16:53 ` Christoph Hellwig @ 2018-09-10 18:14 ` Miklos Szeredi 2018-09-11 6:33 ` Christoph Hellwig 2018-10-17 23:55 ` Andrea Arcangeli 1 sibling, 1 reply; 7+ messages in thread From: Miklos Szeredi @ 2018-09-10 18:14 UTC (permalink / raw) To: Christoph Hellwig Cc: syzbot, bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, Al Viro, Andrea Arcangeli, Andrew Morton On Mon, Sep 10, 2018 at 6:53 PM, Christoph Hellwig <hch@infradead.org> wrote: > On Mon, Sep 10, 2018 at 12:41:05AM -0700, syzbot wrote: >> ===================================================== >> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected >> 4.19.0-rc2+ #229 Not tainted >> ----------------------------------------------------- >> syz-executor2/9399 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: >> 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: spin_lock >> include/linux/spinlock.h:329 [inline] >> 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: aio_poll+0x760/0x1420 >> fs/aio.c:1747 >> >> and this task is already holding: >> 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq >> include/linux/spinlock.h:354 [inline] >> 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 >> fs/aio.c:1746 >> which would create a new lock dependency: >> (&(&ctx->ctx_lock)->rlock){..-.} -> (&ctx->fd_wqh){+.+.} > > ctx->fd_wqh seems to only exist in userfaultfd, which indeed seems > to do strange open coded waitqueue locking, and seems to fail to disable > irqs. Something like this should fix it: Why do pollable waitqueues need to disable interrupts generally? I don't see anything fundamental in the poll interface to force this requirement on users of that interface. Thanks, Miklos ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-10 18:14 ` Miklos Szeredi @ 2018-09-11 6:33 ` Christoph Hellwig 2018-09-11 7:20 ` Miklos Szeredi 0 siblings, 1 reply; 7+ messages in thread From: Christoph Hellwig @ 2018-09-11 6:33 UTC (permalink / raw) To: Miklos Szeredi Cc: Christoph Hellwig, syzbot, bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, Al Viro, Andrea Arcangeli, Andrew Morton On Mon, Sep 10, 2018 at 08:14:20PM +0200, Miklos Szeredi wrote: > Why do pollable waitqueues need to disable interrupts generally? Any waitqueue needs to disable interrupts for consistency. We always use spin_lock_irqsave in __wake_up_common_lock() for example. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-11 6:33 ` Christoph Hellwig @ 2018-09-11 7:20 ` Miklos Szeredi 0 siblings, 0 replies; 7+ messages in thread From: Miklos Szeredi @ 2018-09-11 7:20 UTC (permalink / raw) To: Christoph Hellwig Cc: syzbot, bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, Al Viro, Andrea Arcangeli, Andrew Morton On Tue, Sep 11, 2018 at 8:33 AM, Christoph Hellwig <hch@infradead.org> wrote: > On Mon, Sep 10, 2018 at 08:14:20PM +0200, Miklos Szeredi wrote: >> Why do pollable waitqueues need to disable interrupts generally? > > Any waitqueue needs to disable interrupts for consistency. We > always use spin_lock_irqsave in __wake_up_common_lock() for example. There are the _locked (non _irq) variants that do not. And poll/select/etc don't impose non-interuptibility on wakeups either. So it looks like it's just aio that has weird spin lock dependencies that forces this requirement on a waitq used in ->poll(). Thanks, Miklos ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-10 16:53 ` Christoph Hellwig 2018-09-10 18:14 ` Miklos Szeredi @ 2018-10-17 23:55 ` Andrea Arcangeli 1 sibling, 0 replies; 7+ messages in thread From: Andrea Arcangeli @ 2018-10-17 23:55 UTC (permalink / raw) To: Christoph Hellwig Cc: syzbot, bcrl, linux-aio, linux-fsdevel, linux-kernel, syzkaller-bugs, viro, akpm On Mon, Sep 10, 2018 at 09:53:17AM -0700, Christoph Hellwig wrote: > On Mon, Sep 10, 2018 at 12:41:05AM -0700, syzbot wrote: > > ===================================================== > > WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected > > 4.19.0-rc2+ #229 Not tainted > > ----------------------------------------------------- > > syz-executor2/9399 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: > > 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: spin_lock > > include/linux/spinlock.h:329 [inline] > > 00000000126506e0 (&ctx->fd_wqh){+.+.}, at: aio_poll+0x760/0x1420 > > fs/aio.c:1747 > > > > and this task is already holding: > > 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq > > include/linux/spinlock.h:354 [inline] > > 000000002bed6bf6 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 > > fs/aio.c:1746 > > which would create a new lock dependency: > > (&(&ctx->ctx_lock)->rlock){..-.} -> (&ctx->fd_wqh){+.+.} > > ctx->fd_wqh seems to only exist in userfaultfd, which indeed seems > to do strange open coded waitqueue locking, and seems to fail to disable > irqs. Something like this should fix it: > > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > index bfa0ec69f924..356d2b8568c1 100644 > --- a/fs/userfaultfd.c > +++ b/fs/userfaultfd.c > @@ -1026,7 +1026,7 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait, > struct userfaultfd_ctx *fork_nctx = NULL; > > /* always take the fd_wqh lock before the fault_pending_wqh lock */ > - spin_lock(&ctx->fd_wqh.lock); > + spin_lock_irq(&ctx->fd_wqh.lock); > __add_wait_queue(&ctx->fd_wqh, &wait); > for (;;) { > set_current_state(TASK_INTERRUPTIBLE); > @@ -1112,13 +1112,13 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait, > ret = -EAGAIN; > break; > } > - spin_unlock(&ctx->fd_wqh.lock); > + spin_unlock_irq(&ctx->fd_wqh.lock); > schedule(); > - spin_lock(&ctx->fd_wqh.lock); > + spin_lock_irq(&ctx->fd_wqh.lock); > } > __remove_wait_queue(&ctx->fd_wqh, &wait); > __set_current_state(TASK_RUNNING); > - spin_unlock(&ctx->fd_wqh.lock); > + spin_unlock_irq(&ctx->fd_wqh.lock); > > if (!ret && msg->event == UFFD_EVENT_FORK) { > ret = resolve_userfault_fork(ctx, fork_nctx, msg); > Reviewed-by: Andrea Arcangeli <aarcange@redhat.com> This is lock inversion with userfaultfd_poll that takes the fq_wqh after the irqsafe aio lock. And the aio lock can be taken from softirq (so potentially for interrupts) leading to a potential lock inversion deadlock. I suggest to add a comment about the above in the code before the first spin_lock_irq to explain why it needs to be _irq or it's not obvious. c430d1e848ff1240d126e79780f3c26208b8aed9 was just a false positive instead. I didn't comment on c430d1e848ff1240d126e79780f3c26208b8aed9 because I was too busy with the speculative execution issues at the time and it was just fine to drop the microoptimization, but while at it can we look in how to add a spin_acquire or find a way to teach lockdep in another way, so it's happy even if we restore the microoptimization? If we do that, in addition we should also initialize the ctx->fault_wqh spinlock to locked in the same patch (a spin_lock during uffd ctx creation will do) to be sure nobody takes it as further robustness feature against future modification (it gets more self documenting too that it's not supposed to be taken and the fault_pending_wq.lock has to be taken instead). Thanks, Andrea ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible deadlock in aio_poll 2018-09-10 7:41 possible deadlock in aio_poll syzbot 2018-09-10 16:53 ` Christoph Hellwig @ 2018-10-27 6:16 ` syzbot 1 sibling, 0 replies; 7+ messages in thread From: syzbot @ 2018-10-27 6:16 UTC (permalink / raw) To: aarcange, akpm, bcrl, hch, linux-aio, linux-fsdevel, linux-kernel, miklos, syzkaller-bugs, viro syzbot has found a reproducer for the following crash on: HEAD commit: 18d0eae30e6a Merge tag 'char-misc-4.20-rc1' of git://git.k.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=14728be5400000 kernel config: https://syzkaller.appspot.com/x/.config?x=342f43de913c81b9 dashboard link: https://syzkaller.appspot.com/bug?extid=5b1df0420c523b45a953 compiler: gcc (GCC) 8.0.1 20180413 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=161d6999400000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=110f4cf5400000 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+5b1df0420c523b45a953@syzkaller.appspotmail.com ===================================================== WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected 4.19.0+ #84 Not tainted ----------------------------------------------------- syz-executor781/7254 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: 00000000e70e96f7 (&ctx->fd_wqh){+.+.}, at: spin_lock include/linux/spinlock.h:329 [inline] 00000000e70e96f7 (&ctx->fd_wqh){+.+.}, at: aio_poll+0x760/0x1420 fs/aio.c:1747 and this task is already holding: 000000009957d7d7 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:354 [inline] 000000009957d7d7 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 fs/aio.c:1746 which would create a new lock dependency: (&(&ctx->ctx_lock)->rlock){..-.} -> (&ctx->fd_wqh){+.+.} but this new dependency connects a SOFTIRQ-irq-safe lock: (&(&ctx->ctx_lock)->rlock){..-.} ... which became SOFTIRQ-irq-safe at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:285 [inline] percpu_ref_put include/linux/percpu-refcount.h:301 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x563/0x730 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:240 [inline] rcu_do_batch kernel/rcu/tree.c:2437 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline] rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697 __do_softirq+0x308/0xb7e kernel/softirq.c:292 invoke_softirq kernel/softirq.c:373 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:413 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:801 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:151 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x49b/0x5c0 kernel/sched/idle.c:262 cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:353 start_secondary+0x487/0x5f0 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 to a SOFTIRQ-irq-unsafe lock: (&ctx->fd_wqh){+.+.} ... which became SOFTIRQ-irq-unsafe at: ... lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 __vfs_read+0x117/0x9b0 fs/read_write.c:416 vfs_read+0x17f/0x3c0 fs/read_write.c:452 ksys_read+0x101/0x260 fs/read_write.c:578 __do_sys_read fs/read_write.c:588 [inline] __se_sys_read fs/read_write.c:586 [inline] __x64_sys_read+0x73/0xb0 fs/read_write.c:586 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&ctx->fd_wqh); local_irq_disable(); lock(&(&ctx->ctx_lock)->rlock); lock(&ctx->fd_wqh); <Interrupt> lock(&(&ctx->ctx_lock)->rlock); *** DEADLOCK *** 1 lock held by syz-executor781/7254: #0: 000000009957d7d7 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:354 [inline] #0: 000000009957d7d7 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll+0x738/0x1420 fs/aio.c:1746 the dependencies between SOFTIRQ-irq-safe lock and the holding lock: -> (&(&ctx->ctx_lock)->rlock){..-.} { IN-SOFTIRQ-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:285 [inline] percpu_ref_put include/linux/percpu-refcount.h:301 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x563/0x730 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:240 [inline] rcu_do_batch kernel/rcu/tree.c:2437 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline] rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697 __do_softirq+0x308/0xb7e kernel/softirq.c:292 invoke_softirq kernel/softirq.c:373 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:413 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:801 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:151 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x49b/0x5c0 kernel/sched/idle.c:262 cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:353 start_secondary+0x487/0x5f0 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 INITIAL USE at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline] _raw_spin_lock_irq+0x61/0x80 kernel/locking/spinlock.c:160 spin_lock_irq include/linux/spinlock.h:354 [inline] free_ioctx_users+0xbc/0x710 fs/aio.c:603 percpu_ref_put_many include/linux/percpu-refcount.h:285 [inline] percpu_ref_put include/linux/percpu-refcount.h:301 [inline] percpu_ref_call_confirm_rcu lib/percpu-refcount.c:123 [inline] percpu_ref_switch_to_atomic_rcu+0x563/0x730 lib/percpu-refcount.c:158 __rcu_reclaim kernel/rcu/rcu.h:240 [inline] rcu_do_batch kernel/rcu/tree.c:2437 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline] rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697 __do_softirq+0x308/0xb7e kernel/softirq.c:292 invoke_softirq kernel/softirq.c:373 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:413 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:801 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:151 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x49b/0x5c0 kernel/sched/idle.c:262 cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:353 start_secondary+0x487/0x5f0 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 } ... key at: [<ffffffff8aed9b20>] __key.50623+0x0/0x40 ... acquired at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xa49/0xf80 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b7/0x580 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: -> (&ctx->fd_wqh){+.+.} { HARDIRQ-ON-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 __vfs_read+0x117/0x9b0 fs/read_write.c:416 vfs_read+0x17f/0x3c0 fs/read_write.c:452 ksys_read+0x101/0x260 fs/read_write.c:578 __do_sys_read fs/read_write.c:588 [inline] __se_sys_read fs/read_write.c:586 [inline] __x64_sys_read+0x73/0xb0 fs/read_write.c:586 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe SOFTIRQ-ON-W at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 __vfs_read+0x117/0x9b0 fs/read_write.c:416 vfs_read+0x17f/0x3c0 fs/read_write.c:452 ksys_read+0x101/0x260 fs/read_write.c:578 __do_sys_read fs/read_write.c:588 [inline] __se_sys_read fs/read_write.c:586 [inline] __x64_sys_read+0x73/0xb0 fs/read_write.c:586 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe INITIAL USE at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] userfaultfd_ctx_read+0x2e4/0x2180 fs/userfaultfd.c:1029 userfaultfd_read+0x1e2/0x2c0 fs/userfaultfd.c:1191 __vfs_read+0x117/0x9b0 fs/read_write.c:416 vfs_read+0x17f/0x3c0 fs/read_write.c:452 ksys_read+0x101/0x260 fs/read_write.c:578 __do_sys_read fs/read_write.c:588 [inline] __se_sys_read fs/read_write.c:586 [inline] __x64_sys_read+0x73/0xb0 fs/read_write.c:586 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe } ... key at: [<ffffffff8aed98a0>] __key.44253+0x0/0x40 ... acquired at: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xa49/0xf80 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b7/0x580 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe stack backtrace: CPU: 0 PID: 7254 Comm: syz-executor781 Not tainted 4.19.0+ #84 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x244/0x39d lib/dump_stack.c:113 print_bad_irq_dependency kernel/locking/lockdep.c:1570 [inline] check_usage.cold.58+0x6d5/0xad1 kernel/locking/lockdep.c:1602 check_irq_usage kernel/locking/lockdep.c:1658 [inline] check_prev_add_irq kernel/locking/lockdep_states.h:8 [inline] check_prev_add kernel/locking/lockdep.c:1868 [inline] check_prevs_add kernel/locking/lockdep.c:1976 [inline] validate_chain kernel/locking/lockdep.c:2347 [inline] __lock_acquire+0x238a/0x4c20 kernel/locking/lockdep.c:3341 lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] aio_poll+0x760/0x1420 fs/aio.c:1747 io_submit_one+0xa49/0xf80 fs/aio.c:1850 __do_sys_io_submit fs/aio.c:1916 [inline] __se_sys_io_submit fs/aio.c:1887 [inline] __x64_sys_io_submit+0x1b7/0x580 fs/aio.c:1887 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x447dc9 Code: e8 9c ba 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 5b 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007fc840e69da8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d1 RAX: ffffffffffffffda RBX: 00000000006e39e8 RCX: 0000000000447dc9 RDX: 0000000020000b00 RSI: 0000000000000001 RDI: 00007fc840e39000 RBP: 00000000006e39e0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000293 R12: 00000000006e39ec R13: 702f74656e2f666c R14: 65732f636f72702f R15: 0000000000000000 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-10-27 14:55 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-10 7:41 possible deadlock in aio_poll syzbot 2018-09-10 16:53 ` Christoph Hellwig 2018-09-10 18:14 ` Miklos Szeredi 2018-09-11 6:33 ` Christoph Hellwig 2018-09-11 7:20 ` Miklos Szeredi 2018-10-17 23:55 ` Andrea Arcangeli 2018-10-27 6:16 ` syzbot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).