* [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 2:01 syzbot 2022-02-17 7:34 ` Jason Wang 0 siblings, 1 reply; 17+ messages in thread From: syzbot @ 2022-02-17 2:01 UTC (permalink / raw) To: jasowang, kvm, linux-kernel, mst, netdev, syzkaller-bugs, virtualization Hello, syzbot found the following issue on: HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 Unfortunately, I don't have any reproducer for this issue yet. IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 Modules linked in: CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 __fput+0x286/0x9f0 fs/file_table.c:313 task_work_run+0xdd/0x1a0 kernel/task_work.c:164 exit_task_work include/linux/task_work.h:32 [inline] do_exit+0xb29/0x2a30 kernel/exit.c:806 do_group_exit+0xd2/0x2f0 kernel/exit.c:935 get_signal+0x45a/0x2490 kernel/signal.c:2863 arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 handle_signal_work kernel/entry/common.c:148 [inline] exit_to_user_mode_loop kernel/entry/common.c:172 [inline] exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f4027a46481 Code: Unable to access opcode bytes at RIP 0x7f4027a46457. RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 </TASK> --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 2:01 [syzbot] WARNING in vhost_dev_cleanup (2) syzbot @ 2022-02-17 7:34 ` Jason Wang 0 siblings, 0 replies; 17+ messages in thread From: Jason Wang @ 2022-02-17 7:34 UTC (permalink / raw) To: syzbot Cc: kvm, linux-kernel, mst, netdev, syzkaller-bugs, virtualization, Stefan Hajnoczi, Stefano Garzarella On Thu, Feb 17, 2022 at 10:01 AM syzbot <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following issue on: > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > Unfortunately, I don't have any reproducer for this issue yet. > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > Modules linked in: > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 Probably a hint that we are missing a flush. Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): static int vhost_vsock_stop(struct vhost_vsock *vsock) { size_t i; int ret; mutex_lock(&vsock->dev.mutex); ret = vhost_dev_check_owner(&vsock->dev); if (ret) goto err; Where it could fail so the device is not actually stopped. I wonder if this is something related. Thanks > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Call Trace: > <TASK> > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > __fput+0x286/0x9f0 fs/file_table.c:313 > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > exit_task_work include/linux/task_work.h:32 [inline] > do_exit+0xb29/0x2a30 kernel/exit.c:806 > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > get_signal+0x45a/0x2490 kernel/signal.c:2863 > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > handle_signal_work kernel/entry/common.c:148 [inline] > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > entry_SYSCALL_64_after_hwframe+0x44/0xae > RIP: 0033:0x7f4027a46481 > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > </TASK> > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 7:34 ` Jason Wang 0 siblings, 0 replies; 17+ messages in thread From: Jason Wang @ 2022-02-17 7:34 UTC (permalink / raw) To: syzbot Cc: kvm, mst, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 10:01 AM syzbot <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following issue on: > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > Unfortunately, I don't have any reproducer for this issue yet. > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > Modules linked in: > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 Probably a hint that we are missing a flush. Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): static int vhost_vsock_stop(struct vhost_vsock *vsock) { size_t i; int ret; mutex_lock(&vsock->dev.mutex); ret = vhost_dev_check_owner(&vsock->dev); if (ret) goto err; Where it could fail so the device is not actually stopped. I wonder if this is something related. Thanks > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Call Trace: > <TASK> > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > __fput+0x286/0x9f0 fs/file_table.c:313 > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > exit_task_work include/linux/task_work.h:32 [inline] > do_exit+0xb29/0x2a30 kernel/exit.c:806 > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > get_signal+0x45a/0x2490 kernel/signal.c:2863 > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > handle_signal_work kernel/entry/common.c:148 [inline] > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > entry_SYSCALL_64_after_hwframe+0x44/0xae > RIP: 0033:0x7f4027a46481 > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > </TASK> > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 7:34 ` Jason Wang @ 2022-02-17 7:36 ` Michael S. Tsirkin -1 siblings, 0 replies; 17+ messages in thread From: Michael S. Tsirkin @ 2022-02-17 7:36 UTC (permalink / raw) To: Jason Wang Cc: syzbot, kvm, linux-kernel, netdev, syzkaller-bugs, virtualization, Stefan Hajnoczi, Stefano Garzarella On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > On Thu, Feb 17, 2022 at 10:01 AM syzbot > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > Modules linked in: > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > Probably a hint that we are missing a flush. > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > { > size_t i; > int ret; > > mutex_lock(&vsock->dev.mutex); > > ret = vhost_dev_check_owner(&vsock->dev); > if (ret) > goto err; > > Where it could fail so the device is not actually stopped. > > I wonder if this is something related. > > Thanks But then if that is not the owner then no work should be running, right? > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > Call Trace: > > <TASK> > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > __fput+0x286/0x9f0 fs/file_table.c:313 > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > exit_task_work include/linux/task_work.h:32 [inline] > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > handle_signal_work kernel/entry/common.c:148 [inline] > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > RIP: 0033:0x7f4027a46481 > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > </TASK> > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 7:36 ` Michael S. Tsirkin 0 siblings, 0 replies; 17+ messages in thread From: Michael S. Tsirkin @ 2022-02-17 7:36 UTC (permalink / raw) To: Jason Wang Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > On Thu, Feb 17, 2022 at 10:01 AM syzbot > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > Modules linked in: > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > Probably a hint that we are missing a flush. > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > { > size_t i; > int ret; > > mutex_lock(&vsock->dev.mutex); > > ret = vhost_dev_check_owner(&vsock->dev); > if (ret) > goto err; > > Where it could fail so the device is not actually stopped. > > I wonder if this is something related. > > Thanks But then if that is not the owner then no work should be running, right? > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > Call Trace: > > <TASK> > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > __fput+0x286/0x9f0 fs/file_table.c:313 > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > exit_task_work include/linux/task_work.h:32 [inline] > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > handle_signal_work kernel/entry/common.c:148 [inline] > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > RIP: 0033:0x7f4027a46481 > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > </TASK> > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 7:36 ` Michael S. Tsirkin @ 2022-02-17 7:39 ` Jason Wang -1 siblings, 0 replies; 17+ messages in thread From: Jason Wang @ 2022-02-17 7:39 UTC (permalink / raw) To: Michael S. Tsirkin Cc: syzbot, kvm, linux-kernel, netdev, syzkaller-bugs, virtualization, Stefan Hajnoczi, Stefano Garzarella On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > Hello, > > > > > > syzbot found the following issue on: > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > git tree: upstream > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > Modules linked in: > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > Probably a hint that we are missing a flush. > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > { > > size_t i; > > int ret; > > > > mutex_lock(&vsock->dev.mutex); > > > > ret = vhost_dev_check_owner(&vsock->dev); > > if (ret) > > goto err; > > > > Where it could fail so the device is not actually stopped. > > > > I wonder if this is something related. > > > > Thanks > > > But then if that is not the owner then no work should be running, right? Could it be a buggy user space that passes the fd to another process and changes the owner just before the mutex_lock() above? Thanks > > > > > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > Call Trace: > > > <TASK> > > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > > __fput+0x286/0x9f0 fs/file_table.c:313 > > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > > exit_task_work include/linux/task_work.h:32 [inline] > > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > > handle_signal_work kernel/entry/common.c:148 [inline] > > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > > RIP: 0033:0x7f4027a46481 > > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > > </TASK> > > > > > > > > > --- > > > This report is generated by a bot. It may contain errors. > > > See https://goo.gl/tpsmEJ for more information about syzbot. > > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > > > syzbot will keep track of this issue. See: > > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 7:39 ` Jason Wang 0 siblings, 0 replies; 17+ messages in thread From: Jason Wang @ 2022-02-17 7:39 UTC (permalink / raw) To: Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > Hello, > > > > > > syzbot found the following issue on: > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > git tree: upstream > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > Modules linked in: > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > Probably a hint that we are missing a flush. > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > { > > size_t i; > > int ret; > > > > mutex_lock(&vsock->dev.mutex); > > > > ret = vhost_dev_check_owner(&vsock->dev); > > if (ret) > > goto err; > > > > Where it could fail so the device is not actually stopped. > > > > I wonder if this is something related. > > > > Thanks > > > But then if that is not the owner then no work should be running, right? Could it be a buggy user space that passes the fd to another process and changes the owner just before the mutex_lock() above? Thanks > > > > > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > Call Trace: > > > <TASK> > > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > > __fput+0x286/0x9f0 fs/file_table.c:313 > > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > > exit_task_work include/linux/task_work.h:32 [inline] > > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > > handle_signal_work kernel/entry/common.c:148 [inline] > > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > > RIP: 0033:0x7f4027a46481 > > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > > </TASK> > > > > > > > > > --- > > > This report is generated by a bot. It may contain errors. > > > See https://goo.gl/tpsmEJ for more information about syzbot. > > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > > > syzbot will keep track of this issue. See: > > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 7:39 ` Jason Wang @ 2022-02-17 7:50 ` Michael S. Tsirkin -1 siblings, 0 replies; 17+ messages in thread From: Michael S. Tsirkin @ 2022-02-17 7:50 UTC (permalink / raw) To: Jason Wang Cc: syzbot, kvm, linux-kernel, netdev, syzkaller-bugs, virtualization, Stefan Hajnoczi, Stefano Garzarella On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: > On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > > > Hello, > > > > > > > > syzbot found the following issue on: > > > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > > git tree: upstream > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > Modules linked in: > > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > > Probably a hint that we are missing a flush. > > > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > > { > > > size_t i; > > > int ret; > > > > > > mutex_lock(&vsock->dev.mutex); > > > > > > ret = vhost_dev_check_owner(&vsock->dev); > > > if (ret) > > > goto err; > > > > > > Where it could fail so the device is not actually stopped. > > > > > > I wonder if this is something related. > > > > > > Thanks > > > > > > But then if that is not the owner then no work should be running, right? > > Could it be a buggy user space that passes the fd to another process > and changes the owner just before the mutex_lock() above? > > Thanks Maybe, but can you be a bit more explicit? what is the set of conditions you see that can lead to this? > > > > > > > > > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > > Call Trace: > > > > <TASK> > > > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > > > __fput+0x286/0x9f0 fs/file_table.c:313 > > > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > > > exit_task_work include/linux/task_work.h:32 [inline] > > > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > > > handle_signal_work kernel/entry/common.c:148 [inline] > > > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > > > RIP: 0033:0x7f4027a46481 > > > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > > > </TASK> > > > > > > > > > > > > --- > > > > This report is generated by a bot. It may contain errors. > > > > See https://goo.gl/tpsmEJ for more information about syzbot. > > > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > > > > > syzbot will keep track of this issue. See: > > > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 7:50 ` Michael S. Tsirkin 0 siblings, 0 replies; 17+ messages in thread From: Michael S. Tsirkin @ 2022-02-17 7:50 UTC (permalink / raw) To: Jason Wang Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: > On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > > > Hello, > > > > > > > > syzbot found the following issue on: > > > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > > git tree: upstream > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > Modules linked in: > > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > > Probably a hint that we are missing a flush. > > > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > > { > > > size_t i; > > > int ret; > > > > > > mutex_lock(&vsock->dev.mutex); > > > > > > ret = vhost_dev_check_owner(&vsock->dev); > > > if (ret) > > > goto err; > > > > > > Where it could fail so the device is not actually stopped. > > > > > > I wonder if this is something related. > > > > > > Thanks > > > > > > But then if that is not the owner then no work should be running, right? > > Could it be a buggy user space that passes the fd to another process > and changes the owner just before the mutex_lock() above? > > Thanks Maybe, but can you be a bit more explicit? what is the set of conditions you see that can lead to this? > > > > > > > > > > > Code: c7 85 90 01 00 00 00 00 00 00 e8 53 6e a2 fa 48 89 ef 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e 41 5f e9 7d d6 ff ff e8 38 6e a2 fa <0f> 0b e9 46 ff ff ff 48 8b 7c 24 10 e8 87 00 ea fa e9 75 f7 ff ff > > > > RSP: 0018:ffffc9000fe6fa18 EFLAGS: 00010293 > > > > RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 > > > > RDX: ffff888021b63a00 RSI: ffffffff86d66fe8 RDI: ffff88801cc200b0 > > > > RBP: ffff88801cc20000 R08: 0000000000000001 R09: 0000000000000001 > > > > R10: ffffffff817f1e08 R11: 0000000000000000 R12: ffff88801cc200d0 > > > > R13: ffff88801cc20120 R14: ffff88801cc200d0 R15: 0000000000000002 > > > > FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > CR2: 0000001b2de25000 CR3: 000000004c9cd000 CR4: 00000000003506f0 > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > > Call Trace: > > > > <TASK> > > > > vhost_vsock_dev_release+0x36e/0x4b0 drivers/vhost/vsock.c:771 > > > > __fput+0x286/0x9f0 fs/file_table.c:313 > > > > task_work_run+0xdd/0x1a0 kernel/task_work.c:164 > > > > exit_task_work include/linux/task_work.h:32 [inline] > > > > do_exit+0xb29/0x2a30 kernel/exit.c:806 > > > > do_group_exit+0xd2/0x2f0 kernel/exit.c:935 > > > > get_signal+0x45a/0x2490 kernel/signal.c:2863 > > > > arch_do_signal_or_restart+0x2a9/0x1c40 arch/x86/kernel/signal.c:868 > > > > handle_signal_work kernel/entry/common.c:148 [inline] > > > > exit_to_user_mode_loop kernel/entry/common.c:172 [inline] > > > > exit_to_user_mode_prepare+0x17d/0x290 kernel/entry/common.c:207 > > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline] > > > > syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300 > > > > do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 > > > > entry_SYSCALL_64_after_hwframe+0x44/0xae > > > > RIP: 0033:0x7f4027a46481 > > > > Code: Unable to access opcode bytes at RIP 0x7f4027a46457. > > > > RSP: 002b:00007f402808ba68 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 > > > > RAX: fffffffffffffffc RBX: 00007f402622e700 RCX: 00007f4027a46481 > > > > RDX: 00007f402622e9d0 RSI: 00007f402622e2f0 RDI: 00000000003d0f00 > > > > RBP: 00007f402808bcb0 R08: 00007f402622e700 R09: 00007f402622e700 > > > > R10: 00007f402622e9d0 R11: 0000000000000206 R12: 00007f402808bb1e > > > > R13: 00007f402808bb1f R14: 00007f402622e300 R15: 0000000000022000 > > > > </TASK> > > > > > > > > > > > > --- > > > > This report is generated by a bot. It may contain errors. > > > > See https://goo.gl/tpsmEJ for more information about syzbot. > > > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > > > > > syzbot will keep track of this issue. See: > > > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > > > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 7:50 ` Michael S. Tsirkin @ 2022-02-17 9:48 ` Stefano Garzarella -1 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2022-02-17 9:48 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Jason Wang, syzbot, kvm, linux-kernel, netdev, syzkaller-bugs, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: > > On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > > > > > Hello, > > > > > > > > > > syzbot found the following issue on: > > > > > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > > > git tree: upstream > > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > Modules linked in: > > > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > > > > Probably a hint that we are missing a flush. > > > > > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > > > { > > > > size_t i; > > > > int ret; > > > > > > > > mutex_lock(&vsock->dev.mutex); > > > > > > > > ret = vhost_dev_check_owner(&vsock->dev); > > > > if (ret) > > > > goto err; > > > > > > > > Where it could fail so the device is not actually stopped. > > > > > > > > I wonder if this is something related. > > > > > > > > Thanks > > > > > > > > > But then if that is not the owner then no work should be running, right? > > > > Could it be a buggy user space that passes the fd to another process > > and changes the owner just before the mutex_lock() above? > > > > Thanks > > Maybe, but can you be a bit more explicit? what is the set of > conditions you see that can lead to this? I think the issue could be in the vhost_vsock_stop() as Jason mentioned, but not related to fd passing, but related to the do_exit() function. Looking the stack trace, we are in exit_task_work(), that is called after exit_mm(), so the vhost_dev_check_owner() can fail because current->mm should be NULL at that point. It seems the fput work is queued by fput_many() in a worker queue, and in some cases (maybe a lot of files opened?) the work is still queued when we enter in do_exit(). That said, I don't know if we can simply remove that check in vhost_vsock_stop(), or check if current->mm is NULL, to understand if the process is exiting. Stefano ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-17 9:48 ` Stefano Garzarella 0 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2022-02-17 9:48 UTC (permalink / raw) To: Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: > > On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: > > > > On Thu, Feb 17, 2022 at 10:01 AM syzbot > > > > <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: > > > > > > > > > > Hello, > > > > > > > > > > syzbot found the following issue on: > > > > > > > > > > HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. > > > > > git tree: upstream > > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=132e687c700000 > > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912 > > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0 > > > > > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > > Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com > > > > > > > > > > WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > Modules linked in: > > > > > CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 > > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > > > RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 > > > > > > > > Probably a hint that we are missing a flush. > > > > > > > > Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): > > > > > > > > static int vhost_vsock_stop(struct vhost_vsock *vsock) > > > > { > > > > size_t i; > > > > int ret; > > > > > > > > mutex_lock(&vsock->dev.mutex); > > > > > > > > ret = vhost_dev_check_owner(&vsock->dev); > > > > if (ret) > > > > goto err; > > > > > > > > Where it could fail so the device is not actually stopped. > > > > > > > > I wonder if this is something related. > > > > > > > > Thanks > > > > > > > > > But then if that is not the owner then no work should be running, right? > > > > Could it be a buggy user space that passes the fd to another process > > and changes the owner just before the mutex_lock() above? > > > > Thanks > > Maybe, but can you be a bit more explicit? what is the set of > conditions you see that can lead to this? I think the issue could be in the vhost_vsock_stop() as Jason mentioned, but not related to fd passing, but related to the do_exit() function. Looking the stack trace, we are in exit_task_work(), that is called after exit_mm(), so the vhost_dev_check_owner() can fail because current->mm should be NULL at that point. It seems the fput work is queued by fput_many() in a worker queue, and in some cases (maybe a lot of files opened?) the work is still queued when we enter in do_exit(). That said, I don't know if we can simply remove that check in vhost_vsock_stop(), or check if current->mm is NULL, to understand if the process is exiting. Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-17 9:48 ` Stefano Garzarella @ 2022-02-18 17:53 ` Mike Christie -1 siblings, 0 replies; 17+ messages in thread From: Mike Christie @ 2022-02-18 17:53 UTC (permalink / raw) To: Stefano Garzarella, Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On 2/17/22 3:48 AM, Stefano Garzarella wrote: > > On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>> >>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> syzbot found the following issue on: >>>>>> >>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>> git tree: upstream >>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>> >>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>> >>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>> >>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>> Modules linked in: >>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>> >>>>> Probably a hint that we are missing a flush. >>>>> >>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>> >>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>> { >>>>> size_t i; >>>>> int ret; >>>>> >>>>> mutex_lock(&vsock->dev.mutex); >>>>> >>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>> if (ret) >>>>> goto err; >>>>> >>>>> Where it could fail so the device is not actually stopped. >>>>> >>>>> I wonder if this is something related. >>>>> >>>>> Thanks >>>> >>>> >>>> But then if that is not the owner then no work should be running, right? >>> >>> Could it be a buggy user space that passes the fd to another process >>> and changes the owner just before the mutex_lock() above? >>> >>> Thanks >> >> Maybe, but can you be a bit more explicit? what is the set of >> conditions you see that can lead to this? > > I think the issue could be in the vhost_vsock_stop() as Jason mentioned, > but not related to fd passing, but related to the do_exit() function. > > Looking the stack trace, we are in exit_task_work(), that is called > after exit_mm(), so the vhost_dev_check_owner() can fail because > current->mm should be NULL at that point. > > It seems the fput work is queued by fput_many() in a worker queue, and > in some cases (maybe a lot of files opened?) the work is still queued > when we enter in do_exit(). It normally happens if userspace doesn't do a close() when the VM is shutdown and instead let's the kernel's reaper code cleanup. The qemu vhost-scsi code doesn't do a close() during shutdown and so this is our normal code path. It also happens when something like qemu is not gracefully shutdown like during a crash. So fire up qemu, start IO, then crash it or kill 9 it while IO is still running and you can hit it. > > That said, I don't know if we can simply remove that check in > vhost_vsock_stop(), or check if current->mm is NULL, to understand if > the process is exiting. > Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop when to check? - vhost_vsock_dev_ioctl always wants to check for ownership right? - For vhost_vsock_dev_release ownership doesn't matter because we always want to clean up or it doesn't hurt too much. For the case where we just do open then close and no ioctls then running vhost_vq_set_backend in vhost_vsock_stop is just a minor hit of extra work. If we've done ioctls, but are now in vhost_vsock_dev_release then we know for the graceful and ungraceful case that nothing is going to be accessing this device in the future and it's getting completely freed so we must completely clean it up. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-18 17:53 ` Mike Christie 0 siblings, 0 replies; 17+ messages in thread From: Mike Christie @ 2022-02-18 17:53 UTC (permalink / raw) To: Stefano Garzarella, Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On 2/17/22 3:48 AM, Stefano Garzarella wrote: > > On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>> >>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> syzbot found the following issue on: >>>>>> >>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>> git tree: upstream >>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>> >>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>> >>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>> >>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>> Modules linked in: >>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>> >>>>> Probably a hint that we are missing a flush. >>>>> >>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>> >>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>> { >>>>> size_t i; >>>>> int ret; >>>>> >>>>> mutex_lock(&vsock->dev.mutex); >>>>> >>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>> if (ret) >>>>> goto err; >>>>> >>>>> Where it could fail so the device is not actually stopped. >>>>> >>>>> I wonder if this is something related. >>>>> >>>>> Thanks >>>> >>>> >>>> But then if that is not the owner then no work should be running, right? >>> >>> Could it be a buggy user space that passes the fd to another process >>> and changes the owner just before the mutex_lock() above? >>> >>> Thanks >> >> Maybe, but can you be a bit more explicit? what is the set of >> conditions you see that can lead to this? > > I think the issue could be in the vhost_vsock_stop() as Jason mentioned, > but not related to fd passing, but related to the do_exit() function. > > Looking the stack trace, we are in exit_task_work(), that is called > after exit_mm(), so the vhost_dev_check_owner() can fail because > current->mm should be NULL at that point. > > It seems the fput work is queued by fput_many() in a worker queue, and > in some cases (maybe a lot of files opened?) the work is still queued > when we enter in do_exit(). It normally happens if userspace doesn't do a close() when the VM is shutdown and instead let's the kernel's reaper code cleanup. The qemu vhost-scsi code doesn't do a close() during shutdown and so this is our normal code path. It also happens when something like qemu is not gracefully shutdown like during a crash. So fire up qemu, start IO, then crash it or kill 9 it while IO is still running and you can hit it. > > That said, I don't know if we can simply remove that check in > vhost_vsock_stop(), or check if current->mm is NULL, to understand if > the process is exiting. > Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop when to check? - vhost_vsock_dev_ioctl always wants to check for ownership right? - For vhost_vsock_dev_release ownership doesn't matter because we always want to clean up or it doesn't hurt too much. For the case where we just do open then close and no ioctls then running vhost_vq_set_backend in vhost_vsock_stop is just a minor hit of extra work. If we've done ioctls, but are now in vhost_vsock_dev_release then we know for the graceful and ungraceful case that nothing is going to be accessing this device in the future and it's getting completely freed so we must completely clean it up. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-18 17:53 ` Mike Christie @ 2022-02-18 18:23 ` Mike Christie -1 siblings, 0 replies; 17+ messages in thread From: Mike Christie @ 2022-02-18 18:23 UTC (permalink / raw) To: Stefano Garzarella, Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On 2/18/22 11:53 AM, Mike Christie wrote: > On 2/17/22 3:48 AM, Stefano Garzarella wrote: >> >> On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>> >>> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>> >>>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> syzbot found the following issue on: >>>>>>> >>>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>>> git tree: upstream >>>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>>> >>>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>>> >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>>> >>>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>> Modules linked in: >>>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>> >>>>>> Probably a hint that we are missing a flush. >>>>>> >>>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>>> >>>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>>> { >>>>>> size_t i; >>>>>> int ret; >>>>>> >>>>>> mutex_lock(&vsock->dev.mutex); >>>>>> >>>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>>> if (ret) >>>>>> goto err; >>>>>> >>>>>> Where it could fail so the device is not actually stopped. >>>>>> >>>>>> I wonder if this is something related. >>>>>> >>>>>> Thanks >>>>> >>>>> >>>>> But then if that is not the owner then no work should be running, right? >>>> >>>> Could it be a buggy user space that passes the fd to another process >>>> and changes the owner just before the mutex_lock() above? >>>> >>>> Thanks >>> >>> Maybe, but can you be a bit more explicit? what is the set of >>> conditions you see that can lead to this? >> >> I think the issue could be in the vhost_vsock_stop() as Jason mentioned, >> but not related to fd passing, but related to the do_exit() function. >> >> Looking the stack trace, we are in exit_task_work(), that is called >> after exit_mm(), so the vhost_dev_check_owner() can fail because >> current->mm should be NULL at that point. >> >> It seems the fput work is queued by fput_many() in a worker queue, and >> in some cases (maybe a lot of files opened?) the work is still queued >> when we enter in do_exit(). > It normally happens if userspace doesn't do a close() when the VM Just one clarification. I meant to say it "always" happens when userspace doesn't do a close. It doesn't have anything to do with lots of files or something like that. We are actually running the vhost device's release function from do_exit->task_work_run and so all those __fputs are done from something like qemu's context (current == that process). We are *not* hitting the case: do_exit->exit_files->put_files_struct->filp_close->fput->fput_many and then in there hitting the schedule_delayed_work path. For that the last __fput would be done from a workqueue thread and so the current pointer would point to a completely different thread. > is shutdown and instead let's the kernel's reaper code cleanup. The qemu > vhost-scsi code doesn't do a close() during shutdown and so this is our > normal code path. It also happens when something like qemu is not > gracefully shutdown like during a crash. > > So fire up qemu, start IO, then crash it or kill 9 it while IO is still > running and you can hit it. > >> >> That said, I don't know if we can simply remove that check in >> vhost_vsock_stop(), or check if current->mm is NULL, to understand if >> the process is exiting. >> > > Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop > when to check? > > - vhost_vsock_dev_ioctl always wants to check for ownership right? > > - For vhost_vsock_dev_release ownership doesn't matter because we > always want to clean up or it doesn't hurt too much. > > For the case where we just do open then close and no ioctls then > running vhost_vq_set_backend in vhost_vsock_stop is just a minor > hit of extra work. If we've done ioctls, but are now in > vhost_vsock_dev_release then we know for the graceful and ungraceful > case that nothing is going to be accessing this device in the future > and it's getting completely freed so we must completely clean it up. > > > > > > _______________________________________________ > Virtualization mailing list > Virtualization@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-18 18:23 ` Mike Christie 0 siblings, 0 replies; 17+ messages in thread From: Mike Christie @ 2022-02-18 18:23 UTC (permalink / raw) To: Stefano Garzarella, Michael S. Tsirkin Cc: syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On 2/18/22 11:53 AM, Mike Christie wrote: > On 2/17/22 3:48 AM, Stefano Garzarella wrote: >> >> On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>> >>> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>> >>>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> syzbot found the following issue on: >>>>>>> >>>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>>> git tree: upstream >>>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>>> >>>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>>> >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>>> >>>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>> Modules linked in: >>>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>> >>>>>> Probably a hint that we are missing a flush. >>>>>> >>>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>>> >>>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>>> { >>>>>> size_t i; >>>>>> int ret; >>>>>> >>>>>> mutex_lock(&vsock->dev.mutex); >>>>>> >>>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>>> if (ret) >>>>>> goto err; >>>>>> >>>>>> Where it could fail so the device is not actually stopped. >>>>>> >>>>>> I wonder if this is something related. >>>>>> >>>>>> Thanks >>>>> >>>>> >>>>> But then if that is not the owner then no work should be running, right? >>>> >>>> Could it be a buggy user space that passes the fd to another process >>>> and changes the owner just before the mutex_lock() above? >>>> >>>> Thanks >>> >>> Maybe, but can you be a bit more explicit? what is the set of >>> conditions you see that can lead to this? >> >> I think the issue could be in the vhost_vsock_stop() as Jason mentioned, >> but not related to fd passing, but related to the do_exit() function. >> >> Looking the stack trace, we are in exit_task_work(), that is called >> after exit_mm(), so the vhost_dev_check_owner() can fail because >> current->mm should be NULL at that point. >> >> It seems the fput work is queued by fput_many() in a worker queue, and >> in some cases (maybe a lot of files opened?) the work is still queued >> when we enter in do_exit(). > It normally happens if userspace doesn't do a close() when the VM Just one clarification. I meant to say it "always" happens when userspace doesn't do a close. It doesn't have anything to do with lots of files or something like that. We are actually running the vhost device's release function from do_exit->task_work_run and so all those __fputs are done from something like qemu's context (current == that process). We are *not* hitting the case: do_exit->exit_files->put_files_struct->filp_close->fput->fput_many and then in there hitting the schedule_delayed_work path. For that the last __fput would be done from a workqueue thread and so the current pointer would point to a completely different thread. > is shutdown and instead let's the kernel's reaper code cleanup. The qemu > vhost-scsi code doesn't do a close() during shutdown and so this is our > normal code path. It also happens when something like qemu is not > gracefully shutdown like during a crash. > > So fire up qemu, start IO, then crash it or kill 9 it while IO is still > running and you can hit it. > >> >> That said, I don't know if we can simply remove that check in >> vhost_vsock_stop(), or check if current->mm is NULL, to understand if >> the process is exiting. >> > > Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop > when to check? > > - vhost_vsock_dev_ioctl always wants to check for ownership right? > > - For vhost_vsock_dev_release ownership doesn't matter because we > always want to clean up or it doesn't hurt too much. > > For the case where we just do open then close and no ioctls then > running vhost_vq_set_backend in vhost_vsock_stop is just a minor > hit of extra work. If we've done ioctls, but are now in > vhost_vsock_dev_release then we know for the graceful and ungraceful > case that nothing is going to be accessing this device in the future > and it's getting completely freed so we must completely clean it up. > > > > > > _______________________________________________ > Virtualization mailing list > Virtualization@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/virtualization _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) 2022-02-18 18:23 ` Mike Christie @ 2022-02-21 10:05 ` Stefano Garzarella -1 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2022-02-21 10:05 UTC (permalink / raw) To: Mike Christie Cc: syzbot, kvm, Michael S. Tsirkin, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Fri, Feb 18, 2022 at 12:23:10PM -0600, Mike Christie wrote: >On 2/18/22 11:53 AM, Mike Christie wrote: >> On 2/17/22 3:48 AM, Stefano Garzarella wrote: >>> >>> On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>>> >>>> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>>>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>> >>>>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> syzbot found the following issue on: >>>>>>>> >>>>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>>>> git tree: upstream >>>>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>>>> >>>>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>>>> >>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>>>> >>>>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>>> Modules linked in: >>>>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>> >>>>>>> Probably a hint that we are missing a flush. >>>>>>> >>>>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>>>> >>>>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>>>> { >>>>>>> size_t i; >>>>>>> int ret; >>>>>>> >>>>>>> mutex_lock(&vsock->dev.mutex); >>>>>>> >>>>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>>>> if (ret) >>>>>>> goto err; >>>>>>> >>>>>>> Where it could fail so the device is not actually stopped. >>>>>>> >>>>>>> I wonder if this is something related. >>>>>>> >>>>>>> Thanks >>>>>> >>>>>> >>>>>> But then if that is not the owner then no work should be running, right? >>>>> >>>>> Could it be a buggy user space that passes the fd to another process >>>>> and changes the owner just before the mutex_lock() above? >>>>> >>>>> Thanks >>>> >>>> Maybe, but can you be a bit more explicit? what is the set of >>>> conditions you see that can lead to this? >>> >>> I think the issue could be in the vhost_vsock_stop() as Jason mentioned, >>> but not related to fd passing, but related to the do_exit() function. >>> >>> Looking the stack trace, we are in exit_task_work(), that is called >>> after exit_mm(), so the vhost_dev_check_owner() can fail because >>> current->mm should be NULL at that point. >>> >>> It seems the fput work is queued by fput_many() in a worker queue, and >>> in some cases (maybe a lot of files opened?) the work is still queued >>> when we enter in do_exit(). >> It normally happens if userspace doesn't do a close() when the VM > >Just one clarification. I meant to say it "always" happens when userspace >doesn't do a close. > >It doesn't have anything to do with lots of files or something like that. >We are actually running the vhost device's release function from >do_exit->task_work_run and so all those __fputs are done from something >like qemu's context (current == that process). > >We are *not* hitting the case: > >do_exit->exit_files->put_files_struct->filp_close->fput->fput_many > >and then in there hitting the schedule_delayed_work path. For that >the last __fput would be done from a workqueue thread and so the current >pointer would point to a completely different thread. > > > >> is shutdown and instead let's the kernel's reaper code cleanup. The qemu >> vhost-scsi code doesn't do a close() during shutdown and so this is our >> normal code path. It also happens when something like qemu is not >> gracefully shutdown like during a crash. >> >> So fire up qemu, start IO, then crash it or kill 9 it while IO is still >> running and you can hit it. Thank you very much for this detailed explanation! >> >>> >>> That said, I don't know if we can simply remove that check in >>> vhost_vsock_stop(), or check if current->mm is NULL, to understand if >>> the process is exiting. >>> >> >> Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop >> when to check? >> >> - vhost_vsock_dev_ioctl always wants to check for ownership right? >> >> - For vhost_vsock_dev_release ownership doesn't matter because we >> always want to clean up or it doesn't hurt too much. >> >> For the case where we just do open then close and no ioctls then >> running vhost_vq_set_backend in vhost_vsock_stop is just a minor >> hit of extra work. If we've done ioctls, but are now in >> vhost_vsock_dev_release then we know for the graceful and ungraceful >> case that nothing is going to be accessing this device in the future >> and it's getting completely freed so we must completely clean it up. Yep, I think the easiest way is to add a parameter to vhost_vsock_stop() to tell when to call vhost_dev_check_owner() or not. This is because dev->mm is protected by dev->mutex, acquired in vhost_vsock_stop(). I will send a patch right away, it would be great if you can review. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot] WARNING in vhost_dev_cleanup (2) @ 2022-02-21 10:05 ` Stefano Garzarella 0 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2022-02-21 10:05 UTC (permalink / raw) To: Mike Christie Cc: Michael S. Tsirkin, syzbot, kvm, netdev, syzkaller-bugs, linux-kernel, virtualization, Stefan Hajnoczi On Fri, Feb 18, 2022 at 12:23:10PM -0600, Mike Christie wrote: >On 2/18/22 11:53 AM, Mike Christie wrote: >> On 2/17/22 3:48 AM, Stefano Garzarella wrote: >>> >>> On Thu, Feb 17, 2022 at 8:50 AM Michael S. Tsirkin <mst@redhat.com> wrote: >>>> >>>> On Thu, Feb 17, 2022 at 03:39:48PM +0800, Jason Wang wrote: >>>>> On Thu, Feb 17, 2022 at 3:36 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>> >>>>>> On Thu, Feb 17, 2022 at 03:34:13PM +0800, Jason Wang wrote: >>>>>>> On Thu, Feb 17, 2022 at 10:01 AM syzbot >>>>>>> <syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> syzbot found the following issue on: >>>>>>>> >>>>>>>> HEAD commit: c5d9ae265b10 Merge tag 'for-linus' of git://git.kernel.org.. >>>>>>>> git tree: upstream >>>>>>>> console output: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/log.txt?x=132e687c700000__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9YisMihn$ >>>>>>>> kernel config: https://urldefense.com/v3/__https://syzkaller.appspot.com/x/.config?x=a78b064590b9f912__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9RjOhplp$ >>>>>>>> dashboard link: https://urldefense.com/v3/__https://syzkaller.appspot.com/bug?extid=1e3ea63db39f2b4440e0__;!!ACWV5N9M2RV99hQ!fLqQTyosTBm7FK50IVmo0ozZhsvUEPFCivEHFDGU3GjlAHDWl07UdOa-t9uf9bBf5tv0$ >>>>>>>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>>>>>>> >>>>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>>>> >>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>>>>>> Reported-by: syzbot+1e3ea63db39f2b4440e0@syzkaller.appspotmail.com >>>>>>>> >>>>>>>> WARNING: CPU: 1 PID: 10828 at drivers/vhost/vhost.c:715 vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>>> Modules linked in: >>>>>>>> CPU: 0 PID: 10828 Comm: syz-executor.0 Not tainted 5.17.0-rc4-syzkaller-00051-gc5d9ae265b10 #0 >>>>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>>>> RIP: 0010:vhost_dev_cleanup+0x8b8/0xbc0 drivers/vhost/vhost.c:715 >>>>>>> >>>>>>> Probably a hint that we are missing a flush. >>>>>>> >>>>>>> Looking at vhost_vsock_stop() that is called by vhost_vsock_dev_release(): >>>>>>> >>>>>>> static int vhost_vsock_stop(struct vhost_vsock *vsock) >>>>>>> { >>>>>>> size_t i; >>>>>>> int ret; >>>>>>> >>>>>>> mutex_lock(&vsock->dev.mutex); >>>>>>> >>>>>>> ret = vhost_dev_check_owner(&vsock->dev); >>>>>>> if (ret) >>>>>>> goto err; >>>>>>> >>>>>>> Where it could fail so the device is not actually stopped. >>>>>>> >>>>>>> I wonder if this is something related. >>>>>>> >>>>>>> Thanks >>>>>> >>>>>> >>>>>> But then if that is not the owner then no work should be running, right? >>>>> >>>>> Could it be a buggy user space that passes the fd to another process >>>>> and changes the owner just before the mutex_lock() above? >>>>> >>>>> Thanks >>>> >>>> Maybe, but can you be a bit more explicit? what is the set of >>>> conditions you see that can lead to this? >>> >>> I think the issue could be in the vhost_vsock_stop() as Jason mentioned, >>> but not related to fd passing, but related to the do_exit() function. >>> >>> Looking the stack trace, we are in exit_task_work(), that is called >>> after exit_mm(), so the vhost_dev_check_owner() can fail because >>> current->mm should be NULL at that point. >>> >>> It seems the fput work is queued by fput_many() in a worker queue, and >>> in some cases (maybe a lot of files opened?) the work is still queued >>> when we enter in do_exit(). >> It normally happens if userspace doesn't do a close() when the VM > >Just one clarification. I meant to say it "always" happens when userspace >doesn't do a close. > >It doesn't have anything to do with lots of files or something like that. >We are actually running the vhost device's release function from >do_exit->task_work_run and so all those __fputs are done from something >like qemu's context (current == that process). > >We are *not* hitting the case: > >do_exit->exit_files->put_files_struct->filp_close->fput->fput_many > >and then in there hitting the schedule_delayed_work path. For that >the last __fput would be done from a workqueue thread and so the current >pointer would point to a completely different thread. > > > >> is shutdown and instead let's the kernel's reaper code cleanup. The qemu >> vhost-scsi code doesn't do a close() during shutdown and so this is our >> normal code path. It also happens when something like qemu is not >> gracefully shutdown like during a crash. >> >> So fire up qemu, start IO, then crash it or kill 9 it while IO is still >> running and you can hit it. Thank you very much for this detailed explanation! >> >>> >>> That said, I don't know if we can simply remove that check in >>> vhost_vsock_stop(), or check if current->mm is NULL, to understand if >>> the process is exiting. >>> >> >> Should the caller do the vhost_dev_check_owner or tell vhost_vsock_stop >> when to check? >> >> - vhost_vsock_dev_ioctl always wants to check for ownership right? >> >> - For vhost_vsock_dev_release ownership doesn't matter because we >> always want to clean up or it doesn't hurt too much. >> >> For the case where we just do open then close and no ioctls then >> running vhost_vq_set_backend in vhost_vsock_stop is just a minor >> hit of extra work. If we've done ioctls, but are now in >> vhost_vsock_dev_release then we know for the graceful and ungraceful >> case that nothing is going to be accessing this device in the future >> and it's getting completely freed so we must completely clean it up. Yep, I think the easiest way is to add a parameter to vhost_vsock_stop() to tell when to call vhost_dev_check_owner() or not. This is because dev->mm is protected by dev->mutex, acquired in vhost_vsock_stop(). I will send a patch right away, it would be great if you can review. Thanks, Stefano ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2022-02-21 10:44 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-02-17 2:01 [syzbot] WARNING in vhost_dev_cleanup (2) syzbot 2022-02-17 7:34 ` Jason Wang 2022-02-17 7:34 ` Jason Wang 2022-02-17 7:36 ` Michael S. Tsirkin 2022-02-17 7:36 ` Michael S. Tsirkin 2022-02-17 7:39 ` Jason Wang 2022-02-17 7:39 ` Jason Wang 2022-02-17 7:50 ` Michael S. Tsirkin 2022-02-17 7:50 ` Michael S. Tsirkin 2022-02-17 9:48 ` Stefano Garzarella 2022-02-17 9:48 ` Stefano Garzarella 2022-02-18 17:53 ` Mike Christie 2022-02-18 17:53 ` Mike Christie 2022-02-18 18:23 ` Mike Christie 2022-02-18 18:23 ` Mike Christie 2022-02-21 10:05 ` Stefano Garzarella 2022-02-21 10:05 ` Stefano Garzarella
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.