* virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs)
@ 2020-10-02 16:28 Qian Cai
2020-10-03 2:44 ` Qian Cai
0 siblings, 1 reply; 5+ messages in thread
From: Qian Cai @ 2020-10-02 16:28 UTC (permalink / raw)
To: Vivek Goyal, Stefan Hajnoczi, Miklos Szeredi
Cc: linux-fsdevel, linux-kernel, virtio-fs
Running some fuzzing on virtiofs from a non-privileged user could trigger a
warning in virtio_fs_enqueue_req():
WARN_ON(out_sgs + in_sgs != total_sgs);
# /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu -o source=$TESTDIR -o cache=always -o no_posix_lock
...
# mount -t virtiofs myfs /tmp
$ cd /tmp
$ trinity -C 48 --arch 64
From the log, the final piece of the code from the process was:
ioctl(fd=343, cmd=0x5a004000, arg=0x40000000);
[ 4327.977314] WARNING: CPU: 2 PID: 12259 at fs/fuse/virtio_fs.c:1151 virtio_fs_enqueue_req+0xa86/0xdb0 [virtiofs]
[ 4327.983910] Modules linked in: cmtp kernelcapi hidp bnep bridge stp llc dlci pppoe rfcomm nfnetlink pptp gre can_bcm bluetooth ecdh_generic ecc l2tp_ppp l2tp_netlink l2tp_core ip6_udp_tunnel udp_tunnel pppoxw
[ 4327.984068] sunrpc dm_mirror dm_region_hash dm_log dm_mod
[ 4328.046826] CPU: 2 PID: 12259 Comm: trinity-c20 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #5
[ 4328.053714] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 4328.059513] RIP: 0010:virtio_fs_enqueue_req+0xa86/0xdb0 [virtiofs]
[ 4328.063812] Code: c1 e7 05 48 03 7c 24 10 6a 00 e8 85 a4 ff ff 8d 48 01 58 41 8d 54 0d 00 e9 d2 fb ff ff 48 89 ef e8 8f 33 5e f9 e9 42 fe ff ff <0f> 0b e9 c7 fb ff ff 48 8b 7c 24 08 e8 c9 49 cf f8 0f b6 45 19
[ 4328.076709] RSP: 0018:ffff8889fbb4f9c0 EFLAGS: 00010297
[ 4328.079112] RAX: 0000000000000000 RBX: ffff8889c9ad88a8 RCX: 0000000000000003
[ 4328.083725] RDX: 0000000000000007 RSI: 0000000000000000 RDI: ffff88810575c1cc
[ 4328.089156] RBP: ffff8889fbb4fe20 R08: ffffed1020aeb83c R09: 0000000000001000
[ 4328.095906] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000008
[ 4328.101870] R13: 0000000000000004 R14: 0000000000000003 R15: ffff8889c9ad88d8
[ 4328.106674] FS: 00007f1129d21740(0000) GS:ffff888a7e900000(0000) knlGS:0000000000000000
[ 4328.111642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4328.114333] CR2: 000000000000002f CR3: 000000090f4ea005 CR4: 0000000000770ee0
[ 4328.117623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4328.122782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4328.128516] PKRU: 55555550
[ 4328.130769] Call Trace:
[ 4328.131992] ? virtio_fs_probe+0x14d0/0x14d0 [virtiofs]
[ 4328.134465] ? trace_hardirqs_on+0x1c/0x110
[ 4328.136419] ? make_kprojid+0x20/0x20
[ 4328.138936] ? __is_kernel_percpu_address+0x63/0x1e0
[ 4328.141899] ? __module_address+0x3f/0x370
[ 4328.143835] ? lockdep_hardirqs_on_prepare+0x4d0/0x4d0
[ 4328.146248] ? virtio_fs_wake_pending_and_unlock+0x18b/0x610 [virtiofs]
[ 4328.149323] ? lock_downgrade+0x730/0x730
[ 4328.151217] ? lock_acquire+0x17f/0x7e0
[ 4328.152998] ? fuse_simple_request+0x233/0x9f0 [fuse]
[ 4328.155360] ? rcu_read_unlock+0x40/0x40
[ 4328.157169] virtio_fs_wake_pending_and_unlock+0x1f0/0x610 [virtiofs]
virtio_fs_wake_pending_and_unlock at fs/fuse/virtio_fs.c:1227 (discriminator 10)
[ 4328.160173] ? queue_request_and_unlock+0x11e/0x290 [fuse]
[ 4328.162685] fuse_simple_request+0x3b2/0x9f0 [fuse]
__fuse_request_send at fs/fuse/dev.c:421
(inlined by) fuse_simple_request at fs/fuse/dev.c:503
[ 4328.164933] fuse_do_ioctl+0x6c6/0x1280 [fuse]
[ 4328.166992] ? fuse_readahead+0x1410/0x1410 [fuse]
[ 4328.169213] ? hrtimer_forward+0x1b0/0x1b0
[ 4328.171113] ? hrtimer_cancel+0x20/0x20
[ 4328.172903] ? ioctl_file_clone+0x120/0x120
[ 4328.174849] ? _raw_spin_unlock_irq+0x24/0x30
[ 4328.176871] ? fuse_allow_current_process+0x235/0x2a0 [fuse]
[ 4328.181615] __x64_sys_ioctl+0x128/0x190
[ 4328.184832] do_syscall_64+0x33/0x40
[ 4328.190405] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 4328.196680] RIP: 0033:0x7f112963478d
[ 4328.200415] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
[ 4328.214734] RSP: 002b:00007ffd75a76ea8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 4328.220222] RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007f112963478d
[ 4328.224383] RDX: 0000000040000000 RSI: 000000005a004000 RDI: 0000000000000157
[ 4328.228838] RBP: 0000000000000010 R08: 00000000000000a6 R09: 000000002e2e2e2e
[ 4328.233241] R10: fffffffffffffffc R11: 0000000000000246 R12: 0000000000000002
[ 4328.237136] R13: 00007f1129c8e058 R14: 00007f1129d216c0 R15: 00007f1129c8e000
[ 4328.240635] CPU: 2 PID: 12259 Comm: trinity-c20 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #5
[ 4328.248370] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 4328.254499] Call Trace:
[ 4328.256522] dump_stack+0x99/0xcb
[ 4328.259336] __warn.cold.11+0xe/0x55
[ 4328.261944] ? virtio_fs_enqueue_req+0xa86/0xdb0 [virtiofs]
[ 4328.264929] report_bug+0x1af/0x260
[ 4328.266673] handle_bug+0x44/0x80
[ 4328.270439] exc_invalid_op+0x13/0x40
[ 4328.273490] asm_exc_invalid_op+0x12/0x20
[ 4328.276814] RIP: 0010:virtio_fs_enqueue_req+0xa86/0xdb0 [virtiofs]
[ 4328.281866] Code: c1 e7 05 48 03 7c 24 10 6a 00 e8 85 a4 ff ff 8d 48 01 58 41 8d 54 0d 00 e9 d2 fb ff ff 48 89 ef e8 8f 33 5e f9 e9 42 fe ff ff <0f> 0b e9 c7 fb ff ff 48 8b 7c 24 08 e8 c9 49 cf f8 0f b6 45 19
[ 4328.294322] RSP: 0018:ffff8889fbb4f9c0 EFLAGS: 00010297
[ 4328.299571] RAX: 0000000000000000 RBX: ffff8889c9ad88a8 RCX: 0000000000000003
[ 4328.305197] RDX: 0000000000000007 RSI: 0000000000000000 RDI: ffff88810575c1cc
[ 4328.308930] RBP: ffff8889fbb4fe20 R08: ffffed1020aeb83c R09: 0000000000001000
[ 4328.313548] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000008
[ 4328.318783] R13: 0000000000000004 R14: 0000000000000003 R15: ffff8889c9ad88d8
[ 4328.322338] ? virtio_fs_probe+0x14d0/0x14d0 [virtiofs]
[ 4328.324902] ? trace_hardirqs_on+0x1c/0x110
[ 4328.328759] ? make_kprojid+0x20/0x20
[ 4328.331336] ? __is_kernel_percpu_address+0x63/0x1e0
[ 4328.333882] ? __module_address+0x3f/0x370
[ 4328.337281] ? lockdep_hardirqs_on_prepare+0x4d0/0x4d0
[ 4328.341248] ? virtio_fs_wake_pending_and_unlock+0x18b/0x610 [virtiofs]
[ 4328.345799] ? lock_downgrade+0x730/0x730
[ 4328.348017] ? lock_acquire+0x17f/0x7e0
[ 4328.350546] ? fuse_simple_request+0x233/0x9f0 [fuse]
[ 4328.355082] ? rcu_read_unlock+0x40/0x40
[ 4328.358741] virtio_fs_wake_pending_and_unlock+0x1f0/0x610 [virtiofs]
[ 4328.362663] ? queue_request_and_unlock+0x11e/0x290 [fuse]
[ 4328.366070] fuse_simple_request+0x3b2/0x9f0 [fuse]
[ 4328.368684] fuse_do_ioctl+0x6c6/0x1280 [fuse]
[ 4328.371398] ? fuse_readahead+0x1410/0x1410 [fuse]
[ 4328.373750] ? hrtimer_forward+0x1b0/0x1b0
[ 4328.375807] ? hrtimer_cancel+0x20/0x20
[ 4328.378899] ? ioctl_file_clone+0x120/0x120
[ 4328.380978] ? _raw_spin_unlock_irq+0x24/0x30
[ 4328.383097] ? fuse_allow_current_process+0x235/0x2a0 [fuse]
[ 4328.387317] __x64_sys_ioctl+0x128/0x190
[ 4328.390560] do_syscall_64+0x33/0x40
[ 4328.393175] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 4328.396953] RIP: 0033:0x7f112963478d
[ 4328.399000] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
[ 4328.411726] RSP: 002b:00007ffd75a76ea8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 4328.417652] RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007f112963478d
[ 4328.422766] RDX: 0000000040000000 RSI: 000000005a004000 RDI: 0000000000000157
[ 4328.427831] RBP: 0000000000000010 R08: 00000000000000a6 R09: 000000002e2e2e2e
[ 4328.433501] R10: fffffffffffffffc R11: 0000000000000246 R12: 0000000000000002
[ 4328.438662] R13: 00007f1129c8e058 R14: 00007f1129d216c0 R15: 00007f1129c8e000
[ 4328.443667] irq event stamp: 0
[ 4328.446682] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 4328.451788] hardirqs last disabled at (0): [<ffffffffb8fa08d7>] copy_process+0x18a7/0x5f00
[ 4328.456792] softirqs last enabled at (0): [<ffffffffb8fa0913>] copy_process+0x18e3/0x5f00
[ 4328.462852] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 4328.467521] ---[ end trace d6b440e9dac66d6a ]---
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs)
2020-10-02 16:28 virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs) Qian Cai
@ 2020-10-03 2:44 ` Qian Cai
2020-10-04 14:31 ` Vivek Goyal
0 siblings, 1 reply; 5+ messages in thread
From: Qian Cai @ 2020-10-03 2:44 UTC (permalink / raw)
To: Vivek Goyal, Stefan Hajnoczi, Miklos Szeredi
Cc: linux-fsdevel, linux-kernel, virtio-fs
On Fri, 2020-10-02 at 12:28 -0400, Qian Cai wrote:
> Running some fuzzing on virtiofs from a non-privileged user could trigger a
> warning in virtio_fs_enqueue_req():
>
> WARN_ON(out_sgs + in_sgs != total_sgs);
Okay, I can reproduce this after running for a few hours:
out_sgs = 3, in_sgs = 2, total_sgs = 6
and this time from flush_bg_queue() instead of fuse_simple_request().
From the log, the last piece of code is:
ftruncate(fd=186, length=4)
which is a test file on virtiofs:
[main] testfile fd:186 filename:trinity-testfile3 flags:2 fopened:1 fcntl_flags:2000 global:1
[main] start: 0x7f47c1199000 size:4KB name: trinity-testfile3 global:1
[ 9863.468502] WARNING: CPU: 16 PID: 286083 at fs/fuse/virtio_fs.c:1152 virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
[ 9863.474442] Modules linked in: dlci 8021q garp mrp bridge stp llc ieee802154_socket ieee802154 vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock mpls_router vmw_vmci ip_tunnel as
[ 9863.474555] ata_piix fuse serio_raw libata e1000 sunrpc dm_mirror dm_region_hash dm_log dm_mod
[ 9863.535805] CPU: 16 PID: 286083 Comm: trinity-c5 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #2
[ 9863.544368] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 9863.550129] RIP: 0010:virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
[ 9863.552998] Code: 60 09 23 d9 e9 44 fa ff ff e8 56 09 23 d9 e9 70 fa ff ff 48 89 cf 48 89 4c 24 08 e8 44 09 23 d9 48 8b 4c 24 08 e9 7c fa ff ff <0f> 0b 48 c7 c7 c0 85 60 c0 44 89 e1 44 89 fa 44 89 ee e8 e3 b7
[ 9863.561720] RSP: 0018:ffff888a696ef6f8 EFLAGS: 00010202
[ 9863.565420] RAX: 0000000000000000 RBX: ffff88892e030008 RCX: 0000000000000000
[ 9863.568735] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff888a696ef8ac
[ 9863.572037] RBP: ffff888a49d03d30 R08: ffffed114d2ddf18 R09: ffff888a696ef8a0
[ 9863.575383] R10: ffff888a696ef8bf R11: ffffed114d2ddf17 R12: 0000000000000006
[ 9863.578668] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002
[ 9863.581971] FS: 00007f47c12f5740(0000) GS:ffff888a7f800000(0000) knlGS:0000000000000000
[ 9863.585752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9863.590232] CR2: 0000000000000000 CR3: 0000000a63570005 CR4: 0000000000770ee0
[ 9863.594698] DR0: 00007f6642e43000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9863.598521] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 9863.601861] PKRU: 55555540
[ 9863.603173] Call Trace:
[ 9863.604382] ? virtio_fs_probe+0x13e0/0x13e0 [virtiofs]
[ 9863.606838] ? is_bpf_text_address+0x21/0x30
[ 9863.608869] ? kernel_text_address+0x125/0x140
[ 9863.610962] ? __kernel_text_address+0xe/0x30
[ 9863.613117] ? unwind_get_return_address+0x5f/0xa0
[ 9863.615427] ? create_prof_cpu_mask+0x20/0x20
[ 9863.617435] ? _raw_write_lock_irqsave+0xe0/0xe0
[ 9863.619627] virtio_fs_wake_pending_and_unlock+0x1ea/0x610 [virtiofs]
[ 9863.622638] ? queue_request_and_unlock+0x115/0x280 [fuse]
[ 9863.625224] flush_bg_queue+0x24c/0x3e0 [fuse]
[ 9863.627325] fuse_simple_background+0x3d7/0x6c0 [fuse]
[ 9863.629735] fuse_send_writepage+0x173/0x420 [fuse]
[ 9863.632031] fuse_flush_writepages+0x1fe/0x330 [fuse]
[ 9863.634463] ? make_kgid+0x13/0x20
[ 9863.636064] ? fuse_change_attributes_common+0x2de/0x940 [fuse]
[ 9863.638850] fuse_do_setattr+0xe84/0x13c0 [fuse]
[ 9863.641024] ? migrate_swap_stop+0x8d1/0x920
[ 9863.643041] ? fuse_flush_times+0x390/0x390 [fuse]
[ 9863.645347] ? avc_has_perm_noaudit+0x390/0x390
[ 9863.647465] fuse_setattr+0x197/0x400 [fuse]
[ 9863.649466] notify_change+0x744/0xda0
[ 9863.651247] ? __down_timeout+0x2a0/0x2a0
[ 9863.653125] ? do_truncate+0xe2/0x180
[ 9863.654854] do_truncate+0xe2/0x180
[ 9863.656509] ? __x64_sys_openat2+0x1c0/0x1c0
[ 9863.658512] ? alarm_setitimer+0xa0/0x110
[ 9863.660418] do_sys_ftruncate+0x1ee/0x2c0
[ 9863.662311] do_syscall_64+0x33/0x40
[ 9863.663980] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9863.666384] RIP: 0033:0x7f47c0c0878d
[ 9863.668061] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
[ 9863.676717] RSP: 002b:00007fff515c2598 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
[ 9863.680226] RAX: ffffffffffffffda RBX: 000000000000004d RCX: 00007f47c0c0878d
[ 9863.688055] RDX: 0000000000800000 RSI: 0000000000000004 RDI: 00000000000000ba
[ 9863.693672] RBP: 000000000000004d R08: 000000000000003a R09: 0000000000000001
[ 9863.699423] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000002
[ 9863.708897] R13: 00007f47c12cb058 R14: 00007f47c12f56c0 R15: 00007f47c12cb000
[ 9863.713106] CPU: 16 PID: 286083 Comm: trinity-c5 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #2
[ 9863.717465] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 9863.721389] Call Trace:
[ 9863.722547] dump_stack+0x7c/0xa2
[ 9863.724110] __warn.cold.13+0xe/0x47
[ 9863.725804] ? virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
[ 9863.728427] report_bug+0x1af/0x260
[ 9863.730054] handle_bug+0x44/0x80
[ 9863.731652] exc_invalid_op+0x13/0x40
[ 9863.734911] asm_exc_invalid_op+0x12/0x20
[ 9863.736940] RIP: 0010:virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
[ 9863.739833] Code: 60 09 23 d9 e9 44 fa ff ff e8 56 09 23 d9 e9 70 fa ff ff 48 89 cf 48 89 4c 24 08 e8 44 09 23 d9 48 8b 4c 24 08 e9 7c fa ff ff <0f> 0b 48 c7 c7 c0 85 60 c0 44 89 e1 44 89 fa 44 89 ee e8 e3 b7
[ 9863.748519] RSP: 0018:ffff888a696ef6f8 EFLAGS: 00010202
[ 9863.750935] RAX: 0000000000000000 RBX: ffff88892e030008 RCX: 0000000000000000
[ 9863.754247] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff888a696ef8ac
[ 9863.760885] RBP: ffff888a49d03d30 R08: ffffed114d2ddf18 R09: ffff888a696ef8a0
[ 9863.764814] R10: ffff888a696ef8bf R11: ffffed114d2ddf17 R12: 0000000000000006
[ 9863.768148] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002
[ 9863.771492] ? virtio_fs_probe+0x13e0/0x13e0 [virtiofs]
[ 9863.773950] ? is_bpf_text_address+0x21/0x30
[ 9863.775979] ? kernel_text_address+0x125/0x140
[ 9863.778061] ? __kernel_text_address+0xe/0x30
[ 9863.780124] ? unwind_get_return_address+0x5f/0xa0
[ 9863.782395] ? create_prof_cpu_mask+0x20/0x20
[ 9863.784451] ? _raw_write_lock_irqsave+0xe0/0xe0
[ 9863.786602] virtio_fs_wake_pending_and_unlock+0x1ea/0x610 [virtiofs]
[ 9863.789614] ? queue_request_and_unlock+0x115/0x280 [fuse]
[ 9863.792178] flush_bg_queue+0x24c/0x3e0 [fuse]
[ 9863.796678] fuse_simple_background+0x3d7/0x6c0 [fuse]
[ 9863.802329] fuse_send_writepage+0x173/0x420 [fuse]
[ 9863.808342] fuse_flush_writepages+0x1fe/0x330 [fuse]
[ 9863.812086] ? make_kgid+0x13/0x20
[ 9863.813681] ? fuse_change_attributes_common+0x2de/0x940 [fuse]
[ 9863.816465] fuse_do_setattr+0xe84/0x13c0 [fuse]
[ 9863.819633] ? migrate_swap_stop+0x8d1/0x920
[ 9863.824285] ? fuse_flush_times+0x390/0x390 [fuse]
[ 9863.827331] ? avc_has_perm_noaudit+0x390/0x390
[ 9863.875278] fuse_setattr+0x197/0x400 [fuse]
[ 9863.878496] notify_change+0x744/0xda0
[ 9863.880640] ? __down_timeout+0x2a0/0x2a0
[ 9863.882960] ? do_truncate+0xe2/0x180
[ 9863.886311] do_truncate+0xe2/0x180
[ 9863.888392] ? __x64_sys_openat2+0x1c0/0x1c0
[ 9863.890418] ? alarm_setitimer+0xa0/0x110
[ 9863.894430] do_sys_ftruncate+0x1ee/0x2c0
[ 9863.896468] do_syscall_64+0x33/0x40
[ 9863.898167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9863.901089] RIP: 0033:0x7f47c0c0878d
[ 9863.903447] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
[ 9863.914356] RSP: 002b:00007fff515c2598 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
[ 9863.917998] RAX: ffffffffffffffda RBX: 000000000000004d RCX: 00007f47c0c0878d
[ 9863.921364] RDX: 0000000000800000 RSI: 0000000000000004 RDI: 00000000000000ba
[ 9863.928285] RBP: 000000000000004d R08: 000000000000003a R09: 0000000000000001
[ 9863.932523] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000002
[ 9863.935835] R13: 00007f47c12cb058 R14: 00007f47c12f56c0 R15: 00007f47c12cb000
[ 9863.939183] ---[ end trace f6f5d958c186bcee ]---
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs)
2020-10-03 2:44 ` Qian Cai
@ 2020-10-04 14:31 ` Vivek Goyal
2020-10-06 9:04 ` Stefan Hajnoczi
0 siblings, 1 reply; 5+ messages in thread
From: Vivek Goyal @ 2020-10-04 14:31 UTC (permalink / raw)
To: Qian Cai
Cc: Stefan Hajnoczi, Miklos Szeredi, linux-fsdevel, linux-kernel, virtio-fs
On Fri, Oct 02, 2020 at 10:44:37PM -0400, Qian Cai wrote:
> On Fri, 2020-10-02 at 12:28 -0400, Qian Cai wrote:
> > Running some fuzzing on virtiofs from a non-privileged user could trigger a
> > warning in virtio_fs_enqueue_req():
> >
> > WARN_ON(out_sgs + in_sgs != total_sgs);
>
> Okay, I can reproduce this after running for a few hours:
>
> out_sgs = 3, in_sgs = 2, total_sgs = 6
Thanks. I can also reproduce it simply by calling.
ioctl(fd, 0x5a004000, buf);
I think following WARN_ON() is not correct.
WARN_ON(out_sgs + in_sgs != total_sgs)
toal_sgs should actually be max sgs. It looks at ap->num_pages and
counts one sg for each page. And it assumes that same number of
pages will be used both for input and output.
But there are no such guarantees. With above ioctl() call, I noticed
we are using 2 pages for input (out_sgs) and one page for output (in_sgs).
So out_sgs=4, in_sgs=3 and total_sgs=8 and warning triggers.
I think total sgs is actually max number of sgs and warning
should probably be.
WARN_ON(out_sgs + in_sgs > total_sgs)
Stefan, WDYT?
I will send a patch for this.
Thanks
Vivek
>
> and this time from flush_bg_queue() instead of fuse_simple_request().
>
> From the log, the last piece of code is:
>
> ftruncate(fd=186, length=4)
>
> which is a test file on virtiofs:
>
> [main] testfile fd:186 filename:trinity-testfile3 flags:2 fopened:1 fcntl_flags:2000 global:1
> [main] start: 0x7f47c1199000 size:4KB name: trinity-testfile3 global:1
>
>
> [ 9863.468502] WARNING: CPU: 16 PID: 286083 at fs/fuse/virtio_fs.c:1152 virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
> [ 9863.474442] Modules linked in: dlci 8021q garp mrp bridge stp llc ieee802154_socket ieee802154 vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock mpls_router vmw_vmci ip_tunnel as
> [ 9863.474555] ata_piix fuse serio_raw libata e1000 sunrpc dm_mirror dm_region_hash dm_log dm_mod
> [ 9863.535805] CPU: 16 PID: 286083 Comm: trinity-c5 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #2
> [ 9863.544368] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
> [ 9863.550129] RIP: 0010:virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
> [ 9863.552998] Code: 60 09 23 d9 e9 44 fa ff ff e8 56 09 23 d9 e9 70 fa ff ff 48 89 cf 48 89 4c 24 08 e8 44 09 23 d9 48 8b 4c 24 08 e9 7c fa ff ff <0f> 0b 48 c7 c7 c0 85 60 c0 44 89 e1 44 89 fa 44 89 ee e8 e3 b7
> [ 9863.561720] RSP: 0018:ffff888a696ef6f8 EFLAGS: 00010202
> [ 9863.565420] RAX: 0000000000000000 RBX: ffff88892e030008 RCX: 0000000000000000
> [ 9863.568735] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff888a696ef8ac
> [ 9863.572037] RBP: ffff888a49d03d30 R08: ffffed114d2ddf18 R09: ffff888a696ef8a0
> [ 9863.575383] R10: ffff888a696ef8bf R11: ffffed114d2ddf17 R12: 0000000000000006
> [ 9863.578668] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002
> [ 9863.581971] FS: 00007f47c12f5740(0000) GS:ffff888a7f800000(0000) knlGS:0000000000000000
> [ 9863.585752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 9863.590232] CR2: 0000000000000000 CR3: 0000000a63570005 CR4: 0000000000770ee0
> [ 9863.594698] DR0: 00007f6642e43000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 9863.598521] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
> [ 9863.601861] PKRU: 55555540
> [ 9863.603173] Call Trace:
> [ 9863.604382] ? virtio_fs_probe+0x13e0/0x13e0 [virtiofs]
> [ 9863.606838] ? is_bpf_text_address+0x21/0x30
> [ 9863.608869] ? kernel_text_address+0x125/0x140
> [ 9863.610962] ? __kernel_text_address+0xe/0x30
> [ 9863.613117] ? unwind_get_return_address+0x5f/0xa0
> [ 9863.615427] ? create_prof_cpu_mask+0x20/0x20
> [ 9863.617435] ? _raw_write_lock_irqsave+0xe0/0xe0
> [ 9863.619627] virtio_fs_wake_pending_and_unlock+0x1ea/0x610 [virtiofs]
> [ 9863.622638] ? queue_request_and_unlock+0x115/0x280 [fuse]
> [ 9863.625224] flush_bg_queue+0x24c/0x3e0 [fuse]
> [ 9863.627325] fuse_simple_background+0x3d7/0x6c0 [fuse]
> [ 9863.629735] fuse_send_writepage+0x173/0x420 [fuse]
> [ 9863.632031] fuse_flush_writepages+0x1fe/0x330 [fuse]
> [ 9863.634463] ? make_kgid+0x13/0x20
> [ 9863.636064] ? fuse_change_attributes_common+0x2de/0x940 [fuse]
> [ 9863.638850] fuse_do_setattr+0xe84/0x13c0 [fuse]
> [ 9863.641024] ? migrate_swap_stop+0x8d1/0x920
> [ 9863.643041] ? fuse_flush_times+0x390/0x390 [fuse]
> [ 9863.645347] ? avc_has_perm_noaudit+0x390/0x390
> [ 9863.647465] fuse_setattr+0x197/0x400 [fuse]
> [ 9863.649466] notify_change+0x744/0xda0
> [ 9863.651247] ? __down_timeout+0x2a0/0x2a0
> [ 9863.653125] ? do_truncate+0xe2/0x180
> [ 9863.654854] do_truncate+0xe2/0x180
> [ 9863.656509] ? __x64_sys_openat2+0x1c0/0x1c0
> [ 9863.658512] ? alarm_setitimer+0xa0/0x110
> [ 9863.660418] do_sys_ftruncate+0x1ee/0x2c0
> [ 9863.662311] do_syscall_64+0x33/0x40
> [ 9863.663980] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 9863.666384] RIP: 0033:0x7f47c0c0878d
> [ 9863.668061] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
> [ 9863.676717] RSP: 002b:00007fff515c2598 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
> [ 9863.680226] RAX: ffffffffffffffda RBX: 000000000000004d RCX: 00007f47c0c0878d
> [ 9863.688055] RDX: 0000000000800000 RSI: 0000000000000004 RDI: 00000000000000ba
> [ 9863.693672] RBP: 000000000000004d R08: 000000000000003a R09: 0000000000000001
> [ 9863.699423] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000002
> [ 9863.708897] R13: 00007f47c12cb058 R14: 00007f47c12f56c0 R15: 00007f47c12cb000
> [ 9863.713106] CPU: 16 PID: 286083 Comm: trinity-c5 Kdump: loaded Not tainted 5.9.0-rc7-next-20201002+ #2
> [ 9863.717465] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
> [ 9863.721389] Call Trace:
> [ 9863.722547] dump_stack+0x7c/0xa2
> [ 9863.724110] __warn.cold.13+0xe/0x47
> [ 9863.725804] ? virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
> [ 9863.728427] report_bug+0x1af/0x260
> [ 9863.730054] handle_bug+0x44/0x80
> [ 9863.731652] exc_invalid_op+0x13/0x40
> [ 9863.734911] asm_exc_invalid_op+0x12/0x20
> [ 9863.736940] RIP: 0010:virtio_fs_enqueue_req+0xd36/0xde0 [virtiofs]
> [ 9863.739833] Code: 60 09 23 d9 e9 44 fa ff ff e8 56 09 23 d9 e9 70 fa ff ff 48 89 cf 48 89 4c 24 08 e8 44 09 23 d9 48 8b 4c 24 08 e9 7c fa ff ff <0f> 0b 48 c7 c7 c0 85 60 c0 44 89 e1 44 89 fa 44 89 ee e8 e3 b7
> [ 9863.748519] RSP: 0018:ffff888a696ef6f8 EFLAGS: 00010202
> [ 9863.750935] RAX: 0000000000000000 RBX: ffff88892e030008 RCX: 0000000000000000
> [ 9863.754247] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff888a696ef8ac
> [ 9863.760885] RBP: ffff888a49d03d30 R08: ffffed114d2ddf18 R09: ffff888a696ef8a0
> [ 9863.764814] R10: ffff888a696ef8bf R11: ffffed114d2ddf17 R12: 0000000000000006
> [ 9863.768148] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002
> [ 9863.771492] ? virtio_fs_probe+0x13e0/0x13e0 [virtiofs]
> [ 9863.773950] ? is_bpf_text_address+0x21/0x30
> [ 9863.775979] ? kernel_text_address+0x125/0x140
> [ 9863.778061] ? __kernel_text_address+0xe/0x30
> [ 9863.780124] ? unwind_get_return_address+0x5f/0xa0
> [ 9863.782395] ? create_prof_cpu_mask+0x20/0x20
> [ 9863.784451] ? _raw_write_lock_irqsave+0xe0/0xe0
> [ 9863.786602] virtio_fs_wake_pending_and_unlock+0x1ea/0x610 [virtiofs]
> [ 9863.789614] ? queue_request_and_unlock+0x115/0x280 [fuse]
> [ 9863.792178] flush_bg_queue+0x24c/0x3e0 [fuse]
> [ 9863.796678] fuse_simple_background+0x3d7/0x6c0 [fuse]
> [ 9863.802329] fuse_send_writepage+0x173/0x420 [fuse]
> [ 9863.808342] fuse_flush_writepages+0x1fe/0x330 [fuse]
> [ 9863.812086] ? make_kgid+0x13/0x20
> [ 9863.813681] ? fuse_change_attributes_common+0x2de/0x940 [fuse]
> [ 9863.816465] fuse_do_setattr+0xe84/0x13c0 [fuse]
> [ 9863.819633] ? migrate_swap_stop+0x8d1/0x920
> [ 9863.824285] ? fuse_flush_times+0x390/0x390 [fuse]
> [ 9863.827331] ? avc_has_perm_noaudit+0x390/0x390
> [ 9863.875278] fuse_setattr+0x197/0x400 [fuse]
> [ 9863.878496] notify_change+0x744/0xda0
> [ 9863.880640] ? __down_timeout+0x2a0/0x2a0
> [ 9863.882960] ? do_truncate+0xe2/0x180
> [ 9863.886311] do_truncate+0xe2/0x180
> [ 9863.888392] ? __x64_sys_openat2+0x1c0/0x1c0
> [ 9863.890418] ? alarm_setitimer+0xa0/0x110
> [ 9863.894430] do_sys_ftruncate+0x1ee/0x2c0
> [ 9863.896468] do_syscall_64+0x33/0x40
> [ 9863.898167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 9863.901089] RIP: 0033:0x7f47c0c0878d
> [ 9863.903447] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 08
> [ 9863.914356] RSP: 002b:00007fff515c2598 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
> [ 9863.917998] RAX: ffffffffffffffda RBX: 000000000000004d RCX: 00007f47c0c0878d
> [ 9863.921364] RDX: 0000000000800000 RSI: 0000000000000004 RDI: 00000000000000ba
> [ 9863.928285] RBP: 000000000000004d R08: 000000000000003a R09: 0000000000000001
> [ 9863.932523] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000002
> [ 9863.935835] R13: 00007f47c12cb058 R14: 00007f47c12f56c0 R15: 00007f47c12cb000
> [ 9863.939183] ---[ end trace f6f5d958c186bcee ]---
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs)
2020-10-04 14:31 ` Vivek Goyal
@ 2020-10-06 9:04 ` Stefan Hajnoczi
2020-10-06 13:06 ` Vivek Goyal
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Hajnoczi @ 2020-10-06 9:04 UTC (permalink / raw)
To: Vivek Goyal
Cc: Qian Cai, Miklos Szeredi, linux-fsdevel, linux-kernel, virtio-fs
[-- Attachment #1: Type: text/plain, Size: 1600 bytes --]
On Sun, Oct 04, 2020 at 10:31:19AM -0400, Vivek Goyal wrote:
> On Fri, Oct 02, 2020 at 10:44:37PM -0400, Qian Cai wrote:
> > On Fri, 2020-10-02 at 12:28 -0400, Qian Cai wrote:
> > > Running some fuzzing on virtiofs from a non-privileged user could trigger a
> > > warning in virtio_fs_enqueue_req():
> > >
> > > WARN_ON(out_sgs + in_sgs != total_sgs);
> >
> > Okay, I can reproduce this after running for a few hours:
> >
> > out_sgs = 3, in_sgs = 2, total_sgs = 6
>
> Thanks. I can also reproduce it simply by calling.
>
> ioctl(fd, 0x5a004000, buf);
>
> I think following WARN_ON() is not correct.
>
> WARN_ON(out_sgs + in_sgs != total_sgs)
>
> toal_sgs should actually be max sgs. It looks at ap->num_pages and
> counts one sg for each page. And it assumes that same number of
> pages will be used both for input and output.
>
> But there are no such guarantees. With above ioctl() call, I noticed
> we are using 2 pages for input (out_sgs) and one page for output (in_sgs).
>
> So out_sgs=4, in_sgs=3 and total_sgs=8 and warning triggers.
>
> I think total sgs is actually max number of sgs and warning
> should probably be.
>
> WARN_ON(out_sgs + in_sgs > total_sgs)
>
> Stefan, WDYT?
It should be possible to calculate total_sgs precisely (not a maximum).
Treating it as a maximum could hide bugs.
Maybe sg_count_fuse_req() should count in_args/out_args[numargs -
1].size pages instead of adding ap->num_pages.
Do you have the details of struct fuse_req and struct fuse_args_pages
fields for the ioctl in question?
Thanks,
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs)
2020-10-06 9:04 ` Stefan Hajnoczi
@ 2020-10-06 13:06 ` Vivek Goyal
0 siblings, 0 replies; 5+ messages in thread
From: Vivek Goyal @ 2020-10-06 13:06 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Qian Cai, Miklos Szeredi, linux-fsdevel, linux-kernel, virtio-fs
On Tue, Oct 06, 2020 at 10:04:27AM +0100, Stefan Hajnoczi wrote:
> On Sun, Oct 04, 2020 at 10:31:19AM -0400, Vivek Goyal wrote:
> > On Fri, Oct 02, 2020 at 10:44:37PM -0400, Qian Cai wrote:
> > > On Fri, 2020-10-02 at 12:28 -0400, Qian Cai wrote:
> > > > Running some fuzzing on virtiofs from a non-privileged user could trigger a
> > > > warning in virtio_fs_enqueue_req():
> > > >
> > > > WARN_ON(out_sgs + in_sgs != total_sgs);
> > >
> > > Okay, I can reproduce this after running for a few hours:
> > >
> > > out_sgs = 3, in_sgs = 2, total_sgs = 6
> >
> > Thanks. I can also reproduce it simply by calling.
> >
> > ioctl(fd, 0x5a004000, buf);
> >
> > I think following WARN_ON() is not correct.
> >
> > WARN_ON(out_sgs + in_sgs != total_sgs)
> >
> > toal_sgs should actually be max sgs. It looks at ap->num_pages and
> > counts one sg for each page. And it assumes that same number of
> > pages will be used both for input and output.
> >
> > But there are no such guarantees. With above ioctl() call, I noticed
> > we are using 2 pages for input (out_sgs) and one page for output (in_sgs).
> >
> > So out_sgs=4, in_sgs=3 and total_sgs=8 and warning triggers.
> >
> > I think total sgs is actually max number of sgs and warning
> > should probably be.
> >
> > WARN_ON(out_sgs + in_sgs > total_sgs)
> >
> > Stefan, WDYT?
>
> It should be possible to calculate total_sgs precisely (not a maximum).
> Treating it as a maximum could hide bugs.
I thought about calculating total_sgs as well. Then became little lazy.
I will redo the patch and then calculate total_sgs precisely.
>
> Maybe sg_count_fuse_req() should count in_args/out_args[numargs -
> 1].size pages instead of adding ap->num_pages.
That should work, I guess. Will try.
Vivek
>
> Do you have the details of struct fuse_req and struct fuse_args_pages
> fields for the ioctl in question?
>
> Thanks,
> Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-10-06 13:06 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-02 16:28 virtiofs: WARN_ON(out_sgs + in_sgs != total_sgs) Qian Cai
2020-10-03 2:44 ` Qian Cai
2020-10-04 14:31 ` Vivek Goyal
2020-10-06 9:04 ` Stefan Hajnoczi
2020-10-06 13:06 ` Vivek Goyal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).