* BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-04 22:48 Jean-Philippe Menil
2017-06-05 2:08 ` Michael S. Tsirkin
0 siblings, 1 reply; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-04 22:48 UTC (permalink / raw)
To: netdev; +Cc: mst, jasowang
Hi,
while playing with xdp and ebpf, i'm hitting the following:
[ 309.993136]
==================================================================
[ 309.994735] BUG: KASAN: use-after-free in
free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
[ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
[ 310.000650]
[ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
[ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[ 310.006495] Call Trace:
[ 310.007610] dump_stack+0xb8/0x14c
[ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
[ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
[ 310.011203] print_address_description+0x6f/0x280
[ 310.012416] kasan_report+0x27a/0x370
[ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
[ 310.014900] __asan_report_load8_noabort+0x19/0x20
[ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
[ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
[ 310.018759] ? packet_rcv+0x20d0/0x20d0
[ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
[ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
[ 310.022413] ? default_device_exit+0x2d0/0x2d0
[ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
[ 310.024874] ? update_load_avg+0x1281/0x29f0
[ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
[ 310.027247] ? validate_xmit_skb_list+0x100/0x100
[ 310.028470] ? validate_xmit_skb+0x7f/0xc10
[ 310.029731] ? netif_skb_features+0x920/0x920
[ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
[ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
[ 310.037782] sch_direct_xmit+0x2eb/0x7a0
[ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
[ 310.041980] ? netdev_pick_tx+0x212/0x2b0
[ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
[ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
[ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
[ 310.048301] ? update_stack_state+0x402/0x780
[ 310.049307] ? account_entity_enqueue+0x730/0x730
[ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
[ 310.051286] ? update_curr_fair+0x70/0x70
[ 310.052206] ? enqueue_entity+0x2450/0x2450
[ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.054082] ? dequeue_entity+0x27a/0x1520
[ 310.054967] ? bpf_prog_alloc+0x320/0x320
[ 310.055822] ? yield_to_task_fair+0x110/0x110
[ 310.056708] ? set_next_entity+0x2f2/0xa90
[ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
[ 310.058457] dev_queue_xmit+0x10/0x20
[ 310.059298] ip_finish_output2+0xacf/0x12a0
[ 310.060160] ? dequeue_entity+0x1520/0x1520
[ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
[ 310.065078] ? ring_buffer_set_clock+0x50/0x50
[ 310.066677] ? __switch_to+0x685/0xda0
[ 310.068166] ? load_balance+0x38f0/0x38f0
[ 310.069544] ? compat_start_thread+0x80/0x80
[ 310.070989] ? trace_find_cmdline+0x60/0x60
[ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
[ 310.073579] ip_finish_output+0x407/0x880
[ 310.074441] ? ip_finish_output+0x407/0x880
[ 310.075255] ? update_stack_state+0x402/0x780
[ 310.076076] ip_output+0x1c0/0x640
[ 310.076843] ? ip_mc_output+0x1350/0x1350
[ 310.077642] ? __sk_dst_check+0x164/0x370
[ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
[ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
[ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
[ 310.081097] ? memcpy+0x45/0x50
[ 310.081850] ? __copy_skb_header+0x1fa/0x280
[ 310.082676] ip_local_out+0x70/0x90
[ 310.083448] ip_queue_xmit+0x8a1/0x22a0
[ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
[ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
[ 310.085884] tcp_transmit_skb+0x187a/0x3e00
[ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
[ 310.087524] ? sock_sendmsg+0xba/0xf0
[ 310.088298] ? __vfs_write+0x4e0/0x960
[ 310.089074] ? vfs_write+0x155/0x4b0
[ 310.089838] ? SyS_write+0xf7/0x240
[ 310.090593] ? do_syscall_64+0x235/0x5b0
[ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.094690] ? sock_sendmsg+0xba/0xf0
[ 310.096133] ? do_syscall_64+0x235/0x5b0
[ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
[ 310.100539] ? radix_tree_lookup+0xd/0x10
[ 310.101894] ? get_work_pool+0xcd/0x150
[ 310.103216] ? check_flush_dependency+0x330/0x330
[ 310.104113] tcp_write_xmit+0x498/0x52a0
[ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
[ 310.105729] ? kasan_kmalloc+0xad/0xe0
[ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
[ 310.107331] ? memset+0x31/0x40
[ 310.108070] ? __check_object_size+0x22e/0x55c
[ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
[ 310.109690] ? check_stack_object+0x120/0x120
[ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
[ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
[ 310.112159] tcp_push+0x47c/0xbd0
[ 310.112912] ? copy_from_iter_full+0x21e/0xc70
[ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
[ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
[ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
[ 310.116324] tcp_sendmsg+0xd6d/0x43f0
[ 310.117106] ? tcp_sendpage+0x2170/0x2170
[ 310.117911] ? set_fd_set.part.1+0x50/0x50
[ 310.118718] ? remove_wait_queue+0x196/0x3b0
[ 310.119535] ? set_fd_set.part.1+0x50/0x50
[ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
[ 310.121224] ? __wake_up+0x44/0x50
[ 310.121985] ? n_tty_read+0x9f9/0x19d0
[ 310.122898] ? __check_object_size+0x22e/0x55c
[ 310.125380] inet_sendmsg+0x111/0x590
[ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
[ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
[ 310.129817] sock_sendmsg+0xba/0xf0
[ 310.131110] sock_write_iter+0x2e4/0x6a0
[ 310.132433] ? core_sys_select+0x47d/0x780
[ 310.133779] ? sock_sendmsg+0xf0/0xf0
[ 310.134591] __vfs_write+0x4e0/0x960
[ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
[ 310.136160] ? __vfs_read+0x950/0x950
[ 310.136931] ? rw_verify_area+0xbd/0x2b0
[ 310.137711] vfs_write+0x155/0x4b0
[ 310.138454] SyS_write+0xf7/0x240
[ 310.139183] ? SyS_read+0x240/0x240
[ 310.139922] ? SyS_read+0x240/0x240
[ 310.140649] do_syscall_64+0x235/0x5b0
[ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
[ 310.142204] ? syscall_return_slowpath+0x240/0x240
[ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
[ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
[ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
[ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
[ 310.146257] RIP: 0033:0x7f6f868fb070
[ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
00007f6f868fb070
[ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
0000000000000003
[ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
0000000000003000
[ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
0000000000000000
[ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
0000000000000003
[ 310.153578]
[ 310.156362] Allocated by task 483:
[ 310.157812] save_stack_trace+0x1b/0x20
[ 310.159274] save_stack+0x43/0xd0
[ 310.160663] kasan_kmalloc+0xad/0xe0
[ 310.161943] __kmalloc+0x105/0x230
[ 310.163233] __vring_new_virtqueue+0xd1/0xee0
[ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
[ 310.165536] setup_vq+0x136/0x620
[ 310.166286] vp_setup_vq+0x13d/0x6d0
[ 310.167059] vp_find_vqs_msix+0x46c/0xb50
[ 310.167855] vp_find_vqs+0x71/0x410
[ 310.168641] vp_modern_find_vqs+0x21/0x140
[ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
[ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
[ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
[ 310.172077] dev_change_xdp_fd+0x1ca/0x420
[ 310.172918] do_setlink+0x2c33/0x3bc0
[ 310.173703] rtnl_setlink+0x245/0x380
[ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
[ 310.175344] netlink_rcv_skb+0x213/0x450
[ 310.176166] rtnetlink_rcv+0x28/0x30
[ 310.176990] netlink_unicast+0x4a0/0x6c0
[ 310.177807] netlink_sendmsg+0x9ec/0xe50
[ 310.178646] sock_sendmsg+0xba/0xf0
[ 310.179435] SYSC_sendto+0x31d/0x620
[ 310.180229] SyS_sendto+0xe/0x10
[ 310.181004] do_syscall_64+0x235/0x5b0
[ 310.181783] return_from_SYSCALL_64+0x0/0x6a
[ 310.182595]
[ 310.183217] Freed by task 483:
[ 310.183934] save_stack_trace+0x1b/0x20
[ 310.184801] save_stack+0x43/0xd0
[ 310.187187] kasan_slab_free+0x72/0xc0
[ 310.188530] kfree+0x94/0x1a0
[ 310.189797] vring_del_virtqueue+0x19a/0x430
[ 310.191221] del_vq+0x11c/0x250
[ 310.192474] vp_del_vqs+0x379/0xc30
[ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
[ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
[ 310.196231] dev_change_xdp_fd+0x37c/0x420
[ 310.197072] do_setlink+0x2c33/0x3bc0
[ 310.197804] rtnl_setlink+0x245/0x380
[ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
[ 310.199283] netlink_rcv_skb+0x213/0x450
[ 310.200036] rtnetlink_rcv+0x28/0x30
[ 310.200754] netlink_unicast+0x4a0/0x6c0
[ 310.201496] netlink_sendmsg+0x9ec/0xe50
[ 310.202236] sock_sendmsg+0xba/0xf0
[ 310.202947] SYSC_sendto+0x31d/0x620
[ 310.203660] SyS_sendto+0xe/0x10
[ 310.204340] do_syscall_64+0x235/0x5b0
[ 310.205050] return_from_SYSCALL_64+0x0/0x6a
[ 310.205792]
[ 310.206350] The buggy address belongs to the object at ffff88006aa64200
[ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
[ 310.208149] The buggy address is located 32 bytes inside of
[ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
[ 310.209929] The buggy address belongs to the page:
[ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping:
(null) index:0x0 compound_mapcount: 0
[ 310.212499] flags: 0x1ffff8000008100(slab|head)
[ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
0000000100030003
[ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
0000000000000000
[ 310.215635] page dumped because: kasan: bad access detected
[ 310.218989]
[ 310.220398] Memory state around the buggy address:
[ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 310.227400] ^
[ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 310.230639]
==================================================================
[ 310.231788] Disabling lock debugging due to kernel taint
[ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
[ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user
memory access
[ 310.239138] general protection fault: 0000 [#1] SMP KASAN
[ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse irqbypass
i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw
virtio_balloon pata_acpi virtio_net virtio_blk
[ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B
4.12.0-rc3+ #2
[ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
[ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
[ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
[ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000004
[ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
0000000000000020
[ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
ffff880066b3c400
[ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
ffff880066b3c400
[ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
ffff88006afc9001
[ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
knlGS:0000000000000000
[ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
00000000000006f0
[ 310.266178] Call Trace:
[ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
[ 310.268453] ? packet_rcv+0x20d0/0x20d0
[ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
[ 310.270762] ? default_device_exit+0x2d0/0x2d0
[ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
[ 310.273076] ? update_load_avg+0x1281/0x29f0
[ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
[ 310.275295] ? validate_xmit_skb_list+0x100/0x100
[ 310.276425] ? validate_xmit_skb+0x7f/0xc10
[ 310.277548] ? rb_insert_color+0x1590/0x1590
[ 310.280172] ? netif_skb_features+0x920/0x920
[ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
[ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
[ 310.283494] sch_direct_xmit+0x2eb/0x7a0
[ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
[ 310.286448] ? netdev_pick_tx+0x212/0x2b0
[ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
[ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
[ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
[ 310.291837] ? update_curr+0x1ef/0x750
[ 310.292826] ? update_stack_state+0x402/0x780
[ 310.293827] ? account_entity_enqueue+0x730/0x730
[ 310.294831] ? update_stack_state+0x402/0x780
[ 310.295818] ? update_curr_fair+0x70/0x70
[ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.297693] ? dequeue_entity+0x27a/0x1520
[ 310.298591] ? bpf_prog_alloc+0x320/0x320
[ 310.299484] ? yield_to_task_fair+0x110/0x110
[ 310.300385] ? unwind_dump+0x4e0/0x4e0
[ 310.301246] ? __free_insn_slot+0x600/0x600
[ 310.302125] ? unwind_dump+0x4e0/0x4e0
[ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
[ 310.303883] dev_queue_xmit+0x10/0x20
[ 310.304711] ip_finish_output2+0xacf/0x12a0
[ 310.305558] ? dequeue_entity+0x1520/0x1520
[ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
[ 310.307320] ? save_stack_trace+0x1b/0x20
[ 310.308133] ? save_stack+0x43/0xd0
[ 310.309081] ? kasan_slab_free+0x72/0xc0
[ 310.310614] ? kfree_skbmem+0xb6/0x1d0
[ 310.311406] ? tcp_ack+0x2730/0x7450
[ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
[ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
[ 310.313769] ? __release_sock+0x14a/0x2b0
[ 310.314550] ? release_sock+0xa8/0x270
[ 310.315330] ? inet_sendmsg+0x111/0x590
[ 310.316100] ? sock_sendmsg+0xba/0xf0
[ 310.317403] ? sock_write_iter+0x2e4/0x6a0
[ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
[ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
[ 310.320800] ? update_stack_state+0x402/0x780
[ 310.321590] ip_finish_output+0x407/0x880
[ 310.322347] ? ip_finish_output+0x407/0x880
[ 310.323138] ? update_stack_state+0x402/0x780
[ 310.323948] ip_output+0x1c0/0x640
[ 310.324661] ? ip_mc_output+0x1350/0x1350
[ 310.325415] ? __sk_dst_check+0x164/0x370
[ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
[ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
[ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
[ 310.328684] ? memcpy+0x45/0x50
[ 310.329393] ? __copy_skb_header+0x1fa/0x280
[ 310.330180] ip_local_out+0x70/0x90
[ 310.330914] ip_queue_xmit+0x8a1/0x22a0
[ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
[ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
[ 310.333298] tcp_transmit_skb+0x187a/0x3e00
[ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
[ 310.334887] ? sock_sendmsg+0xba/0xf0
[ 310.335637] ? __vfs_write+0x4e0/0x960
[ 310.336391] ? vfs_write+0x155/0x4b0
[ 310.337135] ? SyS_write+0xf7/0x240
[ 310.337861] ? do_syscall_64+0x235/0x5b0
[ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.339443] ? sock_sendmsg+0xba/0xf0
[ 310.341675] ? do_syscall_64+0x235/0x5b0
[ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
[ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
[ 310.344095] ? radix_tree_lookup+0xd/0x10
[ 310.344871] ? get_work_pool+0xcd/0x150
[ 310.345635] ? check_flush_dependency+0x330/0x330
[ 310.346466] tcp_write_xmit+0x498/0x52a0
[ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
[ 310.349243] ? kasan_kmalloc+0xad/0xe0
[ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
[ 310.351261] ? memset+0x31/0x40
[ 310.352054] ? __check_object_size+0x22e/0x55c
[ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
[ 310.353686] ? check_stack_object+0x120/0x120
[ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
[ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
[ 310.356174] ? tcp_cwnd_restart+0x169/0x440
[ 310.357016] tcp_push+0x47c/0xbd0
[ 310.357777] ? copy_from_iter_full+0x21e/0xc70
[ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
[ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
[ 310.360355] ? tcp_send_mss+0x24/0x2b0
[ 310.361135] tcp_sendmsg+0xd6d/0x43f0
[ 310.361908] ? select_estimate_accuracy+0x440/0x440
[ 310.362765] ? tcp_sendpage+0x2170/0x2170
[ 310.363583] ? set_fd_set.part.1+0x50/0x50
[ 310.364392] ? remove_wait_queue+0x196/0x3b0
[ 310.365205] ? set_fd_set.part.1+0x50/0x50
[ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
[ 310.366865] ? __wake_up+0x44/0x50
[ 310.367637] ? n_tty_read+0x9f9/0x19d0
[ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
[ 310.369283] ? __check_object_size+0x22e/0x55c
[ 310.370129] inet_sendmsg+0x111/0x590
[ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
[ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
[ 310.373449] sock_sendmsg+0xba/0xf0
[ 310.374217] sock_write_iter+0x2e4/0x6a0
[ 310.375005] ? core_sys_select+0x47d/0x780
[ 310.375822] ? sock_sendmsg+0xf0/0xf0
[ 310.376607] __vfs_write+0x4e0/0x960
[ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
[ 310.378864] ? __vfs_read+0x950/0x950
[ 310.380178] ? rw_verify_area+0xbd/0x2b0
[ 310.381092] vfs_write+0x155/0x4b0
[ 310.381877] SyS_write+0xf7/0x240
[ 310.382616] ? SyS_read+0x240/0x240
[ 310.383404] ? SyS_read+0x240/0x240
[ 310.384159] do_syscall_64+0x235/0x5b0
[ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
[ 310.385747] ? syscall_return_slowpath+0x240/0x240
[ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
[ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
[ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
[ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
[ 310.390164] RIP: 0033:0x7f301f83c070
[ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
00007f301f83c070
[ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
0000000000000003
[ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
0000000000003000
[ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
0000000000000000
[ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
0000000000000003
[ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02 00
00 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1
e9 03 <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
[ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
RSP: ffff880069e46540
[ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
[ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
[ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt
(gdb) l *(free_old_xmit_skbs+0x2b7)
0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
1046
1047 static void free_old_xmit_skbs(struct send_queue *sq)
1048 {
1049 struct sk_buff *skb;
1050 unsigned int len;
1051 struct virtnet_info *vi = sq->vq->vdev->priv;
1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
1053 unsigned int packets = 0;
1054 unsigned int bytes = 0;
1055
Let me know if i need to provide more informations.
Best regards.
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-04 22:48 BUG: KASAN: use-after-free in free_old_xmit_skbs Jean-Philippe Menil
@ 2017-06-05 2:08 ` Michael S. Tsirkin
2017-06-05 23:52 ` [Qemu-devel] " Michael S. Tsirkin
2017-06-05 23:52 ` Michael S. Tsirkin
0 siblings, 2 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-05 2:08 UTC (permalink / raw)
To: Jean-Philippe Menil; +Cc: netdev, jasowang, John Fastabend
On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> Hi,
>
> while playing with xdp and ebpf, i'm hitting the following:
>
> [ 309.993136]
> ==================================================================
> [ 309.994735] BUG: KASAN: use-after-free in
> free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> [ 310.000650]
> [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-20170228_101828-anatol 04/01/2014
> [ 310.006495] Call Trace:
> [ 310.007610] dump_stack+0xb8/0x14c
> [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> [ 310.011203] print_address_description+0x6f/0x280
> [ 310.012416] kasan_report+0x27a/0x370
> [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> [ 310.024874] ? update_load_avg+0x1281/0x29f0
> [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> [ 310.029731] ? netif_skb_features+0x920/0x920
> [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> [ 310.048301] ? update_stack_state+0x402/0x780
> [ 310.049307] ? account_entity_enqueue+0x730/0x730
> [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> [ 310.051286] ? update_curr_fair+0x70/0x70
> [ 310.052206] ? enqueue_entity+0x2450/0x2450
> [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.054082] ? dequeue_entity+0x27a/0x1520
> [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> [ 310.055822] ? yield_to_task_fair+0x110/0x110
> [ 310.056708] ? set_next_entity+0x2f2/0xa90
> [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> [ 310.058457] dev_queue_xmit+0x10/0x20
> [ 310.059298] ip_finish_output2+0xacf/0x12a0
> [ 310.060160] ? dequeue_entity+0x1520/0x1520
> [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> [ 310.066677] ? __switch_to+0x685/0xda0
> [ 310.068166] ? load_balance+0x38f0/0x38f0
> [ 310.069544] ? compat_start_thread+0x80/0x80
> [ 310.070989] ? trace_find_cmdline+0x60/0x60
> [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> [ 310.073579] ip_finish_output+0x407/0x880
> [ 310.074441] ? ip_finish_output+0x407/0x880
> [ 310.075255] ? update_stack_state+0x402/0x780
> [ 310.076076] ip_output+0x1c0/0x640
> [ 310.076843] ? ip_mc_output+0x1350/0x1350
> [ 310.077642] ? __sk_dst_check+0x164/0x370
> [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> [ 310.081097] ? memcpy+0x45/0x50
> [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> [ 310.082676] ip_local_out+0x70/0x90
> [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> [ 310.087524] ? sock_sendmsg+0xba/0xf0
> [ 310.088298] ? __vfs_write+0x4e0/0x960
> [ 310.089074] ? vfs_write+0x155/0x4b0
> [ 310.089838] ? SyS_write+0xf7/0x240
> [ 310.090593] ? do_syscall_64+0x235/0x5b0
> [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.094690] ? sock_sendmsg+0xba/0xf0
> [ 310.096133] ? do_syscall_64+0x235/0x5b0
> [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> [ 310.100539] ? radix_tree_lookup+0xd/0x10
> [ 310.101894] ? get_work_pool+0xcd/0x150
> [ 310.103216] ? check_flush_dependency+0x330/0x330
> [ 310.104113] tcp_write_xmit+0x498/0x52a0
> [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> [ 310.107331] ? memset+0x31/0x40
> [ 310.108070] ? __check_object_size+0x22e/0x55c
> [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> [ 310.109690] ? check_stack_object+0x120/0x120
> [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> [ 310.112159] tcp_push+0x47c/0xbd0
> [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> [ 310.121224] ? __wake_up+0x44/0x50
> [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> [ 310.122898] ? __check_object_size+0x22e/0x55c
> [ 310.125380] inet_sendmsg+0x111/0x590
> [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> [ 310.129817] sock_sendmsg+0xba/0xf0
> [ 310.131110] sock_write_iter+0x2e4/0x6a0
> [ 310.132433] ? core_sys_select+0x47d/0x780
> [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> [ 310.134591] __vfs_write+0x4e0/0x960
> [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> [ 310.136160] ? __vfs_read+0x950/0x950
> [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> [ 310.137711] vfs_write+0x155/0x4b0
> [ 310.138454] SyS_write+0xf7/0x240
> [ 310.139183] ? SyS_read+0x240/0x240
> [ 310.139922] ? SyS_read+0x240/0x240
> [ 310.140649] do_syscall_64+0x235/0x5b0
> [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.146257] RIP: 0033:0x7f6f868fb070
> [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000001
> [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> 00007f6f868fb070
> [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> 0000000000000003
> [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> 0000000000003000
> [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> 0000000000000000
> [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> 0000000000000003
> [ 310.153578]
> [ 310.156362] Allocated by task 483:
> [ 310.157812] save_stack_trace+0x1b/0x20
> [ 310.159274] save_stack+0x43/0xd0
> [ 310.160663] kasan_kmalloc+0xad/0xe0
> [ 310.161943] __kmalloc+0x105/0x230
> [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> [ 310.165536] setup_vq+0x136/0x620
> [ 310.166286] vp_setup_vq+0x13d/0x6d0
> [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> [ 310.167855] vp_find_vqs+0x71/0x410
> [ 310.168641] vp_modern_find_vqs+0x21/0x140
> [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> [ 310.172918] do_setlink+0x2c33/0x3bc0
> [ 310.173703] rtnl_setlink+0x245/0x380
> [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> [ 310.175344] netlink_rcv_skb+0x213/0x450
> [ 310.176166] rtnetlink_rcv+0x28/0x30
> [ 310.176990] netlink_unicast+0x4a0/0x6c0
> [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> [ 310.178646] sock_sendmsg+0xba/0xf0
> [ 310.179435] SYSC_sendto+0x31d/0x620
> [ 310.180229] SyS_sendto+0xe/0x10
> [ 310.181004] do_syscall_64+0x235/0x5b0
> [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> [ 310.182595]
> [ 310.183217] Freed by task 483:
> [ 310.183934] save_stack_trace+0x1b/0x20
> [ 310.184801] save_stack+0x43/0xd0
> [ 310.187187] kasan_slab_free+0x72/0xc0
> [ 310.188530] kfree+0x94/0x1a0
> [ 310.189797] vring_del_virtqueue+0x19a/0x430
> [ 310.191221] del_vq+0x11c/0x250
> [ 310.192474] vp_del_vqs+0x379/0xc30
> [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> [ 310.197072] do_setlink+0x2c33/0x3bc0
> [ 310.197804] rtnl_setlink+0x245/0x380
> [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> [ 310.199283] netlink_rcv_skb+0x213/0x450
> [ 310.200036] rtnetlink_rcv+0x28/0x30
> [ 310.200754] netlink_unicast+0x4a0/0x6c0
> [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> [ 310.202236] sock_sendmsg+0xba/0xf0
> [ 310.202947] SYSC_sendto+0x31d/0x620
> [ 310.203660] SyS_sendto+0xe/0x10
> [ 310.204340] do_syscall_64+0x235/0x5b0
> [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> [ 310.205792]
> [ 310.206350] The buggy address belongs to the object at ffff88006aa64200
> [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> [ 310.208149] The buggy address is located 32 bytes inside of
> [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> [ 310.209929] The buggy address belongs to the page:
> [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping: (null)
> index:0x0 compound_mapcount: 0
> [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> 0000000100030003
> [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> 0000000000000000
> [ 310.215635] page dumped because: kasan: bad access detected
> [ 310.218989]
> [ 310.220398] Memory state around the buggy address:
> [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> [ 310.227400] ^
> [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> [ 310.230639]
> ==================================================================
> [ 310.231788] Disabling lock debugging due to kernel taint
> [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user memory
> access
> [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse irqbypass
> i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw virtio_balloon
> pata_acpi virtio_net virtio_blk
> [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+ #2
> [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-20170228_101828-anatol 04/01/2014
> [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
> [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> 0000000000000004
> [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> 0000000000000020
> [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> ffff880066b3c400
> [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> ffff880066b3c400
> [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> ffff88006afc9001
> [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> knlGS:0000000000000000
> [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> 00000000000006f0
> [ 310.266178] Call Trace:
> [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> [ 310.273076] ? update_load_avg+0x1281/0x29f0
> [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> [ 310.277548] ? rb_insert_color+0x1590/0x1590
> [ 310.280172] ? netif_skb_features+0x920/0x920
> [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> [ 310.291837] ? update_curr+0x1ef/0x750
> [ 310.292826] ? update_stack_state+0x402/0x780
> [ 310.293827] ? account_entity_enqueue+0x730/0x730
> [ 310.294831] ? update_stack_state+0x402/0x780
> [ 310.295818] ? update_curr_fair+0x70/0x70
> [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.297693] ? dequeue_entity+0x27a/0x1520
> [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> [ 310.299484] ? yield_to_task_fair+0x110/0x110
> [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> [ 310.301246] ? __free_insn_slot+0x600/0x600
> [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> [ 310.303883] dev_queue_xmit+0x10/0x20
> [ 310.304711] ip_finish_output2+0xacf/0x12a0
> [ 310.305558] ? dequeue_entity+0x1520/0x1520
> [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> [ 310.307320] ? save_stack_trace+0x1b/0x20
> [ 310.308133] ? save_stack+0x43/0xd0
> [ 310.309081] ? kasan_slab_free+0x72/0xc0
> [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> [ 310.311406] ? tcp_ack+0x2730/0x7450
> [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> [ 310.313769] ? __release_sock+0x14a/0x2b0
> [ 310.314550] ? release_sock+0xa8/0x270
> [ 310.315330] ? inet_sendmsg+0x111/0x590
> [ 310.316100] ? sock_sendmsg+0xba/0xf0
> [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> [ 310.320800] ? update_stack_state+0x402/0x780
> [ 310.321590] ip_finish_output+0x407/0x880
> [ 310.322347] ? ip_finish_output+0x407/0x880
> [ 310.323138] ? update_stack_state+0x402/0x780
> [ 310.323948] ip_output+0x1c0/0x640
> [ 310.324661] ? ip_mc_output+0x1350/0x1350
> [ 310.325415] ? __sk_dst_check+0x164/0x370
> [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> [ 310.328684] ? memcpy+0x45/0x50
> [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> [ 310.330180] ip_local_out+0x70/0x90
> [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> [ 310.334887] ? sock_sendmsg+0xba/0xf0
> [ 310.335637] ? __vfs_write+0x4e0/0x960
> [ 310.336391] ? vfs_write+0x155/0x4b0
> [ 310.337135] ? SyS_write+0xf7/0x240
> [ 310.337861] ? do_syscall_64+0x235/0x5b0
> [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.339443] ? sock_sendmsg+0xba/0xf0
> [ 310.341675] ? do_syscall_64+0x235/0x5b0
> [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> [ 310.344095] ? radix_tree_lookup+0xd/0x10
> [ 310.344871] ? get_work_pool+0xcd/0x150
> [ 310.345635] ? check_flush_dependency+0x330/0x330
> [ 310.346466] tcp_write_xmit+0x498/0x52a0
> [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> [ 310.351261] ? memset+0x31/0x40
> [ 310.352054] ? __check_object_size+0x22e/0x55c
> [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> [ 310.353686] ? check_stack_object+0x120/0x120
> [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> [ 310.357016] tcp_push+0x47c/0xbd0
> [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> [ 310.366865] ? __wake_up+0x44/0x50
> [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> [ 310.369283] ? __check_object_size+0x22e/0x55c
> [ 310.370129] inet_sendmsg+0x111/0x590
> [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> [ 310.373449] sock_sendmsg+0xba/0xf0
> [ 310.374217] sock_write_iter+0x2e4/0x6a0
> [ 310.375005] ? core_sys_select+0x47d/0x780
> [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> [ 310.376607] __vfs_write+0x4e0/0x960
> [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> [ 310.378864] ? __vfs_read+0x950/0x950
> [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> [ 310.381092] vfs_write+0x155/0x4b0
> [ 310.381877] SyS_write+0xf7/0x240
> [ 310.382616] ? SyS_read+0x240/0x240
> [ 310.383404] ? SyS_read+0x240/0x240
> [ 310.384159] do_syscall_64+0x235/0x5b0
> [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> [ 310.390164] RIP: 0033:0x7f301f83c070
> [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000001
> [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> 00007f301f83c070
> [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> 0000000000000003
> [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> 0000000000003000
> [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> 0000000000000000
> [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> 0000000000000003
> [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02 00 00
> 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1 e9 03
> <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net] RSP:
> ffff880069e46540
> [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation
> range: 0xffffffff80000000-0xffffffffbfffffff)
> [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> interrupt
>
>
> (gdb) l *(free_old_xmit_skbs+0x2b7)
> 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> 1046
> 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> 1048 {
> 1049 struct sk_buff *skb;
> 1050 unsigned int len;
> 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> 1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
> 1053 unsigned int packets = 0;
> 1054 unsigned int bytes = 0;
> 1055
>
> Let me know if i need to provide more informations.
>
> Best regards.
>
> Jean-Philippe
So del_vq done during xdp setup seems to race with regular xmit.
Since commit 680557cf79f82623e2c4fd42733077d60a843513
virtio_net: rework mergeable buffer handling
we no longer must do the resets, we now have enough space
to store a bit saying whether a buffer is xdp one or not.
And that's probably a cleaner way to fix these issues than
try to find and fix the race condition.
John?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-05 2:08 ` Michael S. Tsirkin
@ 2017-06-05 23:52 ` Michael S. Tsirkin
2017-06-05 23:52 ` Michael S. Tsirkin
1 sibling, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-05 23:52 UTC (permalink / raw)
To: Jean-Philippe Menil
Cc: netdev, jasowang, John Fastabend, virtualization, qemu-devel
On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > Hi,
> >
> > while playing with xdp and ebpf, i'm hitting the following:
> >
> > [ 309.993136]
> > ==================================================================
> > [ 309.994735] BUG: KASAN: use-after-free in
> > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > [ 310.000650]
> > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.006495] Call Trace:
> > [ 310.007610] dump_stack+0xb8/0x14c
> > [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> > [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> > [ 310.011203] print_address_description+0x6f/0x280
> > [ 310.012416] kasan_report+0x27a/0x370
> > [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> > [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> > [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> > [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> > [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.024874] ? update_load_avg+0x1281/0x29f0
> > [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.029731] ? netif_skb_features+0x920/0x920
> > [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.048301] ? update_stack_state+0x402/0x780
> > [ 310.049307] ? account_entity_enqueue+0x730/0x730
> > [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.051286] ? update_curr_fair+0x70/0x70
> > [ 310.052206] ? enqueue_entity+0x2450/0x2450
> > [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.054082] ? dequeue_entity+0x27a/0x1520
> > [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> > [ 310.055822] ? yield_to_task_fair+0x110/0x110
> > [ 310.056708] ? set_next_entity+0x2f2/0xa90
> > [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.058457] dev_queue_xmit+0x10/0x20
> > [ 310.059298] ip_finish_output2+0xacf/0x12a0
> > [ 310.060160] ? dequeue_entity+0x1520/0x1520
> > [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> > [ 310.066677] ? __switch_to+0x685/0xda0
> > [ 310.068166] ? load_balance+0x38f0/0x38f0
> > [ 310.069544] ? compat_start_thread+0x80/0x80
> > [ 310.070989] ? trace_find_cmdline+0x60/0x60
> > [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.073579] ip_finish_output+0x407/0x880
> > [ 310.074441] ? ip_finish_output+0x407/0x880
> > [ 310.075255] ? update_stack_state+0x402/0x780
> > [ 310.076076] ip_output+0x1c0/0x640
> > [ 310.076843] ? ip_mc_output+0x1350/0x1350
> > [ 310.077642] ? __sk_dst_check+0x164/0x370
> > [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.081097] ? memcpy+0x45/0x50
> > [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> > [ 310.082676] ip_local_out+0x70/0x90
> > [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.087524] ? sock_sendmsg+0xba/0xf0
> > [ 310.088298] ? __vfs_write+0x4e0/0x960
> > [ 310.089074] ? vfs_write+0x155/0x4b0
> > [ 310.089838] ? SyS_write+0xf7/0x240
> > [ 310.090593] ? do_syscall_64+0x235/0x5b0
> > [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.094690] ? sock_sendmsg+0xba/0xf0
> > [ 310.096133] ? do_syscall_64+0x235/0x5b0
> > [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.100539] ? radix_tree_lookup+0xd/0x10
> > [ 310.101894] ? get_work_pool+0xcd/0x150
> > [ 310.103216] ? check_flush_dependency+0x330/0x330
> > [ 310.104113] tcp_write_xmit+0x498/0x52a0
> > [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> > [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.107331] ? memset+0x31/0x40
> > [ 310.108070] ? __check_object_size+0x22e/0x55c
> > [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.109690] ? check_stack_object+0x120/0x120
> > [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.112159] tcp_push+0x47c/0xbd0
> > [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> > [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> > [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> > [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> > [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> > [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.121224] ? __wake_up+0x44/0x50
> > [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> > [ 310.122898] ? __check_object_size+0x22e/0x55c
> > [ 310.125380] inet_sendmsg+0x111/0x590
> > [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.129817] sock_sendmsg+0xba/0xf0
> > [ 310.131110] sock_write_iter+0x2e4/0x6a0
> > [ 310.132433] ? core_sys_select+0x47d/0x780
> > [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> > [ 310.134591] __vfs_write+0x4e0/0x960
> > [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.136160] ? __vfs_read+0x950/0x950
> > [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> > [ 310.137711] vfs_write+0x155/0x4b0
> > [ 310.138454] SyS_write+0xf7/0x240
> > [ 310.139183] ? SyS_read+0x240/0x240
> > [ 310.139922] ? SyS_read+0x240/0x240
> > [ 310.140649] do_syscall_64+0x235/0x5b0
> > [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> > [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.146257] RIP: 0033:0x7f6f868fb070
> > [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> > 00007f6f868fb070
> > [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> > 0000000000000003
> > [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> > 0000000000000003
> > [ 310.153578]
> > [ 310.156362] Allocated by task 483:
> > [ 310.157812] save_stack_trace+0x1b/0x20
> > [ 310.159274] save_stack+0x43/0xd0
> > [ 310.160663] kasan_kmalloc+0xad/0xe0
> > [ 310.161943] __kmalloc+0x105/0x230
> > [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> > [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> > [ 310.165536] setup_vq+0x136/0x620
> > [ 310.166286] vp_setup_vq+0x13d/0x6d0
> > [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> > [ 310.167855] vp_find_vqs+0x71/0x410
> > [ 310.168641] vp_modern_find_vqs+0x21/0x140
> > [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> > [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> > [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> > [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> > [ 310.172918] do_setlink+0x2c33/0x3bc0
> > [ 310.173703] rtnl_setlink+0x245/0x380
> > [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.175344] netlink_rcv_skb+0x213/0x450
> > [ 310.176166] rtnetlink_rcv+0x28/0x30
> > [ 310.176990] netlink_unicast+0x4a0/0x6c0
> > [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> > [ 310.178646] sock_sendmsg+0xba/0xf0
> > [ 310.179435] SYSC_sendto+0x31d/0x620
> > [ 310.180229] SyS_sendto+0xe/0x10
> > [ 310.181004] do_syscall_64+0x235/0x5b0
> > [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.182595]
> > [ 310.183217] Freed by task 483:
> > [ 310.183934] save_stack_trace+0x1b/0x20
> > [ 310.184801] save_stack+0x43/0xd0
> > [ 310.187187] kasan_slab_free+0x72/0xc0
> > [ 310.188530] kfree+0x94/0x1a0
> > [ 310.189797] vring_del_virtqueue+0x19a/0x430
> > [ 310.191221] del_vq+0x11c/0x250
> > [ 310.192474] vp_del_vqs+0x379/0xc30
> > [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> > [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> > [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> > [ 310.197072] do_setlink+0x2c33/0x3bc0
> > [ 310.197804] rtnl_setlink+0x245/0x380
> > [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.199283] netlink_rcv_skb+0x213/0x450
> > [ 310.200036] rtnetlink_rcv+0x28/0x30
> > [ 310.200754] netlink_unicast+0x4a0/0x6c0
> > [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> > [ 310.202236] sock_sendmsg+0xba/0xf0
> > [ 310.202947] SYSC_sendto+0x31d/0x620
> > [ 310.203660] SyS_sendto+0xe/0x10
> > [ 310.204340] do_syscall_64+0x235/0x5b0
> > [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.205792]
> > [ 310.206350] The buggy address belongs to the object at ffff88006aa64200
> > [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> > [ 310.208149] The buggy address is located 32 bytes inside of
> > [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> > [ 310.209929] The buggy address belongs to the page:
> > [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping: (null)
> > index:0x0 compound_mapcount: 0
> > [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> > [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> > 0000000100030003
> > [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> > 0000000000000000
> > [ 310.215635] page dumped because: kasan: bad access detected
> > [ 310.218989]
> > [ 310.220398] Memory state around the buggy address:
> > [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.227400] ^
> > [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.230639]
> > ==================================================================
> > [ 310.231788] Disabling lock debugging due to kernel taint
> > [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> > [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user memory
> > access
> > [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> > [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse irqbypass
> > i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw virtio_balloon
> > pata_acpi virtio_net virtio_blk
> > [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+ #2
> > [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> > [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
> > [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> > [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > 0000000000000004
> > [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> > 0000000000000020
> > [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> > ffff880066b3c400
> > [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> > ffff880066b3c400
> > [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> > ffff88006afc9001
> > [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> > knlGS:0000000000000000
> > [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> > 00000000000006f0
> > [ 310.266178] Call Trace:
> > [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> > [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> > [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.273076] ? update_load_avg+0x1281/0x29f0
> > [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.277548] ? rb_insert_color+0x1590/0x1590
> > [ 310.280172] ? netif_skb_features+0x920/0x920
> > [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.291837] ? update_curr+0x1ef/0x750
> > [ 310.292826] ? update_stack_state+0x402/0x780
> > [ 310.293827] ? account_entity_enqueue+0x730/0x730
> > [ 310.294831] ? update_stack_state+0x402/0x780
> > [ 310.295818] ? update_curr_fair+0x70/0x70
> > [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.297693] ? dequeue_entity+0x27a/0x1520
> > [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> > [ 310.299484] ? yield_to_task_fair+0x110/0x110
> > [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> > [ 310.301246] ? __free_insn_slot+0x600/0x600
> > [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> > [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.303883] dev_queue_xmit+0x10/0x20
> > [ 310.304711] ip_finish_output2+0xacf/0x12a0
> > [ 310.305558] ? dequeue_entity+0x1520/0x1520
> > [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.307320] ? save_stack_trace+0x1b/0x20
> > [ 310.308133] ? save_stack+0x43/0xd0
> > [ 310.309081] ? kasan_slab_free+0x72/0xc0
> > [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> > [ 310.311406] ? tcp_ack+0x2730/0x7450
> > [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> > [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> > [ 310.313769] ? __release_sock+0x14a/0x2b0
> > [ 310.314550] ? release_sock+0xa8/0x270
> > [ 310.315330] ? inet_sendmsg+0x111/0x590
> > [ 310.316100] ? sock_sendmsg+0xba/0xf0
> > [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> > [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.320800] ? update_stack_state+0x402/0x780
> > [ 310.321590] ip_finish_output+0x407/0x880
> > [ 310.322347] ? ip_finish_output+0x407/0x880
> > [ 310.323138] ? update_stack_state+0x402/0x780
> > [ 310.323948] ip_output+0x1c0/0x640
> > [ 310.324661] ? ip_mc_output+0x1350/0x1350
> > [ 310.325415] ? __sk_dst_check+0x164/0x370
> > [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.328684] ? memcpy+0x45/0x50
> > [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> > [ 310.330180] ip_local_out+0x70/0x90
> > [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.334887] ? sock_sendmsg+0xba/0xf0
> > [ 310.335637] ? __vfs_write+0x4e0/0x960
> > [ 310.336391] ? vfs_write+0x155/0x4b0
> > [ 310.337135] ? SyS_write+0xf7/0x240
> > [ 310.337861] ? do_syscall_64+0x235/0x5b0
> > [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.339443] ? sock_sendmsg+0xba/0xf0
> > [ 310.341675] ? do_syscall_64+0x235/0x5b0
> > [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.344095] ? radix_tree_lookup+0xd/0x10
> > [ 310.344871] ? get_work_pool+0xcd/0x150
> > [ 310.345635] ? check_flush_dependency+0x330/0x330
> > [ 310.346466] tcp_write_xmit+0x498/0x52a0
> > [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> > [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.351261] ? memset+0x31/0x40
> > [ 310.352054] ? __check_object_size+0x22e/0x55c
> > [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.353686] ? check_stack_object+0x120/0x120
> > [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> > [ 310.357016] tcp_push+0x47c/0xbd0
> > [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> > [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> > [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> > [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> > [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> > [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> > [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.366865] ? __wake_up+0x44/0x50
> > [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> > [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> > [ 310.369283] ? __check_object_size+0x22e/0x55c
> > [ 310.370129] inet_sendmsg+0x111/0x590
> > [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.373449] sock_sendmsg+0xba/0xf0
> > [ 310.374217] sock_write_iter+0x2e4/0x6a0
> > [ 310.375005] ? core_sys_select+0x47d/0x780
> > [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> > [ 310.376607] __vfs_write+0x4e0/0x960
> > [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.378864] ? __vfs_read+0x950/0x950
> > [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> > [ 310.381092] vfs_write+0x155/0x4b0
> > [ 310.381877] SyS_write+0xf7/0x240
> > [ 310.382616] ? SyS_read+0x240/0x240
> > [ 310.383404] ? SyS_read+0x240/0x240
> > [ 310.384159] do_syscall_64+0x235/0x5b0
> > [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> > [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.390164] RIP: 0033:0x7f301f83c070
> > [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> > 00007f301f83c070
> > [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> > 0000000000000003
> > [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> > 0000000000000003
> > [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02 00 00
> > 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1 e9 03
> > <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> > [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net] RSP:
> > ffff880069e46540
> > [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> > [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation
> > range: 0xffffffff80000000-0xffffffffbfffffff)
> > [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> > interrupt
> >
> >
> > (gdb) l *(free_old_xmit_skbs+0x2b7)
> > 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> > 1046
> > 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> > 1048 {
> > 1049 struct sk_buff *skb;
> > 1050 unsigned int len;
> > 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> > 1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
> > 1053 unsigned int packets = 0;
> > 1054 unsigned int bytes = 0;
> > 1055
> >
> > Let me know if i need to provide more informations.
> >
> > Best regards.
> >
> > Jean-Philippe
>
> So del_vq done during xdp setup seems to race with regular xmit.
>
> Since commit 680557cf79f82623e2c4fd42733077d60a843513
> virtio_net: rework mergeable buffer handling
>
> we no longer must do the resets, we now have enough space
> to store a bit saying whether a buffer is xdp one or not.
>
> And that's probably a cleaner way to fix these issues than
> try to find and fix the race condition.
>
> John?
>
> --
> MST
I think I see the source of the race. virtio net calls
netif_device_detach and assumes no packets will be sent after
this point. However, all it does is stop all queues so
no new packets will be transmitted.
Try locking with HARD_TX_LOCK?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-05 23:52 ` Michael S. Tsirkin
0 siblings, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-05 23:52 UTC (permalink / raw)
To: Jean-Philippe Menil
Cc: netdev, jasowang, John Fastabend, virtualization, qemu-devel
On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > Hi,
> >
> > while playing with xdp and ebpf, i'm hitting the following:
> >
> > [ 309.993136]
> > ==================================================================
> > [ 309.994735] BUG: KASAN: use-after-free in
> > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > [ 310.000650]
> > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.006495] Call Trace:
> > [ 310.007610] dump_stack+0xb8/0x14c
> > [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> > [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> > [ 310.011203] print_address_description+0x6f/0x280
> > [ 310.012416] kasan_report+0x27a/0x370
> > [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> > [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> > [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> > [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> > [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.024874] ? update_load_avg+0x1281/0x29f0
> > [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.029731] ? netif_skb_features+0x920/0x920
> > [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.048301] ? update_stack_state+0x402/0x780
> > [ 310.049307] ? account_entity_enqueue+0x730/0x730
> > [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.051286] ? update_curr_fair+0x70/0x70
> > [ 310.052206] ? enqueue_entity+0x2450/0x2450
> > [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.054082] ? dequeue_entity+0x27a/0x1520
> > [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> > [ 310.055822] ? yield_to_task_fair+0x110/0x110
> > [ 310.056708] ? set_next_entity+0x2f2/0xa90
> > [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.058457] dev_queue_xmit+0x10/0x20
> > [ 310.059298] ip_finish_output2+0xacf/0x12a0
> > [ 310.060160] ? dequeue_entity+0x1520/0x1520
> > [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> > [ 310.066677] ? __switch_to+0x685/0xda0
> > [ 310.068166] ? load_balance+0x38f0/0x38f0
> > [ 310.069544] ? compat_start_thread+0x80/0x80
> > [ 310.070989] ? trace_find_cmdline+0x60/0x60
> > [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.073579] ip_finish_output+0x407/0x880
> > [ 310.074441] ? ip_finish_output+0x407/0x880
> > [ 310.075255] ? update_stack_state+0x402/0x780
> > [ 310.076076] ip_output+0x1c0/0x640
> > [ 310.076843] ? ip_mc_output+0x1350/0x1350
> > [ 310.077642] ? __sk_dst_check+0x164/0x370
> > [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.081097] ? memcpy+0x45/0x50
> > [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> > [ 310.082676] ip_local_out+0x70/0x90
> > [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.087524] ? sock_sendmsg+0xba/0xf0
> > [ 310.088298] ? __vfs_write+0x4e0/0x960
> > [ 310.089074] ? vfs_write+0x155/0x4b0
> > [ 310.089838] ? SyS_write+0xf7/0x240
> > [ 310.090593] ? do_syscall_64+0x235/0x5b0
> > [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.094690] ? sock_sendmsg+0xba/0xf0
> > [ 310.096133] ? do_syscall_64+0x235/0x5b0
> > [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.100539] ? radix_tree_lookup+0xd/0x10
> > [ 310.101894] ? get_work_pool+0xcd/0x150
> > [ 310.103216] ? check_flush_dependency+0x330/0x330
> > [ 310.104113] tcp_write_xmit+0x498/0x52a0
> > [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> > [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.107331] ? memset+0x31/0x40
> > [ 310.108070] ? __check_object_size+0x22e/0x55c
> > [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.109690] ? check_stack_object+0x120/0x120
> > [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.112159] tcp_push+0x47c/0xbd0
> > [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> > [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> > [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> > [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> > [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> > [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.121224] ? __wake_up+0x44/0x50
> > [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> > [ 310.122898] ? __check_object_size+0x22e/0x55c
> > [ 310.125380] inet_sendmsg+0x111/0x590
> > [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.129817] sock_sendmsg+0xba/0xf0
> > [ 310.131110] sock_write_iter+0x2e4/0x6a0
> > [ 310.132433] ? core_sys_select+0x47d/0x780
> > [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> > [ 310.134591] __vfs_write+0x4e0/0x960
> > [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.136160] ? __vfs_read+0x950/0x950
> > [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> > [ 310.137711] vfs_write+0x155/0x4b0
> > [ 310.138454] SyS_write+0xf7/0x240
> > [ 310.139183] ? SyS_read+0x240/0x240
> > [ 310.139922] ? SyS_read+0x240/0x240
> > [ 310.140649] do_syscall_64+0x235/0x5b0
> > [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> > [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.146257] RIP: 0033:0x7f6f868fb070
> > [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> > 00007f6f868fb070
> > [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> > 0000000000000003
> > [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> > 0000000000000003
> > [ 310.153578]
> > [ 310.156362] Allocated by task 483:
> > [ 310.157812] save_stack_trace+0x1b/0x20
> > [ 310.159274] save_stack+0x43/0xd0
> > [ 310.160663] kasan_kmalloc+0xad/0xe0
> > [ 310.161943] __kmalloc+0x105/0x230
> > [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> > [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> > [ 310.165536] setup_vq+0x136/0x620
> > [ 310.166286] vp_setup_vq+0x13d/0x6d0
> > [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> > [ 310.167855] vp_find_vqs+0x71/0x410
> > [ 310.168641] vp_modern_find_vqs+0x21/0x140
> > [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> > [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> > [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> > [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> > [ 310.172918] do_setlink+0x2c33/0x3bc0
> > [ 310.173703] rtnl_setlink+0x245/0x380
> > [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.175344] netlink_rcv_skb+0x213/0x450
> > [ 310.176166] rtnetlink_rcv+0x28/0x30
> > [ 310.176990] netlink_unicast+0x4a0/0x6c0
> > [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> > [ 310.178646] sock_sendmsg+0xba/0xf0
> > [ 310.179435] SYSC_sendto+0x31d/0x620
> > [ 310.180229] SyS_sendto+0xe/0x10
> > [ 310.181004] do_syscall_64+0x235/0x5b0
> > [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.182595]
> > [ 310.183217] Freed by task 483:
> > [ 310.183934] save_stack_trace+0x1b/0x20
> > [ 310.184801] save_stack+0x43/0xd0
> > [ 310.187187] kasan_slab_free+0x72/0xc0
> > [ 310.188530] kfree+0x94/0x1a0
> > [ 310.189797] vring_del_virtqueue+0x19a/0x430
> > [ 310.191221] del_vq+0x11c/0x250
> > [ 310.192474] vp_del_vqs+0x379/0xc30
> > [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> > [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> > [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> > [ 310.197072] do_setlink+0x2c33/0x3bc0
> > [ 310.197804] rtnl_setlink+0x245/0x380
> > [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.199283] netlink_rcv_skb+0x213/0x450
> > [ 310.200036] rtnetlink_rcv+0x28/0x30
> > [ 310.200754] netlink_unicast+0x4a0/0x6c0
> > [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> > [ 310.202236] sock_sendmsg+0xba/0xf0
> > [ 310.202947] SYSC_sendto+0x31d/0x620
> > [ 310.203660] SyS_sendto+0xe/0x10
> > [ 310.204340] do_syscall_64+0x235/0x5b0
> > [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.205792]
> > [ 310.206350] The buggy address belongs to the object at ffff88006aa64200
> > [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> > [ 310.208149] The buggy address is located 32 bytes inside of
> > [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> > [ 310.209929] The buggy address belongs to the page:
> > [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping: (null)
> > index:0x0 compound_mapcount: 0
> > [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> > [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> > 0000000100030003
> > [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> > 0000000000000000
> > [ 310.215635] page dumped because: kasan: bad access detected
> > [ 310.218989]
> > [ 310.220398] Memory state around the buggy address:
> > [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.227400] ^
> > [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.230639]
> > ==================================================================
> > [ 310.231788] Disabling lock debugging due to kernel taint
> > [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> > [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user memory
> > access
> > [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> > [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse irqbypass
> > i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw virtio_balloon
> > pata_acpi virtio_net virtio_blk
> > [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+ #2
> > [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> > [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
> > [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> > [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > 0000000000000004
> > [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> > 0000000000000020
> > [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> > ffff880066b3c400
> > [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> > ffff880066b3c400
> > [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> > ffff88006afc9001
> > [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> > knlGS:0000000000000000
> > [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> > 00000000000006f0
> > [ 310.266178] Call Trace:
> > [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> > [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> > [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.273076] ? update_load_avg+0x1281/0x29f0
> > [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.277548] ? rb_insert_color+0x1590/0x1590
> > [ 310.280172] ? netif_skb_features+0x920/0x920
> > [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.291837] ? update_curr+0x1ef/0x750
> > [ 310.292826] ? update_stack_state+0x402/0x780
> > [ 310.293827] ? account_entity_enqueue+0x730/0x730
> > [ 310.294831] ? update_stack_state+0x402/0x780
> > [ 310.295818] ? update_curr_fair+0x70/0x70
> > [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.297693] ? dequeue_entity+0x27a/0x1520
> > [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> > [ 310.299484] ? yield_to_task_fair+0x110/0x110
> > [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> > [ 310.301246] ? __free_insn_slot+0x600/0x600
> > [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> > [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.303883] dev_queue_xmit+0x10/0x20
> > [ 310.304711] ip_finish_output2+0xacf/0x12a0
> > [ 310.305558] ? dequeue_entity+0x1520/0x1520
> > [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.307320] ? save_stack_trace+0x1b/0x20
> > [ 310.308133] ? save_stack+0x43/0xd0
> > [ 310.309081] ? kasan_slab_free+0x72/0xc0
> > [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> > [ 310.311406] ? tcp_ack+0x2730/0x7450
> > [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> > [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> > [ 310.313769] ? __release_sock+0x14a/0x2b0
> > [ 310.314550] ? release_sock+0xa8/0x270
> > [ 310.315330] ? inet_sendmsg+0x111/0x590
> > [ 310.316100] ? sock_sendmsg+0xba/0xf0
> > [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> > [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.320800] ? update_stack_state+0x402/0x780
> > [ 310.321590] ip_finish_output+0x407/0x880
> > [ 310.322347] ? ip_finish_output+0x407/0x880
> > [ 310.323138] ? update_stack_state+0x402/0x780
> > [ 310.323948] ip_output+0x1c0/0x640
> > [ 310.324661] ? ip_mc_output+0x1350/0x1350
> > [ 310.325415] ? __sk_dst_check+0x164/0x370
> > [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.328684] ? memcpy+0x45/0x50
> > [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> > [ 310.330180] ip_local_out+0x70/0x90
> > [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.334887] ? sock_sendmsg+0xba/0xf0
> > [ 310.335637] ? __vfs_write+0x4e0/0x960
> > [ 310.336391] ? vfs_write+0x155/0x4b0
> > [ 310.337135] ? SyS_write+0xf7/0x240
> > [ 310.337861] ? do_syscall_64+0x235/0x5b0
> > [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.339443] ? sock_sendmsg+0xba/0xf0
> > [ 310.341675] ? do_syscall_64+0x235/0x5b0
> > [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.344095] ? radix_tree_lookup+0xd/0x10
> > [ 310.344871] ? get_work_pool+0xcd/0x150
> > [ 310.345635] ? check_flush_dependency+0x330/0x330
> > [ 310.346466] tcp_write_xmit+0x498/0x52a0
> > [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> > [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.351261] ? memset+0x31/0x40
> > [ 310.352054] ? __check_object_size+0x22e/0x55c
> > [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.353686] ? check_stack_object+0x120/0x120
> > [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> > [ 310.357016] tcp_push+0x47c/0xbd0
> > [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> > [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> > [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> > [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> > [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> > [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> > [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.366865] ? __wake_up+0x44/0x50
> > [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> > [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> > [ 310.369283] ? __check_object_size+0x22e/0x55c
> > [ 310.370129] inet_sendmsg+0x111/0x590
> > [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.373449] sock_sendmsg+0xba/0xf0
> > [ 310.374217] sock_write_iter+0x2e4/0x6a0
> > [ 310.375005] ? core_sys_select+0x47d/0x780
> > [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> > [ 310.376607] __vfs_write+0x4e0/0x960
> > [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.378864] ? __vfs_read+0x950/0x950
> > [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> > [ 310.381092] vfs_write+0x155/0x4b0
> > [ 310.381877] SyS_write+0xf7/0x240
> > [ 310.382616] ? SyS_read+0x240/0x240
> > [ 310.383404] ? SyS_read+0x240/0x240
> > [ 310.384159] do_syscall_64+0x235/0x5b0
> > [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> > [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.390164] RIP: 0033:0x7f301f83c070
> > [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> > 00007f301f83c070
> > [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> > 0000000000000003
> > [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> > 0000000000000003
> > [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02 00 00
> > 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1 e9 03
> > <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> > [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net] RSP:
> > ffff880069e46540
> > [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> > [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation
> > range: 0xffffffff80000000-0xffffffffbfffffff)
> > [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> > interrupt
> >
> >
> > (gdb) l *(free_old_xmit_skbs+0x2b7)
> > 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> > 1046
> > 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> > 1048 {
> > 1049 struct sk_buff *skb;
> > 1050 unsigned int len;
> > 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> > 1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
> > 1053 unsigned int packets = 0;
> > 1054 unsigned int bytes = 0;
> > 1055
> >
> > Let me know if i need to provide more informations.
> >
> > Best regards.
> >
> > Jean-Philippe
>
> So del_vq done during xdp setup seems to race with regular xmit.
>
> Since commit 680557cf79f82623e2c4fd42733077d60a843513
> virtio_net: rework mergeable buffer handling
>
> we no longer must do the resets, we now have enough space
> to store a bit saying whether a buffer is xdp one or not.
>
> And that's probably a cleaner way to fix these issues than
> try to find and fix the race condition.
>
> John?
>
> --
> MST
I think I see the source of the race. virtio net calls
netif_device_detach and assumes no packets will be sent after
this point. However, all it does is stop all queues so
no new packets will be transmitted.
Try locking with HARD_TX_LOCK?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-05 2:08 ` Michael S. Tsirkin
2017-06-05 23:52 ` [Qemu-devel] " Michael S. Tsirkin
@ 2017-06-05 23:52 ` Michael S. Tsirkin
1 sibling, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-05 23:52 UTC (permalink / raw)
To: Jean-Philippe Menil; +Cc: netdev, John Fastabend, qemu-devel, virtualization
On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > Hi,
> >
> > while playing with xdp and ebpf, i'm hitting the following:
> >
> > [ 309.993136]
> > ==================================================================
> > [ 309.994735] BUG: KASAN: use-after-free in
> > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > [ 310.000650]
> > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.006495] Call Trace:
> > [ 310.007610] dump_stack+0xb8/0x14c
> > [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> > [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> > [ 310.011203] print_address_description+0x6f/0x280
> > [ 310.012416] kasan_report+0x27a/0x370
> > [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> > [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> > [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> > [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> > [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.024874] ? update_load_avg+0x1281/0x29f0
> > [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.029731] ? netif_skb_features+0x920/0x920
> > [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.048301] ? update_stack_state+0x402/0x780
> > [ 310.049307] ? account_entity_enqueue+0x730/0x730
> > [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.051286] ? update_curr_fair+0x70/0x70
> > [ 310.052206] ? enqueue_entity+0x2450/0x2450
> > [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.054082] ? dequeue_entity+0x27a/0x1520
> > [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> > [ 310.055822] ? yield_to_task_fair+0x110/0x110
> > [ 310.056708] ? set_next_entity+0x2f2/0xa90
> > [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.058457] dev_queue_xmit+0x10/0x20
> > [ 310.059298] ip_finish_output2+0xacf/0x12a0
> > [ 310.060160] ? dequeue_entity+0x1520/0x1520
> > [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> > [ 310.066677] ? __switch_to+0x685/0xda0
> > [ 310.068166] ? load_balance+0x38f0/0x38f0
> > [ 310.069544] ? compat_start_thread+0x80/0x80
> > [ 310.070989] ? trace_find_cmdline+0x60/0x60
> > [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.073579] ip_finish_output+0x407/0x880
> > [ 310.074441] ? ip_finish_output+0x407/0x880
> > [ 310.075255] ? update_stack_state+0x402/0x780
> > [ 310.076076] ip_output+0x1c0/0x640
> > [ 310.076843] ? ip_mc_output+0x1350/0x1350
> > [ 310.077642] ? __sk_dst_check+0x164/0x370
> > [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.081097] ? memcpy+0x45/0x50
> > [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> > [ 310.082676] ip_local_out+0x70/0x90
> > [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.087524] ? sock_sendmsg+0xba/0xf0
> > [ 310.088298] ? __vfs_write+0x4e0/0x960
> > [ 310.089074] ? vfs_write+0x155/0x4b0
> > [ 310.089838] ? SyS_write+0xf7/0x240
> > [ 310.090593] ? do_syscall_64+0x235/0x5b0
> > [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.094690] ? sock_sendmsg+0xba/0xf0
> > [ 310.096133] ? do_syscall_64+0x235/0x5b0
> > [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.100539] ? radix_tree_lookup+0xd/0x10
> > [ 310.101894] ? get_work_pool+0xcd/0x150
> > [ 310.103216] ? check_flush_dependency+0x330/0x330
> > [ 310.104113] tcp_write_xmit+0x498/0x52a0
> > [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> > [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.107331] ? memset+0x31/0x40
> > [ 310.108070] ? __check_object_size+0x22e/0x55c
> > [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.109690] ? check_stack_object+0x120/0x120
> > [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.112159] tcp_push+0x47c/0xbd0
> > [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> > [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> > [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> > [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> > [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> > [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.121224] ? __wake_up+0x44/0x50
> > [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> > [ 310.122898] ? __check_object_size+0x22e/0x55c
> > [ 310.125380] inet_sendmsg+0x111/0x590
> > [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.129817] sock_sendmsg+0xba/0xf0
> > [ 310.131110] sock_write_iter+0x2e4/0x6a0
> > [ 310.132433] ? core_sys_select+0x47d/0x780
> > [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> > [ 310.134591] __vfs_write+0x4e0/0x960
> > [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.136160] ? __vfs_read+0x950/0x950
> > [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> > [ 310.137711] vfs_write+0x155/0x4b0
> > [ 310.138454] SyS_write+0xf7/0x240
> > [ 310.139183] ? SyS_read+0x240/0x240
> > [ 310.139922] ? SyS_read+0x240/0x240
> > [ 310.140649] do_syscall_64+0x235/0x5b0
> > [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> > [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.146257] RIP: 0033:0x7f6f868fb070
> > [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> > 00007f6f868fb070
> > [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> > 0000000000000003
> > [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> > 0000000000000003
> > [ 310.153578]
> > [ 310.156362] Allocated by task 483:
> > [ 310.157812] save_stack_trace+0x1b/0x20
> > [ 310.159274] save_stack+0x43/0xd0
> > [ 310.160663] kasan_kmalloc+0xad/0xe0
> > [ 310.161943] __kmalloc+0x105/0x230
> > [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> > [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> > [ 310.165536] setup_vq+0x136/0x620
> > [ 310.166286] vp_setup_vq+0x13d/0x6d0
> > [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> > [ 310.167855] vp_find_vqs+0x71/0x410
> > [ 310.168641] vp_modern_find_vqs+0x21/0x140
> > [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> > [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> > [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> > [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> > [ 310.172918] do_setlink+0x2c33/0x3bc0
> > [ 310.173703] rtnl_setlink+0x245/0x380
> > [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.175344] netlink_rcv_skb+0x213/0x450
> > [ 310.176166] rtnetlink_rcv+0x28/0x30
> > [ 310.176990] netlink_unicast+0x4a0/0x6c0
> > [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> > [ 310.178646] sock_sendmsg+0xba/0xf0
> > [ 310.179435] SYSC_sendto+0x31d/0x620
> > [ 310.180229] SyS_sendto+0xe/0x10
> > [ 310.181004] do_syscall_64+0x235/0x5b0
> > [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.182595]
> > [ 310.183217] Freed by task 483:
> > [ 310.183934] save_stack_trace+0x1b/0x20
> > [ 310.184801] save_stack+0x43/0xd0
> > [ 310.187187] kasan_slab_free+0x72/0xc0
> > [ 310.188530] kfree+0x94/0x1a0
> > [ 310.189797] vring_del_virtqueue+0x19a/0x430
> > [ 310.191221] del_vq+0x11c/0x250
> > [ 310.192474] vp_del_vqs+0x379/0xc30
> > [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> > [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> > [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> > [ 310.197072] do_setlink+0x2c33/0x3bc0
> > [ 310.197804] rtnl_setlink+0x245/0x380
> > [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> > [ 310.199283] netlink_rcv_skb+0x213/0x450
> > [ 310.200036] rtnetlink_rcv+0x28/0x30
> > [ 310.200754] netlink_unicast+0x4a0/0x6c0
> > [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> > [ 310.202236] sock_sendmsg+0xba/0xf0
> > [ 310.202947] SYSC_sendto+0x31d/0x620
> > [ 310.203660] SyS_sendto+0xe/0x10
> > [ 310.204340] do_syscall_64+0x235/0x5b0
> > [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> > [ 310.205792]
> > [ 310.206350] The buggy address belongs to the object at ffff88006aa64200
> > [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> > [ 310.208149] The buggy address is located 32 bytes inside of
> > [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> > [ 310.209929] The buggy address belongs to the page:
> > [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping: (null)
> > index:0x0 compound_mapcount: 0
> > [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> > [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> > 0000000100030003
> > [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> > 0000000000000000
> > [ 310.215635] page dumped because: kasan: bad access detected
> > [ 310.218989]
> > [ 310.220398] Memory state around the buggy address:
> > [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > fc fc
> > [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.227400] ^
> > [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> > fb fb
> > [ 310.230639]
> > ==================================================================
> > [ 310.231788] Disabling lock debugging due to kernel taint
> > [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> > [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user memory
> > access
> > [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> > [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse irqbypass
> > i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw virtio_balloon
> > pata_acpi virtio_net virtio_blk
> > [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+ #2
> > [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-20170228_101828-anatol 04/01/2014
> > [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> > [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net]
> > [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> > [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > 0000000000000004
> > [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> > 0000000000000020
> > [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> > ffff880066b3c400
> > [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> > ffff880066b3c400
> > [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> > ffff88006afc9001
> > [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> > knlGS:0000000000000000
> > [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> > 00000000000006f0
> > [ 310.266178] Call Trace:
> > [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> > [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> > [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> > [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > [ 310.273076] ? update_load_avg+0x1281/0x29f0
> > [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> > [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> > [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> > [ 310.277548] ? rb_insert_color+0x1590/0x1590
> > [ 310.280172] ? netif_skb_features+0x920/0x920
> > [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> > [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> > [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> > [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> > [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> > [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> > [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> > [ 310.291837] ? update_curr+0x1ef/0x750
> > [ 310.292826] ? update_stack_state+0x402/0x780
> > [ 310.293827] ? account_entity_enqueue+0x730/0x730
> > [ 310.294831] ? update_stack_state+0x402/0x780
> > [ 310.295818] ? update_curr_fair+0x70/0x70
> > [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.297693] ? dequeue_entity+0x27a/0x1520
> > [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> > [ 310.299484] ? yield_to_task_fair+0x110/0x110
> > [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> > [ 310.301246] ? __free_insn_slot+0x600/0x600
> > [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> > [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> > [ 310.303883] dev_queue_xmit+0x10/0x20
> > [ 310.304711] ip_finish_output2+0xacf/0x12a0
> > [ 310.305558] ? dequeue_entity+0x1520/0x1520
> > [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> > [ 310.307320] ? save_stack_trace+0x1b/0x20
> > [ 310.308133] ? save_stack+0x43/0xd0
> > [ 310.309081] ? kasan_slab_free+0x72/0xc0
> > [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> > [ 310.311406] ? tcp_ack+0x2730/0x7450
> > [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> > [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> > [ 310.313769] ? __release_sock+0x14a/0x2b0
> > [ 310.314550] ? release_sock+0xa8/0x270
> > [ 310.315330] ? inet_sendmsg+0x111/0x590
> > [ 310.316100] ? sock_sendmsg+0xba/0xf0
> > [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> > [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> > [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> > [ 310.320800] ? update_stack_state+0x402/0x780
> > [ 310.321590] ip_finish_output+0x407/0x880
> > [ 310.322347] ? ip_finish_output+0x407/0x880
> > [ 310.323138] ? update_stack_state+0x402/0x780
> > [ 310.323948] ip_output+0x1c0/0x640
> > [ 310.324661] ? ip_mc_output+0x1350/0x1350
> > [ 310.325415] ? __sk_dst_check+0x164/0x370
> > [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> > [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> > [ 310.328684] ? memcpy+0x45/0x50
> > [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> > [ 310.330180] ip_local_out+0x70/0x90
> > [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> > [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> > [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> > [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> > [ 310.334887] ? sock_sendmsg+0xba/0xf0
> > [ 310.335637] ? __vfs_write+0x4e0/0x960
> > [ 310.336391] ? vfs_write+0x155/0x4b0
> > [ 310.337135] ? SyS_write+0xf7/0x240
> > [ 310.337861] ? do_syscall_64+0x235/0x5b0
> > [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.339443] ? sock_sendmsg+0xba/0xf0
> > [ 310.341675] ? do_syscall_64+0x235/0x5b0
> > [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> > [ 310.344095] ? radix_tree_lookup+0xd/0x10
> > [ 310.344871] ? get_work_pool+0xcd/0x150
> > [ 310.345635] ? check_flush_dependency+0x330/0x330
> > [ 310.346466] tcp_write_xmit+0x498/0x52a0
> > [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> > [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> > [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> > [ 310.351261] ? memset+0x31/0x40
> > [ 310.352054] ? __check_object_size+0x22e/0x55c
> > [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> > [ 310.353686] ? check_stack_object+0x120/0x120
> > [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> > [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> > [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> > [ 310.357016] tcp_push+0x47c/0xbd0
> > [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> > [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> > [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> > [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> > [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> > [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> > [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> > [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> > [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> > [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> > [ 310.366865] ? __wake_up+0x44/0x50
> > [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> > [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> > [ 310.369283] ? __check_object_size+0x22e/0x55c
> > [ 310.370129] inet_sendmsg+0x111/0x590
> > [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> > [ 310.373449] sock_sendmsg+0xba/0xf0
> > [ 310.374217] sock_write_iter+0x2e4/0x6a0
> > [ 310.375005] ? core_sys_select+0x47d/0x780
> > [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> > [ 310.376607] __vfs_write+0x4e0/0x960
> > [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> > [ 310.378864] ? __vfs_read+0x950/0x950
> > [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> > [ 310.381092] vfs_write+0x155/0x4b0
> > [ 310.381877] SyS_write+0xf7/0x240
> > [ 310.382616] ? SyS_read+0x240/0x240
> > [ 310.383404] ? SyS_read+0x240/0x240
> > [ 310.384159] do_syscall_64+0x235/0x5b0
> > [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> > [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> > [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> > [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> > [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> > [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> > [ 310.390164] RIP: 0033:0x7f301f83c070
> > [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000001
> > [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> > 00007f301f83c070
> > [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> > 0000000000000003
> > [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> > 0000000000003000
> > [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> > 0000000000000000
> > [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> > 0000000000000003
> > [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02 00 00
> > 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1 e9 03
> > <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> > [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0 [virtio_net] RSP:
> > ffff880069e46540
> > [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> > [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation
> > range: 0xffffffff80000000-0xffffffffbfffffff)
> > [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> > interrupt
> >
> >
> > (gdb) l *(free_old_xmit_skbs+0x2b7)
> > 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> > 1046
> > 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> > 1048 {
> > 1049 struct sk_buff *skb;
> > 1050 unsigned int len;
> > 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> > 1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
> > 1053 unsigned int packets = 0;
> > 1054 unsigned int bytes = 0;
> > 1055
> >
> > Let me know if i need to provide more informations.
> >
> > Best regards.
> >
> > Jean-Philippe
>
> So del_vq done during xdp setup seems to race with regular xmit.
>
> Since commit 680557cf79f82623e2c4fd42733077d60a843513
> virtio_net: rework mergeable buffer handling
>
> we no longer must do the resets, we now have enough space
> to store a bit saying whether a buffer is xdp one or not.
>
> And that's probably a cleaner way to fix these issues than
> try to find and fix the race condition.
>
> John?
>
> --
> MST
I think I see the source of the race. virtio net calls
netif_device_detach and assumes no packets will be sent after
this point. However, all it does is stop all queues so
no new packets will be transmitted.
Try locking with HARD_TX_LOCK?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-05 23:52 ` [Qemu-devel] " Michael S. Tsirkin
@ 2017-06-22 6:15 ` jean-philippe menil
-1 siblings, 0 replies; 27+ messages in thread
From: jean-philippe menil @ 2017-06-22 6:15 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: netdev, John Fastabend, qemu-devel, virtualization
[-- Attachment #1.1: Type: text/plain, Size: 24075 bytes --]
2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > > Hi,
> > >
> > > while playing with xdp and ebpf, i'm hitting the following:
> > >
> > > [ 309.993136]
> > > ==================================================================
> > > [ 309.994735] BUG: KASAN: use-after-free in
> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > > [ 310.000650]
> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
> > > [ 310.006495] Call Trace:
> > > [ 310.007610] dump_stack+0xb8/0x14c
> > > [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> > > [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> > > [ 310.011203] print_address_description+0x6f/0x280
> > > [ 310.012416] kasan_report+0x27a/0x370
> > > [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> > > [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > > [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> > > [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> > > [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> > > [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> > > [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > > [ 310.024874] ? update_load_avg+0x1281/0x29f0
> > > [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> > > [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> > > [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> > > [ 310.029731] ? netif_skb_features+0x920/0x920
> > > [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> > > [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> > > [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> > > [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > > [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> > > [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> > > [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> > > [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> > > [ 310.048301] ? update_stack_state+0x402/0x780
> > > [ 310.049307] ? account_entity_enqueue+0x730/0x730
> > > [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> > > [ 310.051286] ? update_curr_fair+0x70/0x70
> > > [ 310.052206] ? enqueue_entity+0x2450/0x2450
> > > [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.054082] ? dequeue_entity+0x27a/0x1520
> > > [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> > > [ 310.055822] ? yield_to_task_fair+0x110/0x110
> > > [ 310.056708] ? set_next_entity+0x2f2/0xa90
> > > [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> > > [ 310.058457] dev_queue_xmit+0x10/0x20
> > > [ 310.059298] ip_finish_output2+0xacf/0x12a0
> > > [ 310.060160] ? dequeue_entity+0x1520/0x1520
> > > [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> > > [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> > > [ 310.066677] ? __switch_to+0x685/0xda0
> > > [ 310.068166] ? load_balance+0x38f0/0x38f0
> > > [ 310.069544] ? compat_start_thread+0x80/0x80
> > > [ 310.070989] ? trace_find_cmdline+0x60/0x60
> > > [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> > > [ 310.073579] ip_finish_output+0x407/0x880
> > > [ 310.074441] ? ip_finish_output+0x407/0x880
> > > [ 310.075255] ? update_stack_state+0x402/0x780
> > > [ 310.076076] ip_output+0x1c0/0x640
> > > [ 310.076843] ? ip_mc_output+0x1350/0x1350
> > > [ 310.077642] ? __sk_dst_check+0x164/0x370
> > > [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> > > [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > > [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> > > [ 310.081097] ? memcpy+0x45/0x50
> > > [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> > > [ 310.082676] ip_local_out+0x70/0x90
> > > [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> > > [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> > > [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> > > [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> > > [ 310.087524] ? sock_sendmsg+0xba/0xf0
> > > [ 310.088298] ? __vfs_write+0x4e0/0x960
> > > [ 310.089074] ? vfs_write+0x155/0x4b0
> > > [ 310.089838] ? SyS_write+0xf7/0x240
> > > [ 310.090593] ? do_syscall_64+0x235/0x5b0
> > > [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.094690] ? sock_sendmsg+0xba/0xf0
> > > [ 310.096133] ? do_syscall_64+0x235/0x5b0
> > > [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> > > [ 310.100539] ? radix_tree_lookup+0xd/0x10
> > > [ 310.101894] ? get_work_pool+0xcd/0x150
> > > [ 310.103216] ? check_flush_dependency+0x330/0x330
> > > [ 310.104113] tcp_write_xmit+0x498/0x52a0
> > > [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> > > [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> > > [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> > > [ 310.107331] ? memset+0x31/0x40
> > > [ 310.108070] ? __check_object_size+0x22e/0x55c
> > > [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> > > [ 310.109690] ? check_stack_object+0x120/0x120
> > > [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> > > [ 310.112159] tcp_push+0x47c/0xbd0
> > > [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> > > [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> > > [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> > > [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > > [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> > > [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> > > [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> > > [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> > > [ 310.121224] ? __wake_up+0x44/0x50
> > > [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> > > [ 310.122898] ? __check_object_size+0x22e/0x55c
> > > [ 310.125380] inet_sendmsg+0x111/0x590
> > > [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.129817] sock_sendmsg+0xba/0xf0
> > > [ 310.131110] sock_write_iter+0x2e4/0x6a0
> > > [ 310.132433] ? core_sys_select+0x47d/0x780
> > > [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> > > [ 310.134591] __vfs_write+0x4e0/0x960
> > > [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> > > [ 310.136160] ? __vfs_read+0x950/0x950
> > > [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> > > [ 310.137711] vfs_write+0x155/0x4b0
> > > [ 310.138454] SyS_write+0xf7/0x240
> > > [ 310.139183] ? SyS_read+0x240/0x240
> > > [ 310.139922] ? SyS_read+0x240/0x240
> > > [ 310.140649] do_syscall_64+0x235/0x5b0
> > > [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> > > [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> > > [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> > > [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> > > [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> > > [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.146257] RIP: 0033:0x7f6f868fb070
> > > [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000001
> > > [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> > > 00007f6f868fb070
> > > [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> > > 0000000000000003
> > > [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> > > 0000000000003000
> > > [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> > > 0000000000000000
> > > [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> > > 0000000000000003
> > > [ 310.153578]
> > > [ 310.156362] Allocated by task 483:
> > > [ 310.157812] save_stack_trace+0x1b/0x20
> > > [ 310.159274] save_stack+0x43/0xd0
> > > [ 310.160663] kasan_kmalloc+0xad/0xe0
> > > [ 310.161943] __kmalloc+0x105/0x230
> > > [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> > > [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> > > [ 310.165536] setup_vq+0x136/0x620
> > > [ 310.166286] vp_setup_vq+0x13d/0x6d0
> > > [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> > > [ 310.167855] vp_find_vqs+0x71/0x410
> > > [ 310.168641] vp_modern_find_vqs+0x21/0x140
> > > [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> > > [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> > > [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> > > [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> > > [ 310.172918] do_setlink+0x2c33/0x3bc0
> > > [ 310.173703] rtnl_setlink+0x245/0x380
> > > [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> > > [ 310.175344] netlink_rcv_skb+0x213/0x450
> > > [ 310.176166] rtnetlink_rcv+0x28/0x30
> > > [ 310.176990] netlink_unicast+0x4a0/0x6c0
> > > [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> > > [ 310.178646] sock_sendmsg+0xba/0xf0
> > > [ 310.179435] SYSC_sendto+0x31d/0x620
> > > [ 310.180229] SyS_sendto+0xe/0x10
> > > [ 310.181004] do_syscall_64+0x235/0x5b0
> > > [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> > > [ 310.182595]
> > > [ 310.183217] Freed by task 483:
> > > [ 310.183934] save_stack_trace+0x1b/0x20
> > > [ 310.184801] save_stack+0x43/0xd0
> > > [ 310.187187] kasan_slab_free+0x72/0xc0
> > > [ 310.188530] kfree+0x94/0x1a0
> > > [ 310.189797] vring_del_virtqueue+0x19a/0x430
> > > [ 310.191221] del_vq+0x11c/0x250
> > > [ 310.192474] vp_del_vqs+0x379/0xc30
> > > [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> > > [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> > > [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> > > [ 310.197072] do_setlink+0x2c33/0x3bc0
> > > [ 310.197804] rtnl_setlink+0x245/0x380
> > > [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> > > [ 310.199283] netlink_rcv_skb+0x213/0x450
> > > [ 310.200036] rtnetlink_rcv+0x28/0x30
> > > [ 310.200754] netlink_unicast+0x4a0/0x6c0
> > > [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> > > [ 310.202236] sock_sendmsg+0xba/0xf0
> > > [ 310.202947] SYSC_sendto+0x31d/0x620
> > > [ 310.203660] SyS_sendto+0xe/0x10
> > > [ 310.204340] do_syscall_64+0x235/0x5b0
> > > [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> > > [ 310.205792]
> > > [ 310.206350] The buggy address belongs to the object at
> ffff88006aa64200
> > > [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> > > [ 310.208149] The buggy address is located 32 bytes inside of
> > > [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> > > [ 310.209929] The buggy address belongs to the page:
> > > [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping:
> (null)
> > > index:0x0 compound_mapcount: 0
> > > [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> > > [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> > > 0000000100030003
> > > [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> > > 0000000000000000
> > > [ 310.215635] page dumped because: kasan: bad access detected
> > > [ 310.218989]
> > > [ 310.220398] Memory state around the buggy address:
> > > [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> > > fc fc
> > > [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> > > fc fc
> > > [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.227400] ^
> > > [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.230639]
> > > ==================================================================
> > > [ 310.231788] Disabling lock debugging due to kernel taint
> > > [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> > > [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user
> memory
> > > access
> > > [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> > > [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse
> irqbypass
> > > i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw
> virtio_balloon
> > > pata_acpi virtio_net virtio_blk
> > > [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+
> #2
> > > [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
> > > [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> > > [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0
> [virtio_net]
> > > [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> > > [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > > 0000000000000004
> > > [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> > > 0000000000000020
> > > [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> > > ffff880066b3c400
> > > [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> > > ffff880066b3c400
> > > [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> > > ffff88006afc9001
> > > [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> > > knlGS:0000000000000000
> > > [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> > > 00000000000006f0
> > > [ 310.266178] Call Trace:
> > > [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > > [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> > > [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> > > [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> > > [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > > [ 310.273076] ? update_load_avg+0x1281/0x29f0
> > > [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> > > [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> > > [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> > > [ 310.277548] ? rb_insert_color+0x1590/0x1590
> > > [ 310.280172] ? netif_skb_features+0x920/0x920
> > > [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> > > [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> > > [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> > > [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > > [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> > > [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> > > [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> > > [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> > > [ 310.291837] ? update_curr+0x1ef/0x750
> > > [ 310.292826] ? update_stack_state+0x402/0x780
> > > [ 310.293827] ? account_entity_enqueue+0x730/0x730
> > > [ 310.294831] ? update_stack_state+0x402/0x780
> > > [ 310.295818] ? update_curr_fair+0x70/0x70
> > > [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.297693] ? dequeue_entity+0x27a/0x1520
> > > [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> > > [ 310.299484] ? yield_to_task_fair+0x110/0x110
> > > [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> > > [ 310.301246] ? __free_insn_slot+0x600/0x600
> > > [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> > > [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> > > [ 310.303883] dev_queue_xmit+0x10/0x20
> > > [ 310.304711] ip_finish_output2+0xacf/0x12a0
> > > [ 310.305558] ? dequeue_entity+0x1520/0x1520
> > > [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> > > [ 310.307320] ? save_stack_trace+0x1b/0x20
> > > [ 310.308133] ? save_stack+0x43/0xd0
> > > [ 310.309081] ? kasan_slab_free+0x72/0xc0
> > > [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> > > [ 310.311406] ? tcp_ack+0x2730/0x7450
> > > [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> > > [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> > > [ 310.313769] ? __release_sock+0x14a/0x2b0
> > > [ 310.314550] ? release_sock+0xa8/0x270
> > > [ 310.315330] ? inet_sendmsg+0x111/0x590
> > > [ 310.316100] ? sock_sendmsg+0xba/0xf0
> > > [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> > > [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> > > [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> > > [ 310.320800] ? update_stack_state+0x402/0x780
> > > [ 310.321590] ip_finish_output+0x407/0x880
> > > [ 310.322347] ? ip_finish_output+0x407/0x880
> > > [ 310.323138] ? update_stack_state+0x402/0x780
> > > [ 310.323948] ip_output+0x1c0/0x640
> > > [ 310.324661] ? ip_mc_output+0x1350/0x1350
> > > [ 310.325415] ? __sk_dst_check+0x164/0x370
> > > [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> > > [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > > [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> > > [ 310.328684] ? memcpy+0x45/0x50
> > > [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> > > [ 310.330180] ip_local_out+0x70/0x90
> > > [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> > > [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> > > [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> > > [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> > > [ 310.334887] ? sock_sendmsg+0xba/0xf0
> > > [ 310.335637] ? __vfs_write+0x4e0/0x960
> > > [ 310.336391] ? vfs_write+0x155/0x4b0
> > > [ 310.337135] ? SyS_write+0xf7/0x240
> > > [ 310.337861] ? do_syscall_64+0x235/0x5b0
> > > [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.339443] ? sock_sendmsg+0xba/0xf0
> > > [ 310.341675] ? do_syscall_64+0x235/0x5b0
> > > [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> > > [ 310.344095] ? radix_tree_lookup+0xd/0x10
> > > [ 310.344871] ? get_work_pool+0xcd/0x150
> > > [ 310.345635] ? check_flush_dependency+0x330/0x330
> > > [ 310.346466] tcp_write_xmit+0x498/0x52a0
> > > [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> > > [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> > > [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> > > [ 310.351261] ? memset+0x31/0x40
> > > [ 310.352054] ? __check_object_size+0x22e/0x55c
> > > [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> > > [ 310.353686] ? check_stack_object+0x120/0x120
> > > [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> > > [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> > > [ 310.357016] tcp_push+0x47c/0xbd0
> > > [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> > > [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> > > [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > > [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> > > [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> > > [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> > > [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> > > [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> > > [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> > > [ 310.366865] ? __wake_up+0x44/0x50
> > > [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> > > [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> > > [ 310.369283] ? __check_object_size+0x22e/0x55c
> > > [ 310.370129] inet_sendmsg+0x111/0x590
> > > [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.373449] sock_sendmsg+0xba/0xf0
> > > [ 310.374217] sock_write_iter+0x2e4/0x6a0
> > > [ 310.375005] ? core_sys_select+0x47d/0x780
> > > [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> > > [ 310.376607] __vfs_write+0x4e0/0x960
> > > [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> > > [ 310.378864] ? __vfs_read+0x950/0x950
> > > [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> > > [ 310.381092] vfs_write+0x155/0x4b0
> > > [ 310.381877] SyS_write+0xf7/0x240
> > > [ 310.382616] ? SyS_read+0x240/0x240
> > > [ 310.383404] ? SyS_read+0x240/0x240
> > > [ 310.384159] do_syscall_64+0x235/0x5b0
> > > [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> > > [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> > > [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> > > [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> > > [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> > > [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.390164] RIP: 0033:0x7f301f83c070
> > > [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000001
> > > [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> > > 00007f301f83c070
> > > [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> > > 0000000000000003
> > > [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> > > 0000000000003000
> > > [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> > > 0000000000000000
> > > [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> > > 0000000000000003
> > > [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02
> 00 00
> > > 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1
> e9 03
> > > <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> > > [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0
> [virtio_net] RSP:
> > > ffff880069e46540
> > > [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> > > [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> > > [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000
> (relocation
> > > range: 0xffffffff80000000-0xffffffffbfffffff)
> > > [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > >
> > >
> > > (gdb) l *(free_old_xmit_skbs+0x2b7)
> > > 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> > > 1046
> > > 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> > > 1048 {
> > > 1049 struct sk_buff *skb;
> > > 1050 unsigned int len;
> > > 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> > > 1052 struct virtnet_stats *stats =
> this_cpu_ptr(vi->stats);
> > > 1053 unsigned int packets = 0;
> > > 1054 unsigned int bytes = 0;
> > > 1055
> > >
> > > Let me know if i need to provide more informations.
> > >
> > > Best regards.
> > >
> > > Jean-Philippe
> >
> > So del_vq done during xdp setup seems to race with regular xmit.
> >
> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
> > virtio_net: rework mergeable buffer handling
> >
> > we no longer must do the resets, we now have enough space
> > to store a bit saying whether a buffer is xdp one or not.
> >
> > And that's probably a cleaner way to fix these issues than
> > try to find and fix the race condition.
> >
> > John?
> >
> > --
> > MST
>
>
> I think I see the source of the race. virtio net calls
> netif_device_detach and assumes no packets will be sent after
> this point. However, all it does is stop all queues so
> no new packets will be transmitted.
>
> Try locking with HARD_TX_LOCK?
>
>
> --
> MST
>
Hi Michael,
from what i see, the race appear when we hit virtnet_reset in
virtnet_xdp_set.
virtnet_reset
_remove_vq_common
virtnet_del_vqs
virtnet_free_queues
kfree(vi->sq)
when the xdp program (with two instances of the program to trigger it
faster) is added or removed.
It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
running the xdp_ttl tool from Jesper.
For now, i'm able to continue my qualification, testing if xdp_qp is not
null, but do not seem to be a sustainable trick.
if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
Maybe it will be more clear to you with theses informations.
Best regards.
Jean-Philippe
[-- Attachment #1.2: Type: text/html, Size: 30354 bytes --]
[-- Attachment #2: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-22 6:15 ` jean-philippe menil
0 siblings, 0 replies; 27+ messages in thread
From: jean-philippe menil @ 2017-06-22 6:15 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: netdev, jasowang, John Fastabend, virtualization, qemu-devel
2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > > Hi,
> > >
> > > while playing with xdp and ebpf, i'm hitting the following:
> > >
> > > [ 309.993136]
> > > ==================================================================
> > > [ 309.994735] BUG: KASAN: use-after-free in
> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > > [ 310.000650]
> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
> > > [ 310.006495] Call Trace:
> > > [ 310.007610] dump_stack+0xb8/0x14c
> > > [ 310.008748] ? _atomic_dec_and_lock+0x174/0x174
> > > [ 310.009998] ? pm_qos_get_value.part.7+0x6/0x6
> > > [ 310.011203] print_address_description+0x6f/0x280
> > > [ 310.012416] kasan_report+0x27a/0x370
> > > [ 310.013573] ? free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 310.014900] __asan_report_load8_noabort+0x19/0x20
> > > [ 310.016136] free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 310.017467] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > > [ 310.018759] ? packet_rcv+0x20d0/0x20d0
> > > [ 310.019950] ? dev_queue_xmit_nit+0x5cd/0xaf0
> > > [ 310.021168] start_xmit+0x1b4/0x1b10 [virtio_net]
> > > [ 310.022413] ? default_device_exit+0x2d0/0x2d0
> > > [ 310.023634] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > > [ 310.024874] ? update_load_avg+0x1281/0x29f0
> > > [ 310.026059] dev_hard_start_xmit+0x1ea/0x7f0
> > > [ 310.027247] ? validate_xmit_skb_list+0x100/0x100
> > > [ 310.028470] ? validate_xmit_skb+0x7f/0xc10
> > > [ 310.029731] ? netif_skb_features+0x920/0x920
> > > [ 310.033469] ? __skb_tx_hash+0x2f0/0x2f0
> > > [ 310.035615] ? validate_xmit_skb_list+0xa3/0x100
> > > [ 310.037782] sch_direct_xmit+0x2eb/0x7a0
> > > [ 310.039842] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > > [ 310.041980] ? netdev_pick_tx+0x212/0x2b0
> > > [ 310.043868] __dev_queue_xmit+0x12fa/0x20b0
> > > [ 310.045564] ? netdev_pick_tx+0x2b0/0x2b0
> > > [ 310.047210] ? __account_cfs_rq_runtime+0x630/0x630
> > > [ 310.048301] ? update_stack_state+0x402/0x780
> > > [ 310.049307] ? account_entity_enqueue+0x730/0x730
> > > [ 310.050322] ? __rb_erase_color+0x27d0/0x27d0
> > > [ 310.051286] ? update_curr_fair+0x70/0x70
> > > [ 310.052206] ? enqueue_entity+0x2450/0x2450
> > > [ 310.053124] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.054082] ? dequeue_entity+0x27a/0x1520
> > > [ 310.054967] ? bpf_prog_alloc+0x320/0x320
> > > [ 310.055822] ? yield_to_task_fair+0x110/0x110
> > > [ 310.056708] ? set_next_entity+0x2f2/0xa90
> > > [ 310.057574] ? dequeue_task_fair+0xc09/0x2ec0
> > > [ 310.058457] dev_queue_xmit+0x10/0x20
> > > [ 310.059298] ip_finish_output2+0xacf/0x12a0
> > > [ 310.060160] ? dequeue_entity+0x1520/0x1520
> > > [ 310.063410] ? ip_fragment.constprop.47+0x220/0x220
> > > [ 310.065078] ? ring_buffer_set_clock+0x50/0x50
> > > [ 310.066677] ? __switch_to+0x685/0xda0
> > > [ 310.068166] ? load_balance+0x38f0/0x38f0
> > > [ 310.069544] ? compat_start_thread+0x80/0x80
> > > [ 310.070989] ? trace_find_cmdline+0x60/0x60
> > > [ 310.072402] ? rt_cpu_seq_show+0x2d0/0x2d0
> > > [ 310.073579] ip_finish_output+0x407/0x880
> > > [ 310.074441] ? ip_finish_output+0x407/0x880
> > > [ 310.075255] ? update_stack_state+0x402/0x780
> > > [ 310.076076] ip_output+0x1c0/0x640
> > > [ 310.076843] ? ip_mc_output+0x1350/0x1350
> > > [ 310.077642] ? __sk_dst_check+0x164/0x370
> > > [ 310.078441] ? complete_formation.isra.53+0xa30/0xa30
> > > [ 310.079313] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > > [ 310.080265] ? sock_prot_inuse_add+0xa0/0xa0
> > > [ 310.081097] ? memcpy+0x45/0x50
> > > [ 310.081850] ? __copy_skb_header+0x1fa/0x280
> > > [ 310.082676] ip_local_out+0x70/0x90
> > > [ 310.083448] ip_queue_xmit+0x8a1/0x22a0
> > > [ 310.084236] ? ip_build_and_send_pkt+0xe80/0xe80
> > > [ 310.085079] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.085884] tcp_transmit_skb+0x187a/0x3e00
> > > [ 310.086696] ? __tcp_select_window+0xaf0/0xaf0
> > > [ 310.087524] ? sock_sendmsg+0xba/0xf0
> > > [ 310.088298] ? __vfs_write+0x4e0/0x960
> > > [ 310.089074] ? vfs_write+0x155/0x4b0
> > > [ 310.089838] ? SyS_write+0xf7/0x240
> > > [ 310.090593] ? do_syscall_64+0x235/0x5b0
> > > [ 310.091372] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.094690] ? sock_sendmsg+0xba/0xf0
> > > [ 310.096133] ? do_syscall_64+0x235/0x5b0
> > > [ 310.097593] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.099157] ? tcp_init_tso_segs+0x1e0/0x1e0
> > > [ 310.100539] ? radix_tree_lookup+0xd/0x10
> > > [ 310.101894] ? get_work_pool+0xcd/0x150
> > > [ 310.103216] ? check_flush_dependency+0x330/0x330
> > > [ 310.104113] tcp_write_xmit+0x498/0x52a0
> > > [ 310.104905] ? kasan_unpoison_shadow+0x35/0x50
> > > [ 310.105729] ? kasan_kmalloc+0xad/0xe0
> > > [ 310.106505] ? tcp_transmit_skb+0x3e00/0x3e00
> > > [ 310.107331] ? memset+0x31/0x40
> > > [ 310.108070] ? __check_object_size+0x22e/0x55c
> > > [ 310.108895] ? skb_pull_rcsum+0x2b0/0x2b0
> > > [ 310.109690] ? check_stack_object+0x120/0x120
> > > [ 310.110512] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.111315] __tcp_push_pending_frames+0x8d/0x2a0
> > > [ 310.112159] tcp_push+0x47c/0xbd0
> > > [ 310.112912] ? copy_from_iter_full+0x21e/0xc70
> > > [ 310.113747] ? sock_warn_obsolete_bsdism+0x70/0x70
> > > [ 310.114604] ? tcp_splice_data_recv+0x1c0/0x1c0
> > > [ 310.115436] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > > [ 310.116324] tcp_sendmsg+0xd6d/0x43f0
> > > [ 310.117106] ? tcp_sendpage+0x2170/0x2170
> > > [ 310.117911] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.118718] ? remove_wait_queue+0x196/0x3b0
> > > [ 310.119535] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.120365] ? add_wait_queue_exclusive+0x290/0x290
> > > [ 310.121224] ? __wake_up+0x44/0x50
> > > [ 310.121985] ? n_tty_read+0x9f9/0x19d0
> > > [ 310.122898] ? __check_object_size+0x22e/0x55c
> > > [ 310.125380] inet_sendmsg+0x111/0x590
> > > [ 310.126863] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.128348] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.129817] sock_sendmsg+0xba/0xf0
> > > [ 310.131110] sock_write_iter+0x2e4/0x6a0
> > > [ 310.132433] ? core_sys_select+0x47d/0x780
> > > [ 310.133779] ? sock_sendmsg+0xf0/0xf0
> > > [ 310.134591] __vfs_write+0x4e0/0x960
> > > [ 310.135351] ? kvm_clock_get_cycles+0x1e/0x20
> > > [ 310.136160] ? __vfs_read+0x950/0x950
> > > [ 310.136931] ? rw_verify_area+0xbd/0x2b0
> > > [ 310.137711] vfs_write+0x155/0x4b0
> > > [ 310.138454] SyS_write+0xf7/0x240
> > > [ 310.139183] ? SyS_read+0x240/0x240
> > > [ 310.139922] ? SyS_read+0x240/0x240
> > > [ 310.140649] do_syscall_64+0x235/0x5b0
> > > [ 310.141390] ? trace_raw_output_sys_exit+0xf0/0xf0
> > > [ 310.142204] ? syscall_return_slowpath+0x240/0x240
> > > [ 310.143018] ? trace_do_page_fault+0xc4/0x3a0
> > > [ 310.143810] ? prepare_exit_to_usermode+0x124/0x160
> > > [ 310.144634] ? perf_trace_sys_enter+0x1080/0x1080
> > > [ 310.145447] entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.146257] RIP: 0033:0x7f6f868fb070
> > > [ 310.146999] RSP: 002b:00007fffed379578 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000001
> > > [ 310.148507] RAX: ffffffffffffffda RBX: 00000000000002e4 RCX:
> > > 00007f6f868fb070
> > > [ 310.149521] RDX: 00000000000002e4 RSI: 000055603b5cfc10 RDI:
> > > 0000000000000003
> > > [ 310.150532] RBP: 000055603b5aca60 R08: 0000000000000000 R09:
> > > 0000000000003000
> > > [ 310.151530] R10: 0000000000000008 R11: 0000000000000246 R12:
> > > 0000000000000000
> > > [ 310.152537] R13: 00007fffed37960f R14: 000055603a832e31 R15:
> > > 0000000000000003
> > > [ 310.153578]
> > > [ 310.156362] Allocated by task 483:
> > > [ 310.157812] save_stack_trace+0x1b/0x20
> > > [ 310.159274] save_stack+0x43/0xd0
> > > [ 310.160663] kasan_kmalloc+0xad/0xe0
> > > [ 310.161943] __kmalloc+0x105/0x230
> > > [ 310.163233] __vring_new_virtqueue+0xd1/0xee0
> > > [ 310.164623] vring_create_virtqueue+0x2e3/0x5e0
> > > [ 310.165536] setup_vq+0x136/0x620
> > > [ 310.166286] vp_setup_vq+0x13d/0x6d0
> > > [ 310.167059] vp_find_vqs_msix+0x46c/0xb50
> > > [ 310.167855] vp_find_vqs+0x71/0x410
> > > [ 310.168641] vp_modern_find_vqs+0x21/0x140
> > > [ 310.169453] init_vqs+0x957/0x1390 [virtio_net]
> > > [ 310.170306] virtnet_restore_up+0x4a/0x590 [virtio_net]
> > > [ 310.171214] virtnet_xdp+0x89f/0xdf0 [virtio_net]
> > > [ 310.172077] dev_change_xdp_fd+0x1ca/0x420
> > > [ 310.172918] do_setlink+0x2c33/0x3bc0
> > > [ 310.173703] rtnl_setlink+0x245/0x380
> > > [ 310.174511] rtnetlink_rcv_msg+0x530/0x9b0
> > > [ 310.175344] netlink_rcv_skb+0x213/0x450
> > > [ 310.176166] rtnetlink_rcv+0x28/0x30
> > > [ 310.176990] netlink_unicast+0x4a0/0x6c0
> > > [ 310.177807] netlink_sendmsg+0x9ec/0xe50
> > > [ 310.178646] sock_sendmsg+0xba/0xf0
> > > [ 310.179435] SYSC_sendto+0x31d/0x620
> > > [ 310.180229] SyS_sendto+0xe/0x10
> > > [ 310.181004] do_syscall_64+0x235/0x5b0
> > > [ 310.181783] return_from_SYSCALL_64+0x0/0x6a
> > > [ 310.182595]
> > > [ 310.183217] Freed by task 483:
> > > [ 310.183934] save_stack_trace+0x1b/0x20
> > > [ 310.184801] save_stack+0x43/0xd0
> > > [ 310.187187] kasan_slab_free+0x72/0xc0
> > > [ 310.188530] kfree+0x94/0x1a0
> > > [ 310.189797] vring_del_virtqueue+0x19a/0x430
> > > [ 310.191221] del_vq+0x11c/0x250
> > > [ 310.192474] vp_del_vqs+0x379/0xc30
> > > [ 310.193772] virtnet_del_vqs+0xad/0xe0 [virtio_net]
> > > [ 310.195064] virtnet_xdp+0x836/0xdf0 [virtio_net]
> > > [ 310.196231] dev_change_xdp_fd+0x37c/0x420
> > > [ 310.197072] do_setlink+0x2c33/0x3bc0
> > > [ 310.197804] rtnl_setlink+0x245/0x380
> > > [ 310.198530] rtnetlink_rcv_msg+0x530/0x9b0
> > > [ 310.199283] netlink_rcv_skb+0x213/0x450
> > > [ 310.200036] rtnetlink_rcv+0x28/0x30
> > > [ 310.200754] netlink_unicast+0x4a0/0x6c0
> > > [ 310.201496] netlink_sendmsg+0x9ec/0xe50
> > > [ 310.202236] sock_sendmsg+0xba/0xf0
> > > [ 310.202947] SYSC_sendto+0x31d/0x620
> > > [ 310.203660] SyS_sendto+0xe/0x10
> > > [ 310.204340] do_syscall_64+0x235/0x5b0
> > > [ 310.205050] return_from_SYSCALL_64+0x0/0x6a
> > > [ 310.205792]
> > > [ 310.206350] The buggy address belongs to the object at
> ffff88006aa64200
> > > [ 310.206350] which belongs to the cache kmalloc-8192 of size 8192
> > > [ 310.208149] The buggy address is located 32 bytes inside of
> > > [ 310.208149] 8192-byte region [ffff88006aa64200, ffff88006aa66200)
> > > [ 310.209929] The buggy address belongs to the page:
> > > [ 310.210763] page:ffffea0001aa9800 count:1 mapcount:0 mapping:
> (null)
> > > index:0x0 compound_mapcount: 0
> > > [ 310.212499] flags: 0x1ffff8000008100(slab|head)
> > > [ 310.213373] raw: 01ffff8000008100 0000000000000000 0000000000000000
> > > 0000000100030003
> > > [ 310.214481] raw: dead000000000100 dead000000000200 ffff88006cc02700
> > > 0000000000000000
> > > [ 310.215635] page dumped because: kasan: bad access detected
> > > [ 310.218989]
> > > [ 310.220398] Memory state around the buggy address:
> > > [ 310.222141] ffff88006aa64100: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> > > fc fc
> > > [ 310.223996] ffff88006aa64180: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc
> > > fc fc
> > > [ 310.225469] >ffff88006aa64200: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.227400] ^
> > > [ 310.228367] ffff88006aa64280: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.229510] ffff88006aa64300: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb
> > > fb fb
> > > [ 310.230639]
> > > ==================================================================
> > > [ 310.231788] Disabling lock debugging due to kernel taint
> > > [ 310.233499] kasan: CONFIG_KASAN_INLINE enabled
> > > [ 310.236846] kasan: GPF could be caused by NULL-ptr deref or user
> memory
> > > access
> > > [ 310.239138] general protection fault: 0000 [#1] SMP KASAN
> > > [ 310.240926] Modules linked in: joydev kvm_intel kvm psmouse
> irqbypass
> > > i2c_piix4 qemu_fw_cfg ip_tables x_tables autofs4 serio_raw
> virtio_balloon
> > > pata_acpi virtio_net virtio_blk
> > > [ 310.243618] CPU: 0 PID: 352 Comm: sshd Tainted: G B 4.12.0-rc3+
> #2
> > > [ 310.245780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
> > > [ 310.249799] task: ffff880066ca8d80 task.stack: ffff880069e40000
> > > [ 310.251090] RIP: 0010:free_old_xmit_skbs.isra.29+0x9d/0x2e0
> [virtio_net]
> > > [ 310.252403] RSP: 0018:ffff880069e46540 EFLAGS: 00010202
> > > [ 310.253631] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > > 0000000000000004
> > > [ 310.255916] RDX: dffffc0000000000 RSI: 0000000000000008 RDI:
> > > 0000000000000020
> > > [ 310.258017] RBP: ffff880069e465e8 R08: ffff880069e45f10 R09:
> > > ffff880066b3c400
> > > [ 310.259430] R10: ffff880069e45e98 R11: 1ffff1000cd952f3 R12:
> > > ffff880066b3c400
> > > [ 310.260797] R13: ffff880066b3c400 R14: ffff88006afc9156 R15:
> > > ffff88006afc9001
> > > [ 310.262139] FS: 00007f3020f26680(0000) GS:ffff88006d000000(0000)
> > > knlGS:0000000000000000
> > > [ 310.263564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 310.264825] CR2: 00007efed4534010 CR3: 000000006986d000 CR4:
> > > 00000000000006f0
> > > [ 310.266178] Call Trace:
> > > [ 310.267231] ? virtnet_del_vqs+0xe0/0xe0 [virtio_net]
> > > [ 310.268453] ? packet_rcv+0x20d0/0x20d0
> > > [ 310.269559] start_xmit+0x1b4/0x1b10 [virtio_net]
> > > [ 310.270762] ? default_device_exit+0x2d0/0x2d0
> > > [ 310.271910] ? virtnet_remove+0xf0/0xf0 [virtio_net]
> > > [ 310.273076] ? update_load_avg+0x1281/0x29f0
> > > [ 310.274189] dev_hard_start_xmit+0x1ea/0x7f0
> > > [ 310.275295] ? validate_xmit_skb_list+0x100/0x100
> > > [ 310.276425] ? validate_xmit_skb+0x7f/0xc10
> > > [ 310.277548] ? rb_insert_color+0x1590/0x1590
> > > [ 310.280172] ? netif_skb_features+0x920/0x920
> > > [ 310.281275] ? __skb_tx_hash+0x2f0/0x2f0
> > > [ 310.282362] ? validate_xmit_skb_list+0xa3/0x100
> > > [ 310.283494] sch_direct_xmit+0x2eb/0x7a0
> > > [ 310.284559] ? dev_deactivate_queue.constprop.29+0x230/0x230
> > > [ 310.286448] ? netdev_pick_tx+0x212/0x2b0
> > > [ 310.288251] ? __account_cfs_rq_runtime+0x630/0x630
> > > [ 310.289707] __dev_queue_xmit+0x12fa/0x20b0
> > > [ 310.290788] ? netdev_pick_tx+0x2b0/0x2b0
> > > [ 310.291837] ? update_curr+0x1ef/0x750
> > > [ 310.292826] ? update_stack_state+0x402/0x780
> > > [ 310.293827] ? account_entity_enqueue+0x730/0x730
> > > [ 310.294831] ? update_stack_state+0x402/0x780
> > > [ 310.295818] ? update_curr_fair+0x70/0x70
> > > [ 310.296737] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.297693] ? dequeue_entity+0x27a/0x1520
> > > [ 310.298591] ? bpf_prog_alloc+0x320/0x320
> > > [ 310.299484] ? yield_to_task_fair+0x110/0x110
> > > [ 310.300385] ? unwind_dump+0x4e0/0x4e0
> > > [ 310.301246] ? __free_insn_slot+0x600/0x600
> > > [ 310.302125] ? unwind_dump+0x4e0/0x4e0
> > > [ 310.302975] ? dequeue_task_fair+0xc09/0x2ec0
> > > [ 310.303883] dev_queue_xmit+0x10/0x20
> > > [ 310.304711] ip_finish_output2+0xacf/0x12a0
> > > [ 310.305558] ? dequeue_entity+0x1520/0x1520
> > > [ 310.306393] ? ip_fragment.constprop.47+0x220/0x220
> > > [ 310.307320] ? save_stack_trace+0x1b/0x20
> > > [ 310.308133] ? save_stack+0x43/0xd0
> > > [ 310.309081] ? kasan_slab_free+0x72/0xc0
> > > [ 310.310614] ? kfree_skbmem+0xb6/0x1d0
> > > [ 310.311406] ? tcp_ack+0x2730/0x7450
> > > [ 310.312167] ? tcp_rcv_established+0xdbb/0x2db0
> > > [ 310.312987] ? tcp_v4_do_rcv+0x2bb/0x7a0
> > > [ 310.313769] ? __release_sock+0x14a/0x2b0
> > > [ 310.314550] ? release_sock+0xa8/0x270
> > > [ 310.315330] ? inet_sendmsg+0x111/0x590
> > > [ 310.316100] ? sock_sendmsg+0xba/0xf0
> > > [ 310.317403] ? sock_write_iter+0x2e4/0x6a0
> > > [ 310.318759] ? __rb_erase_color+0x27d0/0x27d0
> > > [ 310.319949] ? rt_cpu_seq_show+0x2d0/0x2d0
> > > [ 310.320800] ? update_stack_state+0x402/0x780
> > > [ 310.321590] ip_finish_output+0x407/0x880
> > > [ 310.322347] ? ip_finish_output+0x407/0x880
> > > [ 310.323138] ? update_stack_state+0x402/0x780
> > > [ 310.323948] ip_output+0x1c0/0x640
> > > [ 310.324661] ? ip_mc_output+0x1350/0x1350
> > > [ 310.325415] ? __sk_dst_check+0x164/0x370
> > > [ 310.326169] ? complete_formation.isra.53+0xa30/0xa30
> > > [ 310.327013] ? __read_once_size_nocheck.constprop.7+0x20/0x20
> > > [ 310.327896] ? sock_prot_inuse_add+0xa0/0xa0
> > > [ 310.328684] ? memcpy+0x45/0x50
> > > [ 310.329393] ? __copy_skb_header+0x1fa/0x280
> > > [ 310.330180] ip_local_out+0x70/0x90
> > > [ 310.330914] ip_queue_xmit+0x8a1/0x22a0
> > > [ 310.331676] ? ip_build_and_send_pkt+0xe80/0xe80
> > > [ 310.332517] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.333298] tcp_transmit_skb+0x187a/0x3e00
> > > [ 310.334085] ? __tcp_select_window+0xaf0/0xaf0
> > > [ 310.334887] ? sock_sendmsg+0xba/0xf0
> > > [ 310.335637] ? __vfs_write+0x4e0/0x960
> > > [ 310.336391] ? vfs_write+0x155/0x4b0
> > > [ 310.337135] ? SyS_write+0xf7/0x240
> > > [ 310.337861] ? do_syscall_64+0x235/0x5b0
> > > [ 310.338612] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.339443] ? sock_sendmsg+0xba/0xf0
> > > [ 310.341675] ? do_syscall_64+0x235/0x5b0
> > > [ 310.342441] ? entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.343298] ? tcp_init_tso_segs+0x1e0/0x1e0
> > > [ 310.344095] ? radix_tree_lookup+0xd/0x10
> > > [ 310.344871] ? get_work_pool+0xcd/0x150
> > > [ 310.345635] ? check_flush_dependency+0x330/0x330
> > > [ 310.346466] tcp_write_xmit+0x498/0x52a0
> > > [ 310.347826] ? kasan_unpoison_shadow+0x35/0x50
> > > [ 310.349243] ? kasan_kmalloc+0xad/0xe0
> > > [ 310.350156] ? tcp_transmit_skb+0x3e00/0x3e00
> > > [ 310.351261] ? memset+0x31/0x40
> > > [ 310.352054] ? __check_object_size+0x22e/0x55c
> > > [ 310.352881] ? skb_pull_rcsum+0x2b0/0x2b0
> > > [ 310.353686] ? check_stack_object+0x120/0x120
> > > [ 310.354506] ? tcp_v4_md5_lookup+0x13/0x20
> > > [ 310.355327] __tcp_push_pending_frames+0x8d/0x2a0
> > > [ 310.356174] ? tcp_cwnd_restart+0x169/0x440
> > > [ 310.357016] tcp_push+0x47c/0xbd0
> > > [ 310.357777] ? copy_from_iter_full+0x21e/0xc70
> > > [ 310.358618] ? tcp_splice_data_recv+0x1c0/0x1c0
> > > [ 310.359463] ? iov_iter_copy_from_user_atomic+0xeb0/0xeb0
> > > [ 310.360355] ? tcp_send_mss+0x24/0x2b0
> > > [ 310.361135] tcp_sendmsg+0xd6d/0x43f0
> > > [ 310.361908] ? select_estimate_accuracy+0x440/0x440
> > > [ 310.362765] ? tcp_sendpage+0x2170/0x2170
> > > [ 310.363583] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.364392] ? remove_wait_queue+0x196/0x3b0
> > > [ 310.365205] ? set_fd_set.part.1+0x50/0x50
> > > [ 310.366005] ? add_wait_queue_exclusive+0x290/0x290
> > > [ 310.366865] ? __wake_up+0x44/0x50
> > > [ 310.367637] ? n_tty_read+0x9f9/0x19d0
> > > [ 310.368424] ? update_blocked_averages+0x9a0/0x9a0
> > > [ 310.369283] ? __check_object_size+0x22e/0x55c
> > > [ 310.370129] inet_sendmsg+0x111/0x590
> > > [ 310.371104] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.372571] ? inet_recvmsg+0x5e0/0x5e0
> > > [ 310.373449] sock_sendmsg+0xba/0xf0
> > > [ 310.374217] sock_write_iter+0x2e4/0x6a0
> > > [ 310.375005] ? core_sys_select+0x47d/0x780
> > > [ 310.375822] ? sock_sendmsg+0xf0/0xf0
> > > [ 310.376607] __vfs_write+0x4e0/0x960
> > > [ 310.377463] ? kvm_clock_get_cycles+0x1e/0x20
> > > [ 310.378864] ? __vfs_read+0x950/0x950
> > > [ 310.380178] ? rw_verify_area+0xbd/0x2b0
> > > [ 310.381092] vfs_write+0x155/0x4b0
> > > [ 310.381877] SyS_write+0xf7/0x240
> > > [ 310.382616] ? SyS_read+0x240/0x240
> > > [ 310.383404] ? SyS_read+0x240/0x240
> > > [ 310.384159] do_syscall_64+0x235/0x5b0
> > > [ 310.384930] ? trace_raw_output_sys_exit+0xf0/0xf0
> > > [ 310.385747] ? syscall_return_slowpath+0x240/0x240
> > > [ 310.386564] ? trace_do_page_fault+0xc4/0x3a0
> > > [ 310.387424] ? prepare_exit_to_usermode+0x124/0x160
> > > [ 310.388524] ? perf_trace_sys_enter+0x1080/0x1080
> > > [ 310.389347] entry_SYSCALL64_slow_path+0x25/0x25
> > > [ 310.390164] RIP: 0033:0x7f301f83c070
> > > [ 310.390906] RSP: 002b:00007ffff738fc78 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000001
> > > [ 310.391943] RAX: ffffffffffffffda RBX: 0000000000000564 RCX:
> > > 00007f301f83c070
> > > [ 310.392938] RDX: 0000000000000564 RSI: 000055cf87fb0748 RDI:
> > > 0000000000000003
> > > [ 310.393947] RBP: 000055cf87f8f090 R08: 0000000000000000 R09:
> > > 0000000000003000
> > > [ 310.394948] R10: 0000000000000008 R11: 0000000000000246 R12:
> > > 0000000000000000
> > > [ 310.395967] R13: 00007ffff738fd0f R14: 000055cf873dde31 R15:
> > > 0000000000000003
> > > [ 310.396969] Code: 00 00 48 89 5d d0 31 db 80 3c 02 00 0f 85 05 02
> 00 00
> > > 49 8b 45 00 48 ba 00 00 00 00 00 fc ff df 48 8d 78 20 48 89 f9 48 c1
> e9 03
> > > <80> 3c 11 00 0f 85 04 02 00 00 48 8b 58 20 48 ba 00 00 00 00 00
> > > [ 310.399937] RIP: free_old_xmit_skbs.isra.29+0x9d/0x2e0
> [virtio_net] RSP:
> > > ffff880069e46540
> > > [ 310.401120] ---[ end trace 89c5b0ea3f07debe ]---
> > > [ 310.403923] Kernel panic - not syncing: Fatal exception in interrupt
> > > [ 310.405942] Kernel Offset: 0x33200000 from 0xffffffff81000000
> (relocation
> > > range: 0xffffffff80000000-0xffffffffbfffffff)
> > > [ 310.408133] ---[ end Kernel panic - not syncing: Fatal exception in
> > > interrupt
> > >
> > >
> > > (gdb) l *(free_old_xmit_skbs+0x2b7)
> > > 0x22f7 is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
> > > 1046
> > > 1047 static void free_old_xmit_skbs(struct send_queue *sq)
> > > 1048 {
> > > 1049 struct sk_buff *skb;
> > > 1050 unsigned int len;
> > > 1051 struct virtnet_info *vi = sq->vq->vdev->priv;
> > > 1052 struct virtnet_stats *stats =
> this_cpu_ptr(vi->stats);
> > > 1053 unsigned int packets = 0;
> > > 1054 unsigned int bytes = 0;
> > > 1055
> > >
> > > Let me know if i need to provide more informations.
> > >
> > > Best regards.
> > >
> > > Jean-Philippe
> >
> > So del_vq done during xdp setup seems to race with regular xmit.
> >
> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
> > virtio_net: rework mergeable buffer handling
> >
> > we no longer must do the resets, we now have enough space
> > to store a bit saying whether a buffer is xdp one or not.
> >
> > And that's probably a cleaner way to fix these issues than
> > try to find and fix the race condition.
> >
> > John?
> >
> > --
> > MST
>
>
> I think I see the source of the race. virtio net calls
> netif_device_detach and assumes no packets will be sent after
> this point. However, all it does is stop all queues so
> no new packets will be transmitted.
>
> Try locking with HARD_TX_LOCK?
>
>
> --
> MST
>
Hi Michael,
from what i see, the race appear when we hit virtnet_reset in
virtnet_xdp_set.
virtnet_reset
_remove_vq_common
virtnet_del_vqs
virtnet_free_queues
kfree(vi->sq)
when the xdp program (with two instances of the program to trigger it
faster) is added or removed.
It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
running the xdp_ttl tool from Jesper.
For now, i'm able to continue my qualification, testing if xdp_qp is not
null, but do not seem to be a sustainable trick.
if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
Maybe it will be more clear to you with theses informations.
Best regards.
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-22 6:15 ` [Qemu-devel] " jean-philippe menil
@ 2017-06-22 18:53 ` Michael S. Tsirkin
-1 siblings, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-22 18:53 UTC (permalink / raw)
To: jean-philippe menil
Cc: netdev, jasowang, John Fastabend, virtualization, qemu-devel
On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>
> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > > Hi,
> > >
> > > while playing with xdp and ebpf, i'm hitting the following:
> > >
> > > [ 309.993136]
> > > ==================================================================
> > > [ 309.994735] BUG: KASAN: use-after-free in
> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > > [ 310.000650]
> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
...
> >
> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
> > virtio_net: rework mergeable buffer handling
> >
> > we no longer must do the resets, we now have enough space
> > to store a bit saying whether a buffer is xdp one or not.
> >
> > And that's probably a cleaner way to fix these issues than
> > try to find and fix the race condition.
> >
> > John?
> >
> > --
> > MST
>
>
> I think I see the source of the race. virtio net calls
> netif_device_detach and assumes no packets will be sent after
> this point. However, all it does is stop all queues so
> no new packets will be transmitted.
>
> Try locking with HARD_TX_LOCK?
>
>
> --
> MST
>
>
> Hi Michael,
>
> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set.
> virtnet_reset
> _remove_vq_common
> virtnet_del_vqs
> virtnet_free_queues
> kfree(vi->sq)
> when the xdp program (with two instances of the program to trigger it faster)
> is added or removed.
>
> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
> running the xdp_ttl tool from Jesper.
>
> For now, i'm able to continue my qualification, testing if xdp_qp is not null,
> but do not seem to be a sustainable trick.
> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>
> Maybe it will be more clear to you with theses informations.
>
> Best regards.
>
> Jean-Philippe
I'm pretty clear about the issue here, I was trying to figure out a fix.
Jason, any thoughts?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-22 18:53 ` Michael S. Tsirkin
0 siblings, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-22 18:53 UTC (permalink / raw)
To: jean-philippe menil
Cc: netdev, jasowang, John Fastabend, virtualization, qemu-devel
On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>
> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > > Hi,
> > >
> > > while playing with xdp and ebpf, i'm hitting the following:
> > >
> > > [ 309.993136]
> > > ==================================================================
> > > [ 309.994735] BUG: KASAN: use-after-free in
> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > > [ 310.000650]
> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
...
> >
> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
> > virtio_net: rework mergeable buffer handling
> >
> > we no longer must do the resets, we now have enough space
> > to store a bit saying whether a buffer is xdp one or not.
> >
> > And that's probably a cleaner way to fix these issues than
> > try to find and fix the race condition.
> >
> > John?
> >
> > --
> > MST
>
>
> I think I see the source of the race. virtio net calls
> netif_device_detach and assumes no packets will be sent after
> this point. However, all it does is stop all queues so
> no new packets will be transmitted.
>
> Try locking with HARD_TX_LOCK?
>
>
> --
> MST
>
>
> Hi Michael,
>
> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set.
> virtnet_reset
> _remove_vq_common
> virtnet_del_vqs
> virtnet_free_queues
> kfree(vi->sq)
> when the xdp program (with two instances of the program to trigger it faster)
> is added or removed.
>
> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
> running the xdp_ttl tool from Jesper.
>
> For now, i'm able to continue my qualification, testing if xdp_qp is not null,
> but do not seem to be a sustainable trick.
> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>
> Maybe it will be more clear to you with theses informations.
>
> Best regards.
>
> Jean-Philippe
I'm pretty clear about the issue here, I was trying to figure out a fix.
Jason, any thoughts?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-22 6:15 ` [Qemu-devel] " jean-philippe menil
(?)
@ 2017-06-22 18:53 ` Michael S. Tsirkin
-1 siblings, 0 replies; 27+ messages in thread
From: Michael S. Tsirkin @ 2017-06-22 18:53 UTC (permalink / raw)
To: jean-philippe menil; +Cc: netdev, John Fastabend, qemu-devel, virtualization
On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>
> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
> > > Hi,
> > >
> > > while playing with xdp and ebpf, i'm hitting the following:
> > >
> > > [ 309.993136]
> > > ==================================================================
> > > [ 309.994735] BUG: KASAN: use-after-free in
> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
> > > [ 310.000650]
> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS
> > > 1.10.2-20170228_101828-anatol 04/01/2014
...
> >
> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
> > virtio_net: rework mergeable buffer handling
> >
> > we no longer must do the resets, we now have enough space
> > to store a bit saying whether a buffer is xdp one or not.
> >
> > And that's probably a cleaner way to fix these issues than
> > try to find and fix the race condition.
> >
> > John?
> >
> > --
> > MST
>
>
> I think I see the source of the race. virtio net calls
> netif_device_detach and assumes no packets will be sent after
> this point. However, all it does is stop all queues so
> no new packets will be transmitted.
>
> Try locking with HARD_TX_LOCK?
>
>
> --
> MST
>
>
> Hi Michael,
>
> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set.
> virtnet_reset
> _remove_vq_common
> virtnet_del_vqs
> virtnet_free_queues
> kfree(vi->sq)
> when the xdp program (with two instances of the program to trigger it faster)
> is added or removed.
>
> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
> running the xdp_ttl tool from Jesper.
>
> For now, i'm able to continue my qualification, testing if xdp_qp is not null,
> but do not seem to be a sustainable trick.
> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>
> Maybe it will be more clear to you with theses informations.
>
> Best regards.
>
> Jean-Philippe
I'm pretty clear about the issue here, I was trying to figure out a fix.
Jason, any thoughts?
--
MST
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-22 18:53 ` [Qemu-devel] " Michael S. Tsirkin
@ 2017-06-23 8:43 ` Jason Wang
-1 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-23 8:43 UTC (permalink / raw)
To: Michael S. Tsirkin, jean-philippe menil
Cc: netdev, John Fastabend, qemu-devel, virtualization
On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>>
>> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
>> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
>> > > Hi,
>> > >
>> > > while playing with xdp and ebpf, i'm hitting the following:
>> > >
>> > > [ 309.993136]
>> > > ==================================================================
>> > > [ 309.994735] BUG: KASAN: use-after-free in
>> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
>> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
>> > > [ 310.000650]
>> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
>> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS
>> > > 1.10.2-20170228_101828-anatol 04/01/2014
> ...
>
>> >
>> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
>> > virtio_net: rework mergeable buffer handling
>> >
>> > we no longer must do the resets, we now have enough space
>> > to store a bit saying whether a buffer is xdp one or not.
>> >
>> > And that's probably a cleaner way to fix these issues than
>> > try to find and fix the race condition.
>> >
>> > John?
>> >
>> > --
>> > MST
>>
>>
>> I think I see the source of the race. virtio net calls
>> netif_device_detach and assumes no packets will be sent after
>> this point. However, all it does is stop all queues so
>> no new packets will be transmitted.
>>
>> Try locking with HARD_TX_LOCK?
>>
>>
>> --
>> MST
>>
>>
>> Hi Michael,
>>
>> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set.
>> virtnet_reset
>> _remove_vq_common
>> virtnet_del_vqs
>> virtnet_free_queues
>> kfree(vi->sq)
>> when the xdp program (with two instances of the program to trigger it faster)
>> is added or removed.
>>
>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
>> running the xdp_ttl tool from Jesper.
>>
>> For now, i'm able to continue my qualification, testing if xdp_qp is not null,
>> but do not seem to be a sustainable trick.
>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>
>> Maybe it will be more clear to you with theses informations.
>>
>> Best regards.
>>
>> Jean-Philippe
>
> I'm pretty clear about the issue here, I was trying to figure out a fix.
> Jason, any thoughts?
>
>
Hi Jean:
Does the following fix this issue? (I can't reproduce it locally through
xdp_ttl)
Thanks
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 1f8c15c..3e65c3f 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct
virtio_device *vdev)
/* Make sure no work handler is accessing the device */
flush_work(&vi->config_work);
+ netif_tx_lock_bh(vi->dev);
netif_device_detach(vi->dev);
+ netif_tx_unlock_bh(vi->dev);
cancel_delayed_work_sync(&vi->refill);
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-23 8:43 ` Jason Wang
0 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-23 8:43 UTC (permalink / raw)
To: Michael S. Tsirkin, jean-philippe menil
Cc: netdev, John Fastabend, virtualization, qemu-devel
On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>>
>> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
>> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote:
>> > > Hi,
>> > >
>> > > while playing with xdp and ebpf, i'm hitting the following:
>> > >
>> > > [ 309.993136]
>> > > ==================================================================
>> > > [ 309.994735] BUG: KASAN: use-after-free in
>> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
>> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323
>> > > [ 310.000650]
>> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2
>> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS
>> > > 1.10.2-20170228_101828-anatol 04/01/2014
> ...
>
>> >
>> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
>> > virtio_net: rework mergeable buffer handling
>> >
>> > we no longer must do the resets, we now have enough space
>> > to store a bit saying whether a buffer is xdp one or not.
>> >
>> > And that's probably a cleaner way to fix these issues than
>> > try to find and fix the race condition.
>> >
>> > John?
>> >
>> > --
>> > MST
>>
>>
>> I think I see the source of the race. virtio net calls
>> netif_device_detach and assumes no packets will be sent after
>> this point. However, all it does is stop all queues so
>> no new packets will be transmitted.
>>
>> Try locking with HARD_TX_LOCK?
>>
>>
>> --
>> MST
>>
>>
>> Hi Michael,
>>
>> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set.
>> virtnet_reset
>> _remove_vq_common
>> virtnet_del_vqs
>> virtnet_free_queues
>> kfree(vi->sq)
>> when the xdp program (with two instances of the program to trigger it faster)
>> is added or removed.
>>
>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line,
>> running the xdp_ttl tool from Jesper.
>>
>> For now, i'm able to continue my qualification, testing if xdp_qp is not null,
>> but do not seem to be a sustainable trick.
>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>
>> Maybe it will be more clear to you with theses informations.
>>
>> Best regards.
>>
>> Jean-Philippe
>
> I'm pretty clear about the issue here, I was trying to figure out a fix.
> Jason, any thoughts?
>
>
Hi Jean:
Does the following fix this issue? (I can't reproduce it locally through
xdp_ttl)
Thanks
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 1f8c15c..3e65c3f 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct
virtio_device *vdev)
/* Make sure no work handler is accessing the device */
flush_work(&vi->config_work);
+ netif_tx_lock_bh(vi->dev);
netif_device_detach(vi->dev);
+ netif_tx_unlock_bh(vi->dev);
cancel_delayed_work_sync(&vi->refill);
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-23 8:43 ` [Qemu-devel] " Jason Wang
@ 2017-06-23 9:33 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-23 9:33 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, netdev, John Fastabend, virtualization, qemu-devel
On 06/23/2017 10:43 AM, Jason Wang wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>>>
>>> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
>>> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil
>>> wrote:
>>> > > Hi,
>>> > >
>>> > > while playing with xdp and ebpf, i'm hitting the following:
>>> > >
>>> > > [ 309.993136]
>>> > >
>>> ==================================================================
>>> > > [ 309.994735] BUG: KASAN: use-after-free in
>>> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
>>> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by
>>> task sshd/323
>>> > > [ 310.000650]
>>> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted
>>> 4.12.0-rc3+ #2
>>> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX +
>>> PIIX, 1996),
>>> BIOS
>>> > > 1.10.2-20170228_101828-anatol 04/01/2014
>> ...
>>
>>> >
>>> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
>>> > virtio_net: rework mergeable buffer handling
>>> >
>>> > we no longer must do the resets, we now have enough space
>>> > to store a bit saying whether a buffer is xdp one or not.
>>> >
>>> > And that's probably a cleaner way to fix these issues than
>>> > try to find and fix the race condition.
>>> >
>>> > John?
>>> >
>>> > --
>>> > MST
>>>
>>>
>>> I think I see the source of the race. virtio net calls
>>> netif_device_detach and assumes no packets will be sent after
>>> this point. However, all it does is stop all queues so
>>> no new packets will be transmitted.
>>>
>>> Try locking with HARD_TX_LOCK?
>>>
>>> --
>>> MST
>>>
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is
>>> not null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
>
> Thanks
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1f8c15c..3e65c3f 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct
> virtio_device *vdev)
> /* Make sure no work handler is accessing the device */
> flush_work(&vi->config_work);
>
> + netif_tx_lock_bh(vi->dev);
> netif_device_detach(vi->dev);
> + netif_tx_unlock_bh(vi->dev);
> cancel_delayed_work_sync(&vi->refill);
>
Hi Jason,
unfortunately, same crash on same place, the lock did not help.
[ 574.522886]
==================================================================
[ 574.527393] BUG: KASAN: use-after-free in
free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.531934] Read of size 8 at addr ffff88005d220020 by task iperf/2252
[ 574.536296]
[ 574.539729] CPU: 1 PID: 2252 Comm: iperf Not tainted 4.12.0-rc5+ #5
[ 574.543916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[ 574.552046] Call Trace:
[ 574.555648] dump_stack+0xb3/0x10b
[ 574.559471] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.563578] print_address_description+0x6a/0x280
[ 574.567253] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.571223] kasan_report+0x22b/0x340
[ 574.574698] __asan_report_load8_noabort+0x14/0x20
[ 574.578490] free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.582586] ? dev_queue_xmit_nit+0x5fb/0x850
[ 574.586348] ? virtnet_del_vqs+0xf0/0xf0 [virtio_net]
[ 574.590153] ? __skb_clone+0x24a/0x7d0
[ 574.593835] start_xmit+0x15a/0x1620 [virtio_net]
[ 574.597939] dev_hard_start_xmit+0x17f/0x7e0
[ 574.601832] sch_direct_xmit+0x2a8/0x5d0
[ 574.605665] ? dev_deactivate_queue.constprop.31+0x150/0x150
[ 574.609827] __dev_queue_xmit+0x1124/0x18b0
[ 574.613595] ? selinux_ip_postroute+0x4b2/0xa90
[ 574.617928] ? netdev_pick_tx+0x2d0/0x2d0
[ 574.621852] ? mark_held_locks+0xc8/0x120
[ 574.625673] ? ip_finish_output+0x626/0x9b0
[ 574.631679] ? ip_finish_output2+0xb44/0x1160
[ 574.637642] dev_queue_xmit+0x17/0x20
[ 574.641693] ip_finish_output2+0xcd1/0x1160
[ 574.645621] ? do_add_counters+0x480/0x480
[ 574.649554] ? do_add_counters+0x403/0x480
[ 574.653209] ? ip_copy_metadata+0x630/0x630
[ 574.657066] ip_finish_output+0x626/0x9b0
[ 574.660482] ? ip_finish_output+0x626/0x9b0
[ 574.663905] ip_output+0x1e2/0x580
[ 574.667235] ? ip_mc_output+0xe80/0xe80
[ 574.670574] ? ip_fragment.constprop.57+0x200/0x200
[ 574.673949] ip_local_out+0x95/0x160
[ 574.677249] ? __sk_dst_check+0xa7/0x260
[ 574.680446] ip_queue_xmit+0x889/0x17f0
[ 574.683575] ? __tcp_v4_send_check+0x1b8/0x350
[ 574.686801] tcp_transmit_skb+0x194a/0x2db0
[ 574.689832] ? __tcp_select_window+0x500/0x500
[ 574.693310] ? sched_clock_cpu+0x1b/0x190
[ 574.696371] ? tcp_grow_window.isra.24+0x2a8/0x4b0
[ 574.699509] tcp_send_ack+0x46f/0x710
[ 574.702395] __tcp_ack_snd_check+0x233/0x380
[ 574.705365] tcp_rcv_established+0x14eb/0x2230
[ 574.708332] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.711800] ? tcp_data_queue+0x3e70/0x3e70
[ 574.714761] ? sk_wait_data+0x2af/0x400
[ 574.719220] tcp_v4_do_rcv+0x56c/0x820
[ 574.724018] tcp_prequeue_process+0x18f/0x2c0
[ 574.729062] tcp_recvmsg+0xff6/0x26a0
[ 574.734615] ? tcp_tx_timestamp.part.27+0x290/0x290
[ 574.739519] ? _copy_from_user+0x84/0xe0
[ 574.744115] ? rw_copy_check_uvector+0x1f6/0x290
[ 574.748722] ? sock_has_perm+0x1e4/0x270
[ 574.751537] ? selinux_tun_dev_create+0xc0/0xc0
[ 574.754068] inet_recvmsg+0x117/0x530
[ 574.756823] ? memzero_page+0x130/0x130
[ 574.759503] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.762303] ? selinux_socket_recvmsg+0x36/0x40
[ 574.765114] ? security_socket_recvmsg+0x8f/0xc0
[ 574.768156] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.771181] sock_recvmsg+0xd7/0x110
[ 574.773972] ? __sock_recv_wifi_status+0x180/0x180
[ 574.777002] ___sys_recvmsg+0x24d/0x560
[ 574.779789] ? ___sys_sendmsg+0x920/0x920
[ 574.782734] ? __fget+0x200/0x380
[ 574.785657] ? lock_downgrade+0x650/0x650
[ 574.788584] ? __fget+0x229/0x380
[ 574.791362] ? __fget_light+0xa1/0x1f0
[ 574.794162] ? __fdget+0x18/0x20
[ 574.796832] __sys_recvmsg+0xce/0x170
[ 574.799572] ? __sys_recvmsg+0xce/0x170
[ 574.802695] ? SyS_sendmmsg+0x60/0x60
[ 574.805461] ? __schedule+0x7cb/0x1a70
[ 574.808211] ? retint_kernel+0x10/0x10
[ 574.810922] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.813890] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 574.816783] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.819693] SyS_recvmsg+0x2d/0x50
[ 574.822829] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.825697] RIP: 0033:0x7f7fbd77e3c0
[ 574.828366] RSP: 002b:00007f7fba39ed50 EFLAGS: 00000293 ORIG_RAX:
000000000000002f
[ 574.833588] RAX: ffffffffffffffda RBX: 0000000000000008 RCX:
00007f7fbd77e3c0
[ 574.838882] RDX: 0000000000000000 RSI: 00007f7fba39edb0 RDI:
0000000000000008
[ 574.844377] RBP: 0000000000000046 R08: 0000000000000000 R09:
000000a0ff7159c1
[ 574.849937] R10: 00143f7b62d9620b R11: 0000000000000293 R12:
0000000000000000
[ 574.855391] R13: 0000000004000000 R14: 00007f7fa0000b10 R15:
0000000000000001
[ 574.860146]
[ 574.862738] Allocated by task 2291:
[ 574.865528] save_stack_trace+0x16/0x20
[ 574.868370] save_stack+0x46/0xd0
[ 574.871096] kasan_kmalloc+0xad/0xe0
[ 574.873838] __kmalloc+0x115/0x2d0
[ 574.876524] __vring_new_virtqueue+0x6a/0x790
[ 574.879432] vring_create_virtqueue+0x203/0x380
[ 574.882367] setup_vq+0x159/0x660
[ 574.885115] vp_setup_vq+0xbe/0x390
[ 574.887802] vp_find_vqs_msix+0x568/0xb90
[ 574.890494] vp_find_vqs+0x93/0x460
[ 574.893175] vp_modern_find_vqs+0x44/0x170
[ 574.895932] init_vqs+0x8eb/0x1150 [virtio_net]
[ 574.898778] virtnet_restore_up+0x4c/0x5c0 [virtio_net]
[ 574.901889] virtnet_xdp+0x820/0xd00 [virtio_net]
[ 574.904858] dev_change_xdp_fd+0x1bb/0x340
[ 574.907708] do_setlink+0x23fb/0x2c00
[ 574.910491] rtnl_setlink+0x280/0x340
[ 574.913448] rtnetlink_rcv_msg+0x288/0x680
[ 574.916348] netlink_rcv_skb+0x340/0x470
[ 574.919165] rtnetlink_rcv+0x2a/0x40
[ 574.922027] netlink_unicast+0x58d/0x860
[ 574.924897] netlink_sendmsg+0x8d2/0xca0
[ 574.927815] sock_sendmsg+0xca/0x110
[ 574.930708] SYSC_sendto+0x20d/0x340
[ 574.933562] SyS_sendto+0x40/0x50
[ 574.936380] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.939322]
[ 574.941724] Freed by task 2291:
[ 574.944389] save_stack_trace+0x16/0x20
[ 574.947107] save_stack+0x46/0xd0
[ 574.949893] kasan_slab_free+0x72/0xc0
[ 574.952526] kfree+0xe6/0x2c0
[ 574.955082] vring_del_virtqueue+0xef/0x220
[ 574.957773] del_vq+0x126/0x270
[ 574.960283] vp_del_vqs+0x1f5/0xa30
[ 574.962743] virtnet_del_vqs+0xb7/0xf0 [virtio_net]
[ 574.965930] virtnet_xdp+0x7b8/0xd00 [virtio_net]
[ 574.968762] dev_change_xdp_fd+0x309/0x340
[ 574.971487] do_setlink+0x23fb/0x2c00
[ 574.974041] rtnl_setlink+0x280/0x340
[ 574.976727] rtnetlink_rcv_msg+0x288/0x680
[ 574.979366] netlink_rcv_skb+0x340/0x470
[ 574.981949] rtnetlink_rcv+0x2a/0x40
[ 574.984462] netlink_unicast+0x58d/0x860
[ 574.987151] netlink_sendmsg+0x8d2/0xca0
[ 574.989736] sock_sendmsg+0xca/0x110
[ 574.992351] SYSC_sendto+0x20d/0x340
[ 574.995262] SyS_sendto+0x40/0x50
[ 574.998959] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 575.001625]
[ 575.003976] The buggy address belongs to the object at ffff88005d220000
[ 575.003976] which belongs to the cache kmalloc-8192 of size 8192
[ 575.010183] The buggy address is located 32 bytes inside of
[ 575.010183] 8192-byte region [ffff88005d220000, ffff88005d222000)
[ 575.016265] The buggy address belongs to the page:
[ 575.019125] page:ffffea0001748800 count:1 mapcount:0 mapping:
(null) index:0x0 compound_mapcount: 0
[ 575.025320] flags: 0x100000000008100(slab|head)
[ 575.028167] raw: 0100000000008100 0000000000000000 0000000000000000
0000000100030003
[ 575.031632] raw: dead000000000100 dead000000000200 ffff88006c802280
0000000000000000
[ 575.035447] page dumped because: kasan: bad access detected
[ 575.039170]
[ 575.041893] Memory state around the buggy address:
[ 575.045408] ffff88005d21ff00: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.051399] ffff88005d21ff80: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.057558] >ffff88005d220000: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.063760] ^
[ 575.069310] ffff88005d220080: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.075359] ffff88005d220100: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.081461]
==================================================================
[ 575.086914] Disabling lock debugging due to kernel taint
[ 575.090717] virtio_net virtio1: output.0:id 31 is not a head!
[ 575.096336] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.102000] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.107383] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.112785] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.118228] net enp0s4: Unexpected TXQ (0) queue failure: -5
(gdb) l *(free_old_xmit_skbs+0x29b)
0x20db is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
1046
1047 static void free_old_xmit_skbs(struct send_queue *sq)
1048 {
1049 struct sk_buff *skb;
1050 unsigned int len;
1051 struct virtnet_info *vi = sq->vq->vdev->priv;
1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
1053 unsigned int packets = 0;
1054 unsigned int bytes = 0;
1055
Best regards,
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-23 9:33 ` Jean-Philippe Menil
0 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-23 9:33 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, netdev, John Fastabend, virtualization, qemu-devel
On 06/23/2017 10:43 AM, Jason Wang wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>>>
>>> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
>>> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil
>>> wrote:
>>> > > Hi,
>>> > >
>>> > > while playing with xdp and ebpf, i'm hitting the following:
>>> > >
>>> > > [ 309.993136]
>>> > >
>>> ==================================================================
>>> > > [ 309.994735] BUG: KASAN: use-after-free in
>>> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
>>> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by
>>> task sshd/323
>>> > > [ 310.000650]
>>> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted
>>> 4.12.0-rc3+ #2
>>> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX +
>>> PIIX, 1996),
>>> BIOS
>>> > > 1.10.2-20170228_101828-anatol 04/01/2014
>> ...
>>
>>> >
>>> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
>>> > virtio_net: rework mergeable buffer handling
>>> >
>>> > we no longer must do the resets, we now have enough space
>>> > to store a bit saying whether a buffer is xdp one or not.
>>> >
>>> > And that's probably a cleaner way to fix these issues than
>>> > try to find and fix the race condition.
>>> >
>>> > John?
>>> >
>>> > --
>>> > MST
>>>
>>>
>>> I think I see the source of the race. virtio net calls
>>> netif_device_detach and assumes no packets will be sent after
>>> this point. However, all it does is stop all queues so
>>> no new packets will be transmitted.
>>>
>>> Try locking with HARD_TX_LOCK?
>>>
>>> --
>>> MST
>>>
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is
>>> not null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
>
> Thanks
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1f8c15c..3e65c3f 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct
> virtio_device *vdev)
> /* Make sure no work handler is accessing the device */
> flush_work(&vi->config_work);
>
> + netif_tx_lock_bh(vi->dev);
> netif_device_detach(vi->dev);
> + netif_tx_unlock_bh(vi->dev);
> cancel_delayed_work_sync(&vi->refill);
>
Hi Jason,
unfortunately, same crash on same place, the lock did not help.
[ 574.522886]
==================================================================
[ 574.527393] BUG: KASAN: use-after-free in
free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.531934] Read of size 8 at addr ffff88005d220020 by task iperf/2252
[ 574.536296]
[ 574.539729] CPU: 1 PID: 2252 Comm: iperf Not tainted 4.12.0-rc5+ #5
[ 574.543916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[ 574.552046] Call Trace:
[ 574.555648] dump_stack+0xb3/0x10b
[ 574.559471] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.563578] print_address_description+0x6a/0x280
[ 574.567253] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.571223] kasan_report+0x22b/0x340
[ 574.574698] __asan_report_load8_noabort+0x14/0x20
[ 574.578490] free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.582586] ? dev_queue_xmit_nit+0x5fb/0x850
[ 574.586348] ? virtnet_del_vqs+0xf0/0xf0 [virtio_net]
[ 574.590153] ? __skb_clone+0x24a/0x7d0
[ 574.593835] start_xmit+0x15a/0x1620 [virtio_net]
[ 574.597939] dev_hard_start_xmit+0x17f/0x7e0
[ 574.601832] sch_direct_xmit+0x2a8/0x5d0
[ 574.605665] ? dev_deactivate_queue.constprop.31+0x150/0x150
[ 574.609827] __dev_queue_xmit+0x1124/0x18b0
[ 574.613595] ? selinux_ip_postroute+0x4b2/0xa90
[ 574.617928] ? netdev_pick_tx+0x2d0/0x2d0
[ 574.621852] ? mark_held_locks+0xc8/0x120
[ 574.625673] ? ip_finish_output+0x626/0x9b0
[ 574.631679] ? ip_finish_output2+0xb44/0x1160
[ 574.637642] dev_queue_xmit+0x17/0x20
[ 574.641693] ip_finish_output2+0xcd1/0x1160
[ 574.645621] ? do_add_counters+0x480/0x480
[ 574.649554] ? do_add_counters+0x403/0x480
[ 574.653209] ? ip_copy_metadata+0x630/0x630
[ 574.657066] ip_finish_output+0x626/0x9b0
[ 574.660482] ? ip_finish_output+0x626/0x9b0
[ 574.663905] ip_output+0x1e2/0x580
[ 574.667235] ? ip_mc_output+0xe80/0xe80
[ 574.670574] ? ip_fragment.constprop.57+0x200/0x200
[ 574.673949] ip_local_out+0x95/0x160
[ 574.677249] ? __sk_dst_check+0xa7/0x260
[ 574.680446] ip_queue_xmit+0x889/0x17f0
[ 574.683575] ? __tcp_v4_send_check+0x1b8/0x350
[ 574.686801] tcp_transmit_skb+0x194a/0x2db0
[ 574.689832] ? __tcp_select_window+0x500/0x500
[ 574.693310] ? sched_clock_cpu+0x1b/0x190
[ 574.696371] ? tcp_grow_window.isra.24+0x2a8/0x4b0
[ 574.699509] tcp_send_ack+0x46f/0x710
[ 574.702395] __tcp_ack_snd_check+0x233/0x380
[ 574.705365] tcp_rcv_established+0x14eb/0x2230
[ 574.708332] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.711800] ? tcp_data_queue+0x3e70/0x3e70
[ 574.714761] ? sk_wait_data+0x2af/0x400
[ 574.719220] tcp_v4_do_rcv+0x56c/0x820
[ 574.724018] tcp_prequeue_process+0x18f/0x2c0
[ 574.729062] tcp_recvmsg+0xff6/0x26a0
[ 574.734615] ? tcp_tx_timestamp.part.27+0x290/0x290
[ 574.739519] ? _copy_from_user+0x84/0xe0
[ 574.744115] ? rw_copy_check_uvector+0x1f6/0x290
[ 574.748722] ? sock_has_perm+0x1e4/0x270
[ 574.751537] ? selinux_tun_dev_create+0xc0/0xc0
[ 574.754068] inet_recvmsg+0x117/0x530
[ 574.756823] ? memzero_page+0x130/0x130
[ 574.759503] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.762303] ? selinux_socket_recvmsg+0x36/0x40
[ 574.765114] ? security_socket_recvmsg+0x8f/0xc0
[ 574.768156] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.771181] sock_recvmsg+0xd7/0x110
[ 574.773972] ? __sock_recv_wifi_status+0x180/0x180
[ 574.777002] ___sys_recvmsg+0x24d/0x560
[ 574.779789] ? ___sys_sendmsg+0x920/0x920
[ 574.782734] ? __fget+0x200/0x380
[ 574.785657] ? lock_downgrade+0x650/0x650
[ 574.788584] ? __fget+0x229/0x380
[ 574.791362] ? __fget_light+0xa1/0x1f0
[ 574.794162] ? __fdget+0x18/0x20
[ 574.796832] __sys_recvmsg+0xce/0x170
[ 574.799572] ? __sys_recvmsg+0xce/0x170
[ 574.802695] ? SyS_sendmmsg+0x60/0x60
[ 574.805461] ? __schedule+0x7cb/0x1a70
[ 574.808211] ? retint_kernel+0x10/0x10
[ 574.810922] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.813890] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 574.816783] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.819693] SyS_recvmsg+0x2d/0x50
[ 574.822829] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.825697] RIP: 0033:0x7f7fbd77e3c0
[ 574.828366] RSP: 002b:00007f7fba39ed50 EFLAGS: 00000293 ORIG_RAX:
000000000000002f
[ 574.833588] RAX: ffffffffffffffda RBX: 0000000000000008 RCX:
00007f7fbd77e3c0
[ 574.838882] RDX: 0000000000000000 RSI: 00007f7fba39edb0 RDI:
0000000000000008
[ 574.844377] RBP: 0000000000000046 R08: 0000000000000000 R09:
000000a0ff7159c1
[ 574.849937] R10: 00143f7b62d9620b R11: 0000000000000293 R12:
0000000000000000
[ 574.855391] R13: 0000000004000000 R14: 00007f7fa0000b10 R15:
0000000000000001
[ 574.860146]
[ 574.862738] Allocated by task 2291:
[ 574.865528] save_stack_trace+0x16/0x20
[ 574.868370] save_stack+0x46/0xd0
[ 574.871096] kasan_kmalloc+0xad/0xe0
[ 574.873838] __kmalloc+0x115/0x2d0
[ 574.876524] __vring_new_virtqueue+0x6a/0x790
[ 574.879432] vring_create_virtqueue+0x203/0x380
[ 574.882367] setup_vq+0x159/0x660
[ 574.885115] vp_setup_vq+0xbe/0x390
[ 574.887802] vp_find_vqs_msix+0x568/0xb90
[ 574.890494] vp_find_vqs+0x93/0x460
[ 574.893175] vp_modern_find_vqs+0x44/0x170
[ 574.895932] init_vqs+0x8eb/0x1150 [virtio_net]
[ 574.898778] virtnet_restore_up+0x4c/0x5c0 [virtio_net]
[ 574.901889] virtnet_xdp+0x820/0xd00 [virtio_net]
[ 574.904858] dev_change_xdp_fd+0x1bb/0x340
[ 574.907708] do_setlink+0x23fb/0x2c00
[ 574.910491] rtnl_setlink+0x280/0x340
[ 574.913448] rtnetlink_rcv_msg+0x288/0x680
[ 574.916348] netlink_rcv_skb+0x340/0x470
[ 574.919165] rtnetlink_rcv+0x2a/0x40
[ 574.922027] netlink_unicast+0x58d/0x860
[ 574.924897] netlink_sendmsg+0x8d2/0xca0
[ 574.927815] sock_sendmsg+0xca/0x110
[ 574.930708] SYSC_sendto+0x20d/0x340
[ 574.933562] SyS_sendto+0x40/0x50
[ 574.936380] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.939322]
[ 574.941724] Freed by task 2291:
[ 574.944389] save_stack_trace+0x16/0x20
[ 574.947107] save_stack+0x46/0xd0
[ 574.949893] kasan_slab_free+0x72/0xc0
[ 574.952526] kfree+0xe6/0x2c0
[ 574.955082] vring_del_virtqueue+0xef/0x220
[ 574.957773] del_vq+0x126/0x270
[ 574.960283] vp_del_vqs+0x1f5/0xa30
[ 574.962743] virtnet_del_vqs+0xb7/0xf0 [virtio_net]
[ 574.965930] virtnet_xdp+0x7b8/0xd00 [virtio_net]
[ 574.968762] dev_change_xdp_fd+0x309/0x340
[ 574.971487] do_setlink+0x23fb/0x2c00
[ 574.974041] rtnl_setlink+0x280/0x340
[ 574.976727] rtnetlink_rcv_msg+0x288/0x680
[ 574.979366] netlink_rcv_skb+0x340/0x470
[ 574.981949] rtnetlink_rcv+0x2a/0x40
[ 574.984462] netlink_unicast+0x58d/0x860
[ 574.987151] netlink_sendmsg+0x8d2/0xca0
[ 574.989736] sock_sendmsg+0xca/0x110
[ 574.992351] SYSC_sendto+0x20d/0x340
[ 574.995262] SyS_sendto+0x40/0x50
[ 574.998959] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 575.001625]
[ 575.003976] The buggy address belongs to the object at ffff88005d220000
[ 575.003976] which belongs to the cache kmalloc-8192 of size 8192
[ 575.010183] The buggy address is located 32 bytes inside of
[ 575.010183] 8192-byte region [ffff88005d220000, ffff88005d222000)
[ 575.016265] The buggy address belongs to the page:
[ 575.019125] page:ffffea0001748800 count:1 mapcount:0 mapping:
(null) index:0x0 compound_mapcount: 0
[ 575.025320] flags: 0x100000000008100(slab|head)
[ 575.028167] raw: 0100000000008100 0000000000000000 0000000000000000
0000000100030003
[ 575.031632] raw: dead000000000100 dead000000000200 ffff88006c802280
0000000000000000
[ 575.035447] page dumped because: kasan: bad access detected
[ 575.039170]
[ 575.041893] Memory state around the buggy address:
[ 575.045408] ffff88005d21ff00: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.051399] ffff88005d21ff80: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.057558] >ffff88005d220000: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.063760] ^
[ 575.069310] ffff88005d220080: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.075359] ffff88005d220100: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.081461]
==================================================================
[ 575.086914] Disabling lock debugging due to kernel taint
[ 575.090717] virtio_net virtio1: output.0:id 31 is not a head!
[ 575.096336] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.102000] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.107383] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.112785] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.118228] net enp0s4: Unexpected TXQ (0) queue failure: -5
(gdb) l *(free_old_xmit_skbs+0x29b)
0x20db is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
1046
1047 static void free_old_xmit_skbs(struct send_queue *sq)
1048 {
1049 struct sk_buff *skb;
1050 unsigned int len;
1051 struct virtnet_info *vi = sq->vq->vdev->priv;
1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
1053 unsigned int packets = 0;
1054 unsigned int bytes = 0;
1055
Best regards,
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-23 8:43 ` [Qemu-devel] " Jason Wang
(?)
(?)
@ 2017-06-23 9:33 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-23 9:33 UTC (permalink / raw)
To: Jason Wang
Cc: netdev, virtualization, John Fastabend, qemu-devel, Michael S. Tsirkin
On 06/23/2017 10:43 AM, Jason Wang wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin <mst@redhat.com>:
>>>
>>> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote:
>>> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil
>>> wrote:
>>> > > Hi,
>>> > >
>>> > > while playing with xdp and ebpf, i'm hitting the following:
>>> > >
>>> > > [ 309.993136]
>>> > >
>>> ==================================================================
>>> > > [ 309.994735] BUG: KASAN: use-after-free in
>>> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net]
>>> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by
>>> task sshd/323
>>> > > [ 310.000650]
>>> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted
>>> 4.12.0-rc3+ #2
>>> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX +
>>> PIIX, 1996),
>>> BIOS
>>> > > 1.10.2-20170228_101828-anatol 04/01/2014
>> ...
>>
>>> >
>>> > Since commit 680557cf79f82623e2c4fd42733077d60a843513
>>> > virtio_net: rework mergeable buffer handling
>>> >
>>> > we no longer must do the resets, we now have enough space
>>> > to store a bit saying whether a buffer is xdp one or not.
>>> >
>>> > And that's probably a cleaner way to fix these issues than
>>> > try to find and fix the race condition.
>>> >
>>> > John?
>>> >
>>> > --
>>> > MST
>>>
>>>
>>> I think I see the source of the race. virtio net calls
>>> netif_device_detach and assumes no packets will be sent after
>>> this point. However, all it does is stop all queues so
>>> no new packets will be transmitted.
>>>
>>> Try locking with HARD_TX_LOCK?
>>>
>>> --
>>> MST
>>>
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is
>>> not null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
>
> Thanks
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1f8c15c..3e65c3f 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct
> virtio_device *vdev)
> /* Make sure no work handler is accessing the device */
> flush_work(&vi->config_work);
>
> + netif_tx_lock_bh(vi->dev);
> netif_device_detach(vi->dev);
> + netif_tx_unlock_bh(vi->dev);
> cancel_delayed_work_sync(&vi->refill);
>
Hi Jason,
unfortunately, same crash on same place, the lock did not help.
[ 574.522886]
==================================================================
[ 574.527393] BUG: KASAN: use-after-free in
free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.531934] Read of size 8 at addr ffff88005d220020 by task iperf/2252
[ 574.536296]
[ 574.539729] CPU: 1 PID: 2252 Comm: iperf Not tainted 4.12.0-rc5+ #5
[ 574.543916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[ 574.552046] Call Trace:
[ 574.555648] dump_stack+0xb3/0x10b
[ 574.559471] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.563578] print_address_description+0x6a/0x280
[ 574.567253] ? free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.571223] kasan_report+0x22b/0x340
[ 574.574698] __asan_report_load8_noabort+0x14/0x20
[ 574.578490] free_old_xmit_skbs.isra.28+0x29b/0x2b0 [virtio_net]
[ 574.582586] ? dev_queue_xmit_nit+0x5fb/0x850
[ 574.586348] ? virtnet_del_vqs+0xf0/0xf0 [virtio_net]
[ 574.590153] ? __skb_clone+0x24a/0x7d0
[ 574.593835] start_xmit+0x15a/0x1620 [virtio_net]
[ 574.597939] dev_hard_start_xmit+0x17f/0x7e0
[ 574.601832] sch_direct_xmit+0x2a8/0x5d0
[ 574.605665] ? dev_deactivate_queue.constprop.31+0x150/0x150
[ 574.609827] __dev_queue_xmit+0x1124/0x18b0
[ 574.613595] ? selinux_ip_postroute+0x4b2/0xa90
[ 574.617928] ? netdev_pick_tx+0x2d0/0x2d0
[ 574.621852] ? mark_held_locks+0xc8/0x120
[ 574.625673] ? ip_finish_output+0x626/0x9b0
[ 574.631679] ? ip_finish_output2+0xb44/0x1160
[ 574.637642] dev_queue_xmit+0x17/0x20
[ 574.641693] ip_finish_output2+0xcd1/0x1160
[ 574.645621] ? do_add_counters+0x480/0x480
[ 574.649554] ? do_add_counters+0x403/0x480
[ 574.653209] ? ip_copy_metadata+0x630/0x630
[ 574.657066] ip_finish_output+0x626/0x9b0
[ 574.660482] ? ip_finish_output+0x626/0x9b0
[ 574.663905] ip_output+0x1e2/0x580
[ 574.667235] ? ip_mc_output+0xe80/0xe80
[ 574.670574] ? ip_fragment.constprop.57+0x200/0x200
[ 574.673949] ip_local_out+0x95/0x160
[ 574.677249] ? __sk_dst_check+0xa7/0x260
[ 574.680446] ip_queue_xmit+0x889/0x17f0
[ 574.683575] ? __tcp_v4_send_check+0x1b8/0x350
[ 574.686801] tcp_transmit_skb+0x194a/0x2db0
[ 574.689832] ? __tcp_select_window+0x500/0x500
[ 574.693310] ? sched_clock_cpu+0x1b/0x190
[ 574.696371] ? tcp_grow_window.isra.24+0x2a8/0x4b0
[ 574.699509] tcp_send_ack+0x46f/0x710
[ 574.702395] __tcp_ack_snd_check+0x233/0x380
[ 574.705365] tcp_rcv_established+0x14eb/0x2230
[ 574.708332] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.711800] ? tcp_data_queue+0x3e70/0x3e70
[ 574.714761] ? sk_wait_data+0x2af/0x400
[ 574.719220] tcp_v4_do_rcv+0x56c/0x820
[ 574.724018] tcp_prequeue_process+0x18f/0x2c0
[ 574.729062] tcp_recvmsg+0xff6/0x26a0
[ 574.734615] ? tcp_tx_timestamp.part.27+0x290/0x290
[ 574.739519] ? _copy_from_user+0x84/0xe0
[ 574.744115] ? rw_copy_check_uvector+0x1f6/0x290
[ 574.748722] ? sock_has_perm+0x1e4/0x270
[ 574.751537] ? selinux_tun_dev_create+0xc0/0xc0
[ 574.754068] inet_recvmsg+0x117/0x530
[ 574.756823] ? memzero_page+0x130/0x130
[ 574.759503] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.762303] ? selinux_socket_recvmsg+0x36/0x40
[ 574.765114] ? security_socket_recvmsg+0x8f/0xc0
[ 574.768156] ? inet_sk_rebuild_header+0x1880/0x1880
[ 574.771181] sock_recvmsg+0xd7/0x110
[ 574.773972] ? __sock_recv_wifi_status+0x180/0x180
[ 574.777002] ___sys_recvmsg+0x24d/0x560
[ 574.779789] ? ___sys_sendmsg+0x920/0x920
[ 574.782734] ? __fget+0x200/0x380
[ 574.785657] ? lock_downgrade+0x650/0x650
[ 574.788584] ? __fget+0x229/0x380
[ 574.791362] ? __fget_light+0xa1/0x1f0
[ 574.794162] ? __fdget+0x18/0x20
[ 574.796832] __sys_recvmsg+0xce/0x170
[ 574.799572] ? __sys_recvmsg+0xce/0x170
[ 574.802695] ? SyS_sendmmsg+0x60/0x60
[ 574.805461] ? __schedule+0x7cb/0x1a70
[ 574.808211] ? retint_kernel+0x10/0x10
[ 574.810922] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.813890] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 574.816783] ? trace_hardirqs_on_caller+0x3f4/0x560
[ 574.819693] SyS_recvmsg+0x2d/0x50
[ 574.822829] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.825697] RIP: 0033:0x7f7fbd77e3c0
[ 574.828366] RSP: 002b:00007f7fba39ed50 EFLAGS: 00000293 ORIG_RAX:
000000000000002f
[ 574.833588] RAX: ffffffffffffffda RBX: 0000000000000008 RCX:
00007f7fbd77e3c0
[ 574.838882] RDX: 0000000000000000 RSI: 00007f7fba39edb0 RDI:
0000000000000008
[ 574.844377] RBP: 0000000000000046 R08: 0000000000000000 R09:
000000a0ff7159c1
[ 574.849937] R10: 00143f7b62d9620b R11: 0000000000000293 R12:
0000000000000000
[ 574.855391] R13: 0000000004000000 R14: 00007f7fa0000b10 R15:
0000000000000001
[ 574.860146]
[ 574.862738] Allocated by task 2291:
[ 574.865528] save_stack_trace+0x16/0x20
[ 574.868370] save_stack+0x46/0xd0
[ 574.871096] kasan_kmalloc+0xad/0xe0
[ 574.873838] __kmalloc+0x115/0x2d0
[ 574.876524] __vring_new_virtqueue+0x6a/0x790
[ 574.879432] vring_create_virtqueue+0x203/0x380
[ 574.882367] setup_vq+0x159/0x660
[ 574.885115] vp_setup_vq+0xbe/0x390
[ 574.887802] vp_find_vqs_msix+0x568/0xb90
[ 574.890494] vp_find_vqs+0x93/0x460
[ 574.893175] vp_modern_find_vqs+0x44/0x170
[ 574.895932] init_vqs+0x8eb/0x1150 [virtio_net]
[ 574.898778] virtnet_restore_up+0x4c/0x5c0 [virtio_net]
[ 574.901889] virtnet_xdp+0x820/0xd00 [virtio_net]
[ 574.904858] dev_change_xdp_fd+0x1bb/0x340
[ 574.907708] do_setlink+0x23fb/0x2c00
[ 574.910491] rtnl_setlink+0x280/0x340
[ 574.913448] rtnetlink_rcv_msg+0x288/0x680
[ 574.916348] netlink_rcv_skb+0x340/0x470
[ 574.919165] rtnetlink_rcv+0x2a/0x40
[ 574.922027] netlink_unicast+0x58d/0x860
[ 574.924897] netlink_sendmsg+0x8d2/0xca0
[ 574.927815] sock_sendmsg+0xca/0x110
[ 574.930708] SYSC_sendto+0x20d/0x340
[ 574.933562] SyS_sendto+0x40/0x50
[ 574.936380] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 574.939322]
[ 574.941724] Freed by task 2291:
[ 574.944389] save_stack_trace+0x16/0x20
[ 574.947107] save_stack+0x46/0xd0
[ 574.949893] kasan_slab_free+0x72/0xc0
[ 574.952526] kfree+0xe6/0x2c0
[ 574.955082] vring_del_virtqueue+0xef/0x220
[ 574.957773] del_vq+0x126/0x270
[ 574.960283] vp_del_vqs+0x1f5/0xa30
[ 574.962743] virtnet_del_vqs+0xb7/0xf0 [virtio_net]
[ 574.965930] virtnet_xdp+0x7b8/0xd00 [virtio_net]
[ 574.968762] dev_change_xdp_fd+0x309/0x340
[ 574.971487] do_setlink+0x23fb/0x2c00
[ 574.974041] rtnl_setlink+0x280/0x340
[ 574.976727] rtnetlink_rcv_msg+0x288/0x680
[ 574.979366] netlink_rcv_skb+0x340/0x470
[ 574.981949] rtnetlink_rcv+0x2a/0x40
[ 574.984462] netlink_unicast+0x58d/0x860
[ 574.987151] netlink_sendmsg+0x8d2/0xca0
[ 574.989736] sock_sendmsg+0xca/0x110
[ 574.992351] SYSC_sendto+0x20d/0x340
[ 574.995262] SyS_sendto+0x40/0x50
[ 574.998959] entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 575.001625]
[ 575.003976] The buggy address belongs to the object at ffff88005d220000
[ 575.003976] which belongs to the cache kmalloc-8192 of size 8192
[ 575.010183] The buggy address is located 32 bytes inside of
[ 575.010183] 8192-byte region [ffff88005d220000, ffff88005d222000)
[ 575.016265] The buggy address belongs to the page:
[ 575.019125] page:ffffea0001748800 count:1 mapcount:0 mapping:
(null) index:0x0 compound_mapcount: 0
[ 575.025320] flags: 0x100000000008100(slab|head)
[ 575.028167] raw: 0100000000008100 0000000000000000 0000000000000000
0000000100030003
[ 575.031632] raw: dead000000000100 dead000000000200 ffff88006c802280
0000000000000000
[ 575.035447] page dumped because: kasan: bad access detected
[ 575.039170]
[ 575.041893] Memory state around the buggy address:
[ 575.045408] ffff88005d21ff00: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.051399] ffff88005d21ff80: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 575.057558] >ffff88005d220000: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.063760] ^
[ 575.069310] ffff88005d220080: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.075359] ffff88005d220100: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 575.081461]
==================================================================
[ 575.086914] Disabling lock debugging due to kernel taint
[ 575.090717] virtio_net virtio1: output.0:id 31 is not a head!
[ 575.096336] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.102000] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.107383] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.112785] net enp0s4: Unexpected TXQ (0) queue failure: -5
[ 575.118228] net enp0s4: Unexpected TXQ (0) queue failure: -5
(gdb) l *(free_old_xmit_skbs+0x29b)
0x20db is in free_old_xmit_skbs (drivers/net/virtio_net.c:1051).
1046
1047 static void free_old_xmit_skbs(struct send_queue *sq)
1048 {
1049 struct sk_buff *skb;
1050 unsigned int len;
1051 struct virtnet_info *vi = sq->vq->vdev->priv;
1052 struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
1053 unsigned int packets = 0;
1054 unsigned int bytes = 0;
1055
Best regards,
Jean-Philippe
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-23 8:43 ` [Qemu-devel] " Jason Wang
@ 2017-06-23 22:32 ` Cong Wang
-1 siblings, 0 replies; 27+ messages in thread
From: Cong Wang @ 2017-06-23 22:32 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, jean-philippe menil,
Linux Kernel Network Developers, John Fastabend, virtualization,
qemu-devel Developers
On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is not
>>> null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
It is tricky here.
>From my understanding of the code base, the tx_lock is not sufficient
here, because in virtnet_del_vqs() all vqs are deleted and one vp
maps to one txq.
I am afraid you have to add a spinlock somewhere to serialized
free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
they are in different layers, so it is hard to figure out where to add
it...
Also, make sure we don't sleep inside the spinlock, I see a
synchronize_net().
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-23 22:32 ` Cong Wang
0 siblings, 0 replies; 27+ messages in thread
From: Cong Wang @ 2017-06-23 22:32 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, jean-philippe menil,
Linux Kernel Network Developers, John Fastabend, virtualization,
qemu-devel Developers
On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is not
>>> null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
It is tricky here.
>From my understanding of the code base, the tx_lock is not sufficient
here, because in virtnet_del_vqs() all vqs are deleted and one vp
maps to one txq.
I am afraid you have to add a spinlock somewhere to serialized
free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
they are in different layers, so it is hard to figure out where to add
it...
Also, make sure we don't sleep inside the spinlock, I see a
synchronize_net().
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-23 8:43 ` [Qemu-devel] " Jason Wang
` (3 preceding siblings ...)
(?)
@ 2017-06-23 22:32 ` Cong Wang
-1 siblings, 0 replies; 27+ messages in thread
From: Cong Wang @ 2017-06-23 22:32 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, Linux Kernel Network Developers,
John Fastabend, qemu-devel Developers, virtualization,
jean-philippe menil
On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>
>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>
>>> Hi Michael,
>>>
>>> from what i see, the race appear when we hit virtnet_reset in
>>> virtnet_xdp_set.
>>> virtnet_reset
>>> _remove_vq_common
>>> virtnet_del_vqs
>>> virtnet_free_queues
>>> kfree(vi->sq)
>>> when the xdp program (with two instances of the program to trigger it
>>> faster)
>>> is added or removed.
>>>
>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>> line,
>>> running the xdp_ttl tool from Jesper.
>>>
>>> For now, i'm able to continue my qualification, testing if xdp_qp is not
>>> null,
>>> but do not seem to be a sustainable trick.
>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>
>>> Maybe it will be more clear to you with theses informations.
>>>
>>> Best regards.
>>>
>>> Jean-Philippe
>>
>>
>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>> Jason, any thoughts?
>>
>>
>
> Hi Jean:
>
> Does the following fix this issue? (I can't reproduce it locally through
> xdp_ttl)
It is tricky here.
From my understanding of the code base, the tx_lock is not sufficient
here, because in virtnet_del_vqs() all vqs are deleted and one vp
maps to one txq.
I am afraid you have to add a spinlock somewhere to serialized
free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
they are in different layers, so it is hard to figure out where to add
it...
Also, make sure we don't sleep inside the spinlock, I see a
synchronize_net().
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-23 22:32 ` [Qemu-devel] " Cong Wang
@ 2017-06-26 2:50 ` Jason Wang
-1 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-26 2:50 UTC (permalink / raw)
To: Cong Wang
Cc: Michael S. Tsirkin, Linux Kernel Network Developers,
John Fastabend, qemu-devel Developers, virtualization,
jean-philippe menil
On 2017年06月24日 06:32, Cong Wang wrote:
> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>>
>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>> Hi Michael,
>>>>
>>>> from what i see, the race appear when we hit virtnet_reset in
>>>> virtnet_xdp_set.
>>>> virtnet_reset
>>>> _remove_vq_common
>>>> virtnet_del_vqs
>>>> virtnet_free_queues
>>>> kfree(vi->sq)
>>>> when the xdp program (with two instances of the program to trigger it
>>>> faster)
>>>> is added or removed.
>>>>
>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>> line,
>>>> running the xdp_ttl tool from Jesper.
>>>>
>>>> For now, i'm able to continue my qualification, testing if xdp_qp is not
>>>> null,
>>>> but do not seem to be a sustainable trick.
>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>
>>>> Maybe it will be more clear to you with theses informations.
>>>>
>>>> Best regards.
>>>>
>>>> Jean-Philippe
>>>
>>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>>> Jason, any thoughts?
>>>
>>>
>> Hi Jean:
>>
>> Does the following fix this issue? (I can't reproduce it locally through
>> xdp_ttl)
> It is tricky here.
>
> From my understanding of the code base, the tx_lock is not sufficient
> here, because in virtnet_del_vqs() all vqs are deleted and one vp
> maps to one txq.
>
> I am afraid you have to add a spinlock somewhere to serialized
> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
> they are in different layers, so it is hard to figure out where to add
> it...
>
> Also, make sure we don't sleep inside the spinlock, I see a
> synchronize_net().
Looks like I miss something. I thought free_old_xmit_skbs() were
serialized in this case since we disable all tx queues after
netif_tx_unlock_bh()?
Jean:
I thought this could be easily reproduced by e.g produce some traffic
and in the same time try to attach an xdp program. But looks not. How do
you trigger this? What's your qemu command line for this?
Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-26 2:50 ` Jason Wang
0 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-26 2:50 UTC (permalink / raw)
To: Cong Wang
Cc: Michael S. Tsirkin, jean-philippe menil,
Linux Kernel Network Developers, John Fastabend, virtualization,
qemu-devel Developers
On 2017年06月24日 06:32, Cong Wang wrote:
> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>>
>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>> Hi Michael,
>>>>
>>>> from what i see, the race appear when we hit virtnet_reset in
>>>> virtnet_xdp_set.
>>>> virtnet_reset
>>>> _remove_vq_common
>>>> virtnet_del_vqs
>>>> virtnet_free_queues
>>>> kfree(vi->sq)
>>>> when the xdp program (with two instances of the program to trigger it
>>>> faster)
>>>> is added or removed.
>>>>
>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>> line,
>>>> running the xdp_ttl tool from Jesper.
>>>>
>>>> For now, i'm able to continue my qualification, testing if xdp_qp is not
>>>> null,
>>>> but do not seem to be a sustainable trick.
>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>
>>>> Maybe it will be more clear to you with theses informations.
>>>>
>>>> Best regards.
>>>>
>>>> Jean-Philippe
>>>
>>> I'm pretty clear about the issue here, I was trying to figure out a fix.
>>> Jason, any thoughts?
>>>
>>>
>> Hi Jean:
>>
>> Does the following fix this issue? (I can't reproduce it locally through
>> xdp_ttl)
> It is tricky here.
>
> From my understanding of the code base, the tx_lock is not sufficient
> here, because in virtnet_del_vqs() all vqs are deleted and one vp
> maps to one txq.
>
> I am afraid you have to add a spinlock somewhere to serialized
> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
> they are in different layers, so it is hard to figure out where to add
> it...
>
> Also, make sure we don't sleep inside the spinlock, I see a
> synchronize_net().
Looks like I miss something. I thought free_old_xmit_skbs() were
serialized in this case since we disable all tx queues after
netif_tx_unlock_bh()?
Jean:
I thought this could be easily reproduced by e.g produce some traffic
and in the same time try to attach an xdp program. But looks not. How do
you trigger this? What's your qemu command line for this?
Thanks
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-26 2:50 ` [Qemu-devel] " Jason Wang
@ 2017-06-26 7:35 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-26 7:35 UTC (permalink / raw)
To: Jason Wang, Cong Wang
Cc: Michael S. Tsirkin, Linux Kernel Network Developers,
John Fastabend, virtualization, qemu-devel Developers
On 06/26/2017 04:50 AM, Jason Wang wrote:
>
>
> On 2017年06月24日 06:32, Cong Wang wrote:
>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>> Hi Michael,
>>>>>
>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>> virtnet_xdp_set.
>>>>> virtnet_reset
>>>>> _remove_vq_common
>>>>> virtnet_del_vqs
>>>>> virtnet_free_queues
>>>>> kfree(vi->sq)
>>>>> when the xdp program (with two instances of the program to trigger it
>>>>> faster)
>>>>> is added or removed.
>>>>>
>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>> line,
>>>>> running the xdp_ttl tool from Jesper.
>>>>>
>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>> is not
>>>>> null,
>>>>> but do not seem to be a sustainable trick.
>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>
>>>>> Maybe it will be more clear to you with theses informations.
>>>>>
>>>>> Best regards.
>>>>>
>>>>> Jean-Philippe
>>>>
>>>> I'm pretty clear about the issue here, I was trying to figure out a
>>>> fix.
>>>> Jason, any thoughts?
>>>>
>>>>
>>> Hi Jean:
>>>
>>> Does the following fix this issue? (I can't reproduce it locally through
>>> xdp_ttl)
>> It is tricky here.
>>
>> From my understanding of the code base, the tx_lock is not sufficient
>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>> maps to one txq.
>>
>> I am afraid you have to add a spinlock somewhere to serialized
>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>> they are in different layers, so it is hard to figure out where to add
>> it...
>>
>> Also, make sure we don't sleep inside the spinlock, I see a
>> synchronize_net().
>
> Looks like I miss something. I thought free_old_xmit_skbs() were
> serialized in this case since we disable all tx queues after
> netif_tx_unlock_bh()?
>
> Jean:
>
> I thought this could be easily reproduced by e.g produce some traffic
> and in the same time try to attach an xdp program. But looks not. How do
> you trigger this? What's your qemu command line for this?
>
> Thanks
Hi Jason,
this is how i trigger the bug:
- on the guest, tcpdump on on the interface
- on the guest, run iperf against the host
- on the guest, cat /sys/kernel/debug/tracing/trace_pipe
- on the guest, run one or two instances of xdp_ttl compiled with DEBUG
uncommented, that i start stop, until i trigger the bug.
qemu command line is as follow:
qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm -smp
2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048 -rtc
base=localtime,clock=host -usbdevice tablet --balloon virtio -netdev
tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
-device
virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
-vnc 127.0.0.1:3 -nographic -serial
file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
Notice, the smp 2, queues to 4 and vectors to 2.
Seem that if fogot to mention that in the beginning of this thread,
sorry for that.
Best regards.
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-26 7:35 ` Jean-Philippe Menil
0 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-26 7:35 UTC (permalink / raw)
To: Jason Wang, Cong Wang
Cc: Michael S. Tsirkin, Linux Kernel Network Developers,
John Fastabend, virtualization, qemu-devel Developers
On 06/26/2017 04:50 AM, Jason Wang wrote:
>
>
> On 2017年06月24日 06:32, Cong Wang wrote:
>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>> Hi Michael,
>>>>>
>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>> virtnet_xdp_set.
>>>>> virtnet_reset
>>>>> _remove_vq_common
>>>>> virtnet_del_vqs
>>>>> virtnet_free_queues
>>>>> kfree(vi->sq)
>>>>> when the xdp program (with two instances of the program to trigger it
>>>>> faster)
>>>>> is added or removed.
>>>>>
>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>> line,
>>>>> running the xdp_ttl tool from Jesper.
>>>>>
>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>> is not
>>>>> null,
>>>>> but do not seem to be a sustainable trick.
>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>
>>>>> Maybe it will be more clear to you with theses informations.
>>>>>
>>>>> Best regards.
>>>>>
>>>>> Jean-Philippe
>>>>
>>>> I'm pretty clear about the issue here, I was trying to figure out a
>>>> fix.
>>>> Jason, any thoughts?
>>>>
>>>>
>>> Hi Jean:
>>>
>>> Does the following fix this issue? (I can't reproduce it locally through
>>> xdp_ttl)
>> It is tricky here.
>>
>> From my understanding of the code base, the tx_lock is not sufficient
>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>> maps to one txq.
>>
>> I am afraid you have to add a spinlock somewhere to serialized
>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>> they are in different layers, so it is hard to figure out where to add
>> it...
>>
>> Also, make sure we don't sleep inside the spinlock, I see a
>> synchronize_net().
>
> Looks like I miss something. I thought free_old_xmit_skbs() were
> serialized in this case since we disable all tx queues after
> netif_tx_unlock_bh()?
>
> Jean:
>
> I thought this could be easily reproduced by e.g produce some traffic
> and in the same time try to attach an xdp program. But looks not. How do
> you trigger this? What's your qemu command line for this?
>
> Thanks
Hi Jason,
this is how i trigger the bug:
- on the guest, tcpdump on on the interface
- on the guest, run iperf against the host
- on the guest, cat /sys/kernel/debug/tracing/trace_pipe
- on the guest, run one or two instances of xdp_ttl compiled with DEBUG
uncommented, that i start stop, until i trigger the bug.
qemu command line is as follow:
qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm -smp
2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048 -rtc
base=localtime,clock=host -usbdevice tablet --balloon virtio -netdev
tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
-device
virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
-vnc 127.0.0.1:3 -nographic -serial
file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
Notice, the smp 2, queues to 4 and vectors to 2.
Seem that if fogot to mention that in the beginning of this thread,
sorry for that.
Best regards.
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-26 2:50 ` [Qemu-devel] " Jason Wang
(?)
(?)
@ 2017-06-26 7:35 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-26 7:35 UTC (permalink / raw)
To: Jason Wang, Cong Wang
Cc: Linux Kernel Network Developers, virtualization, John Fastabend,
qemu-devel Developers, Michael S. Tsirkin
On 06/26/2017 04:50 AM, Jason Wang wrote:
>
>
> On 2017年06月24日 06:32, Cong Wang wrote:
>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>> Hi Michael,
>>>>>
>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>> virtnet_xdp_set.
>>>>> virtnet_reset
>>>>> _remove_vq_common
>>>>> virtnet_del_vqs
>>>>> virtnet_free_queues
>>>>> kfree(vi->sq)
>>>>> when the xdp program (with two instances of the program to trigger it
>>>>> faster)
>>>>> is added or removed.
>>>>>
>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>> line,
>>>>> running the xdp_ttl tool from Jesper.
>>>>>
>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>> is not
>>>>> null,
>>>>> but do not seem to be a sustainable trick.
>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>
>>>>> Maybe it will be more clear to you with theses informations.
>>>>>
>>>>> Best regards.
>>>>>
>>>>> Jean-Philippe
>>>>
>>>> I'm pretty clear about the issue here, I was trying to figure out a
>>>> fix.
>>>> Jason, any thoughts?
>>>>
>>>>
>>> Hi Jean:
>>>
>>> Does the following fix this issue? (I can't reproduce it locally through
>>> xdp_ttl)
>> It is tricky here.
>>
>> From my understanding of the code base, the tx_lock is not sufficient
>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>> maps to one txq.
>>
>> I am afraid you have to add a spinlock somewhere to serialized
>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>> they are in different layers, so it is hard to figure out where to add
>> it...
>>
>> Also, make sure we don't sleep inside the spinlock, I see a
>> synchronize_net().
>
> Looks like I miss something. I thought free_old_xmit_skbs() were
> serialized in this case since we disable all tx queues after
> netif_tx_unlock_bh()?
>
> Jean:
>
> I thought this could be easily reproduced by e.g produce some traffic
> and in the same time try to attach an xdp program. But looks not. How do
> you trigger this? What's your qemu command line for this?
>
> Thanks
Hi Jason,
this is how i trigger the bug:
- on the guest, tcpdump on on the interface
- on the guest, run iperf against the host
- on the guest, cat /sys/kernel/debug/tracing/trace_pipe
- on the guest, run one or two instances of xdp_ttl compiled with DEBUG
uncommented, that i start stop, until i trigger the bug.
qemu command line is as follow:
qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm -smp
2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048 -rtc
base=localtime,clock=host -usbdevice tablet --balloon virtio -netdev
tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
-device
virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
-vnc 127.0.0.1:3 -nographic -serial
file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
Notice, the smp 2, queues to 4 and vectors to 2.
Seem that if fogot to mention that in the beginning of this thread,
sorry for that.
Best regards.
Jean-Philippe
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-26 7:35 ` [Qemu-devel] " Jean-Philippe Menil
@ 2017-06-27 2:13 ` Jason Wang
-1 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-27 2:13 UTC (permalink / raw)
To: jpmenil, Cong Wang
Cc: Linux Kernel Network Developers, Michael S. Tsirkin,
John Fastabend, qemu-devel Developers, virtualization
On 2017年06月26日 15:35, Jean-Philippe Menil wrote:
> On 06/26/2017 04:50 AM, Jason Wang wrote:
>>
>>
>> On 2017年06月24日 06:32, Cong Wang wrote:
>>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com>
>>> wrote:
>>>>
>>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>>> virtnet_xdp_set.
>>>>>> virtnet_reset
>>>>>> _remove_vq_common
>>>>>> virtnet_del_vqs
>>>>>> virtnet_free_queues
>>>>>> kfree(vi->sq)
>>>>>> when the xdp program (with two instances of the program to
>>>>>> trigger it
>>>>>> faster)
>>>>>> is added or removed.
>>>>>>
>>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>>> line,
>>>>>> running the xdp_ttl tool from Jesper.
>>>>>>
>>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>>> is not
>>>>>> null,
>>>>>> but do not seem to be a sustainable trick.
>>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>>
>>>>>> Maybe it will be more clear to you with theses informations.
>>>>>>
>>>>>> Best regards.
>>>>>>
>>>>>> Jean-Philippe
>>>>>
>>>>> I'm pretty clear about the issue here, I was trying to figure out
>>>>> a fix.
>>>>> Jason, any thoughts?
>>>>>
>>>>>
>>>> Hi Jean:
>>>>
>>>> Does the following fix this issue? (I can't reproduce it locally
>>>> through
>>>> xdp_ttl)
>>> It is tricky here.
>>>
>>> From my understanding of the code base, the tx_lock is not sufficient
>>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>>> maps to one txq.
>>>
>>> I am afraid you have to add a spinlock somewhere to serialized
>>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>>> they are in different layers, so it is hard to figure out where to add
>>> it...
>>>
>>> Also, make sure we don't sleep inside the spinlock, I see a
>>> synchronize_net().
>>
>> Looks like I miss something. I thought free_old_xmit_skbs() were
>> serialized in this case since we disable all tx queues after
>> netif_tx_unlock_bh()?
>>
>> Jean:
>>
>> I thought this could be easily reproduced by e.g produce some traffic
>> and in the same time try to attach an xdp program. But looks not. How
>> do you trigger this? What's your qemu command line for this?
>>
>> Thanks
>
> Hi Jason,
>
> this is how i trigger the bug:
> - on the guest, tcpdump on on the interface
> - on the guest, run iperf against the host
> - on the guest, cat /sys/kernel/debug/tracing/trace_pipe
> - on the guest, run one or two instances of xdp_ttl compiled with
> DEBUG uncommented, that i start stop, until i trigger the bug.
>
> qemu command line is as follow:
>
> qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm
> -smp 2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048
> -rtc base=localtime,clock=host -usbdevice tablet --balloon virtio
> -netdev
> tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
> -device
> virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
> -vnc 127.0.0.1:3 -nographic -serial
> file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
> unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
>
> Notice, the smp 2, queues to 4 and vectors to 2.
> Seem that if fogot to mention that in the beginning of this thread,
> sorry for that.
>
> Best regards.
>
> Jean-Philippe
>
Thanks Jean, I manage to reproduce the issue.
I thought netif_tx_unlock_bh() will do tx lock but looks not, that's why
previous patch doesn't work.
Could you please this this patch? (At least it can't trigger the warning
after more than 20 times of xdp start/stop).
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 1f8c15c..a18f859 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1802,6 +1802,7 @@ static void virtnet_freeze_down(struct
virtio_device *vdev)
flush_work(&vi->config_work);
netif_device_detach(vi->dev);
+ netif_tx_disable(vi->dev);
cancel_delayed_work_sync(&vi->refill);
if (netif_running(vi->dev)) {
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
@ 2017-06-27 2:13 ` Jason Wang
0 siblings, 0 replies; 27+ messages in thread
From: Jason Wang @ 2017-06-27 2:13 UTC (permalink / raw)
To: jpmenil, Cong Wang
Cc: Linux Kernel Network Developers, virtualization, John Fastabend,
qemu-devel Developers, Michael S. Tsirkin
On 2017年06月26日 15:35, Jean-Philippe Menil wrote:
> On 06/26/2017 04:50 AM, Jason Wang wrote:
>>
>>
>> On 2017年06月24日 06:32, Cong Wang wrote:
>>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com>
>>> wrote:
>>>>
>>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>>> virtnet_xdp_set.
>>>>>> virtnet_reset
>>>>>> _remove_vq_common
>>>>>> virtnet_del_vqs
>>>>>> virtnet_free_queues
>>>>>> kfree(vi->sq)
>>>>>> when the xdp program (with two instances of the program to
>>>>>> trigger it
>>>>>> faster)
>>>>>> is added or removed.
>>>>>>
>>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>>> line,
>>>>>> running the xdp_ttl tool from Jesper.
>>>>>>
>>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>>> is not
>>>>>> null,
>>>>>> but do not seem to be a sustainable trick.
>>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>>
>>>>>> Maybe it will be more clear to you with theses informations.
>>>>>>
>>>>>> Best regards.
>>>>>>
>>>>>> Jean-Philippe
>>>>>
>>>>> I'm pretty clear about the issue here, I was trying to figure out
>>>>> a fix.
>>>>> Jason, any thoughts?
>>>>>
>>>>>
>>>> Hi Jean:
>>>>
>>>> Does the following fix this issue? (I can't reproduce it locally
>>>> through
>>>> xdp_ttl)
>>> It is tricky here.
>>>
>>> From my understanding of the code base, the tx_lock is not sufficient
>>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>>> maps to one txq.
>>>
>>> I am afraid you have to add a spinlock somewhere to serialized
>>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>>> they are in different layers, so it is hard to figure out where to add
>>> it...
>>>
>>> Also, make sure we don't sleep inside the spinlock, I see a
>>> synchronize_net().
>>
>> Looks like I miss something. I thought free_old_xmit_skbs() were
>> serialized in this case since we disable all tx queues after
>> netif_tx_unlock_bh()?
>>
>> Jean:
>>
>> I thought this could be easily reproduced by e.g produce some traffic
>> and in the same time try to attach an xdp program. But looks not. How
>> do you trigger this? What's your qemu command line for this?
>>
>> Thanks
>
> Hi Jason,
>
> this is how i trigger the bug:
> - on the guest, tcpdump on on the interface
> - on the guest, run iperf against the host
> - on the guest, cat /sys/kernel/debug/tracing/trace_pipe
> - on the guest, run one or two instances of xdp_ttl compiled with
> DEBUG uncommented, that i start stop, until i trigger the bug.
>
> qemu command line is as follow:
>
> qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm
> -smp 2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048
> -rtc base=localtime,clock=host -usbdevice tablet --balloon virtio
> -netdev
> tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
> -device
> virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
> -vnc 127.0.0.1:3 -nographic -serial
> file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
> unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
>
> Notice, the smp 2, queues to 4 and vectors to 2.
> Seem that if fogot to mention that in the beginning of this thread,
> sorry for that.
>
> Best regards.
>
> Jean-Philippe
>
Thanks Jean, I manage to reproduce the issue.
I thought netif_tx_unlock_bh() will do tx lock but looks not, that's why
previous patch doesn't work.
Could you please this this patch? (At least it can't trigger the warning
after more than 20 times of xdp start/stop).
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 1f8c15c..a18f859 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1802,6 +1802,7 @@ static void virtnet_freeze_down(struct
virtio_device *vdev)
flush_work(&vi->config_work);
netif_device_detach(vi->dev);
+ netif_tx_disable(vi->dev);
cancel_delayed_work_sync(&vi->refill);
if (netif_running(vi->dev)) {
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-27 2:13 ` Jason Wang
(?)
@ 2017-06-27 12:35 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-27 12:35 UTC (permalink / raw)
To: Jason Wang, Cong Wang
Cc: Linux Kernel Network Developers, virtualization, John Fastabend,
qemu-devel Developers, Michael S. Tsirkin
On 06/27/2017 04:13 AM, Jason Wang wrote:
>
>
> On 2017年06月26日 15:35, Jean-Philippe Menil wrote:
>> On 06/26/2017 04:50 AM, Jason Wang wrote:
>>>
>>>
>>> On 2017年06月24日 06:32, Cong Wang wrote:
>>>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com>
>>>> wrote:
>>>>>
>>>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>>>> virtnet_xdp_set.
>>>>>>> virtnet_reset
>>>>>>> _remove_vq_common
>>>>>>> virtnet_del_vqs
>>>>>>> virtnet_free_queues
>>>>>>> kfree(vi->sq)
>>>>>>> when the xdp program (with two instances of the program to
>>>>>>> trigger it
>>>>>>> faster)
>>>>>>> is added or removed.
>>>>>>>
>>>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>>>> line,
>>>>>>> running the xdp_ttl tool from Jesper.
>>>>>>>
>>>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>>>> is not
>>>>>>> null,
>>>>>>> but do not seem to be a sustainable trick.
>>>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>>>
>>>>>>> Maybe it will be more clear to you with theses informations.
>>>>>>>
>>>>>>> Best regards.
>>>>>>>
>>>>>>> Jean-Philippe
>>>>>>
>>>>>> I'm pretty clear about the issue here, I was trying to figure out
>>>>>> a fix.
>>>>>> Jason, any thoughts?
>>>>>>
>>>>>>
>>>>> Hi Jean:
>>>>>
>>>>> Does the following fix this issue? (I can't reproduce it locally
>>>>> through
>>>>> xdp_ttl)
>>>> It is tricky here.
>>>>
>>>> From my understanding of the code base, the tx_lock is not sufficient
>>>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>>>> maps to one txq.
>>>>
>>>> I am afraid you have to add a spinlock somewhere to serialized
>>>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>>>> they are in different layers, so it is hard to figure out where to add
>>>> it...
>>>>
>>>> Also, make sure we don't sleep inside the spinlock, I see a
>>>> synchronize_net().
>>>
>>> Looks like I miss something. I thought free_old_xmit_skbs() were
>>> serialized in this case since we disable all tx queues after
>>> netif_tx_unlock_bh()?
>>>
>>> Jean:
>>>
>>> I thought this could be easily reproduced by e.g produce some traffic
>>> and in the same time try to attach an xdp program. But looks not. How
>>> do you trigger this? What's your qemu command line for this?
>>>
>>> Thanks
>>
>> Hi Jason,
>>
>> this is how i trigger the bug:
>> - on the guest, tcpdump on on the interface
>> - on the guest, run iperf against the host
>> - on the guest, cat /sys/kernel/debug/tracing/trace_pipe
>> - on the guest, run one or two instances of xdp_ttl compiled with
>> DEBUG uncommented, that i start stop, until i trigger the bug.
>>
>> qemu command line is as follow:
>>
>> qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm
>> -smp 2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048
>> -rtc base=localtime,clock=host -usbdevice tablet --balloon virtio
>> -netdev
>> tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
>> -device
>> virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
>> -vnc 127.0.0.1:3 -nographic -serial
>> file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
>> unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
>>
>> Notice, the smp 2, queues to 4 and vectors to 2.
>> Seem that if fogot to mention that in the beginning of this thread,
>> sorry for that.
>>
>> Best regards.
>>
>> Jean-Philippe
>>
>
> Thanks Jean, I manage to reproduce the issue.
>
> I thought netif_tx_unlock_bh() will do tx lock but looks not, that's why
> previous patch doesn't work.
>
> Could you please this this patch? (At least it can't trigger the warning
> after more than 20 times of xdp start/stop).
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1f8c15c..a18f859 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1802,6 +1802,7 @@ static void virtnet_freeze_down(struct
> virtio_device *vdev)
> flush_work(&vi->config_work);
>
> netif_device_detach(vi->dev);
> + netif_tx_disable(vi->dev);
> cancel_delayed_work_sync(&vi->refill);
>
> if (netif_running(vi->dev)) {
>
>
Hi Jason,
Seem to do the trick !
with your patch, i'm unable to repeat the problem anymore (running more
than 2h without any issue).
Best regards.
Jean-Philippe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs
2017-06-27 2:13 ` Jason Wang
(?)
(?)
@ 2017-06-27 12:35 ` Jean-Philippe Menil
-1 siblings, 0 replies; 27+ messages in thread
From: Jean-Philippe Menil @ 2017-06-27 12:35 UTC (permalink / raw)
To: Jason Wang, Cong Wang
Cc: Linux Kernel Network Developers, Michael S. Tsirkin,
John Fastabend, qemu-devel Developers, virtualization
On 06/27/2017 04:13 AM, Jason Wang wrote:
>
>
> On 2017年06月26日 15:35, Jean-Philippe Menil wrote:
>> On 06/26/2017 04:50 AM, Jason Wang wrote:
>>>
>>>
>>> On 2017年06月24日 06:32, Cong Wang wrote:
>>>> On Fri, Jun 23, 2017 at 1:43 AM, Jason Wang <jasowang@redhat.com>
>>>> wrote:
>>>>>
>>>>> On 2017年06月23日 02:53, Michael S. Tsirkin wrote:
>>>>>> On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> from what i see, the race appear when we hit virtnet_reset in
>>>>>>> virtnet_xdp_set.
>>>>>>> virtnet_reset
>>>>>>> _remove_vq_common
>>>>>>> virtnet_del_vqs
>>>>>>> virtnet_free_queues
>>>>>>> kfree(vi->sq)
>>>>>>> when the xdp program (with two instances of the program to
>>>>>>> trigger it
>>>>>>> faster)
>>>>>>> is added or removed.
>>>>>>>
>>>>>>> It's easily repeatable, with 2 cpus and 4 queues on the qemu command
>>>>>>> line,
>>>>>>> running the xdp_ttl tool from Jesper.
>>>>>>>
>>>>>>> For now, i'm able to continue my qualification, testing if xdp_qp
>>>>>>> is not
>>>>>>> null,
>>>>>>> but do not seem to be a sustainable trick.
>>>>>>> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp)
>>>>>>>
>>>>>>> Maybe it will be more clear to you with theses informations.
>>>>>>>
>>>>>>> Best regards.
>>>>>>>
>>>>>>> Jean-Philippe
>>>>>>
>>>>>> I'm pretty clear about the issue here, I was trying to figure out
>>>>>> a fix.
>>>>>> Jason, any thoughts?
>>>>>>
>>>>>>
>>>>> Hi Jean:
>>>>>
>>>>> Does the following fix this issue? (I can't reproduce it locally
>>>>> through
>>>>> xdp_ttl)
>>>> It is tricky here.
>>>>
>>>> From my understanding of the code base, the tx_lock is not sufficient
>>>> here, because in virtnet_del_vqs() all vqs are deleted and one vp
>>>> maps to one txq.
>>>>
>>>> I am afraid you have to add a spinlock somewhere to serialized
>>>> free_old_xmit_skbs() vs. vring_del_virtqueue(). As you can see
>>>> they are in different layers, so it is hard to figure out where to add
>>>> it...
>>>>
>>>> Also, make sure we don't sleep inside the spinlock, I see a
>>>> synchronize_net().
>>>
>>> Looks like I miss something. I thought free_old_xmit_skbs() were
>>> serialized in this case since we disable all tx queues after
>>> netif_tx_unlock_bh()?
>>>
>>> Jean:
>>>
>>> I thought this could be easily reproduced by e.g produce some traffic
>>> and in the same time try to attach an xdp program. But looks not. How
>>> do you trigger this? What's your qemu command line for this?
>>>
>>> Thanks
>>
>> Hi Jason,
>>
>> this is how i trigger the bug:
>> - on the guest, tcpdump on on the interface
>> - on the guest, run iperf against the host
>> - on the guest, cat /sys/kernel/debug/tracing/trace_pipe
>> - on the guest, run one or two instances of xdp_ttl compiled with
>> DEBUG uncommented, that i start stop, until i trigger the bug.
>>
>> qemu command line is as follow:
>>
>> qemu-system-x86_64 -name ubuntu --enable-kvm -machine pc,accel=kvm
>> -smp 2 -drive file=/dev/LocalDisk/ubuntu,if=virtio,format=raw -m 2048
>> -rtc base=localtime,clock=host -usbdevice tablet --balloon virtio
>> -netdev
>> tap,id=ubuntu-0,ifname=ubuntu-0,script=/home/jenfi/WORK/jp/qemu/if-up,downscript=/home/jenfi/WORK/jp/qemu/if-down,vhost=on,queues=4
>> -device
>> virtio-net-pci,netdev=ubuntu-0,mac=de:ad:be:ef:01:03,mq=on,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,vectors=2
>> -vnc 127.0.0.1:3 -nographic -serial
>> file:/home/jenfi/WORK/jp/qemu/ubuntu.out -monitor
>> unix:/home/jenfi/WORK/jp/qemu/ubuntu.sock,server,nowait
>>
>> Notice, the smp 2, queues to 4 and vectors to 2.
>> Seem that if fogot to mention that in the beginning of this thread,
>> sorry for that.
>>
>> Best regards.
>>
>> Jean-Philippe
>>
>
> Thanks Jean, I manage to reproduce the issue.
>
> I thought netif_tx_unlock_bh() will do tx lock but looks not, that's why
> previous patch doesn't work.
>
> Could you please this this patch? (At least it can't trigger the warning
> after more than 20 times of xdp start/stop).
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1f8c15c..a18f859 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1802,6 +1802,7 @@ static void virtnet_freeze_down(struct
> virtio_device *vdev)
> flush_work(&vi->config_work);
>
> netif_device_detach(vi->dev);
> + netif_tx_disable(vi->dev);
> cancel_delayed_work_sync(&vi->refill);
>
> if (netif_running(vi->dev)) {
>
>
Hi Jason,
Seem to do the trick !
with your patch, i'm unable to repeat the problem anymore (running more
than 2h without any issue).
Best regards.
Jean-Philippe
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2017-06-27 12:35 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-04 22:48 BUG: KASAN: use-after-free in free_old_xmit_skbs Jean-Philippe Menil
2017-06-05 2:08 ` Michael S. Tsirkin
2017-06-05 23:52 ` Michael S. Tsirkin
2017-06-05 23:52 ` [Qemu-devel] " Michael S. Tsirkin
2017-06-22 6:15 ` jean-philippe menil
2017-06-22 6:15 ` [Qemu-devel] " jean-philippe menil
2017-06-22 18:53 ` Michael S. Tsirkin
2017-06-22 18:53 ` Michael S. Tsirkin
2017-06-22 18:53 ` [Qemu-devel] " Michael S. Tsirkin
2017-06-23 8:43 ` Jason Wang
2017-06-23 8:43 ` [Qemu-devel] " Jason Wang
2017-06-23 9:33 ` Jean-Philippe Menil
2017-06-23 9:33 ` [Qemu-devel] " Jean-Philippe Menil
2017-06-23 9:33 ` Jean-Philippe Menil
2017-06-23 22:32 ` Cong Wang
2017-06-23 22:32 ` [Qemu-devel] " Cong Wang
2017-06-26 2:50 ` Jason Wang
2017-06-26 2:50 ` [Qemu-devel] " Jason Wang
2017-06-26 7:35 ` Jean-Philippe Menil
2017-06-26 7:35 ` [Qemu-devel] " Jean-Philippe Menil
2017-06-27 2:13 ` Jason Wang
2017-06-27 2:13 ` Jason Wang
2017-06-27 12:35 ` Jean-Philippe Menil
2017-06-27 12:35 ` Jean-Philippe Menil
2017-06-26 7:35 ` Jean-Philippe Menil
2017-06-23 22:32 ` Cong Wang
2017-06-05 23:52 ` Michael S. Tsirkin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.