Re: [Qemu-devel] [PATCH 10/12] virtio-net: multiqueue support

[Jason Wang <jasowang@redhat.com>]

On 01/09/2013 06:01 PM, Wanlong Gao wrote:
> On 01/09/2013 05:30 PM, Jason Wang wrote:
>> On 01/09/2013 04:23 PM, Wanlong Gao wrote:
>>> On 01/08/2013 06:14 PM, Jason Wang wrote:
>>>> On 01/08/2013 06:00 PM, Wanlong Gao wrote:
>>>>> On 01/08/2013 05:51 PM, Jason Wang wrote:
>>>>>> On 01/08/2013 05:49 PM, Wanlong Gao wrote:
>>>>>>> On 01/08/2013 05:29 PM, Jason Wang wrote:
>>>>>>>> On 01/08/2013 05:07 PM, Wanlong Gao wrote:
>>>>>>>>> On 12/28/2012 06:32 PM, Jason Wang wrote:
>>>>>>>>>> +    } else if (nc->peer->info->type !=  NET_CLIENT_OPTIONS_KIND_TAP) {
>>>>>>>>>> +        ret = -1;
>>>>>>>>>> +    } else {
>>>>>>>>>> +        ret = tap_detach(nc->peer);
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    return ret;
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +static void virtio_net_set_queues(VirtIONet *n)
>>>>>>>>>> +{
>>>>>>>>>> +    int i;
>>>>>>>>>> +
>>>>>>>>>> +    for (i = 0; i < n->max_queues; i++) {
>>>>>>>>>> +        if (i < n->curr_queues) {
>>>>>>>>>> +            assert(!peer_attach(n, i));
>>>>>>>>>> +        } else {
>>>>>>>>>> +            assert(!peer_detach(n, i));
>>>>>>>>> I got a assert here,
>>>>>>>>> qemu-system-x86_64: /work/git/qemu/hw/virtio-net.c:330: virtio_net_set_queues: Assertion `!peer_detach(n, i)' failed.
>>>>>>>>>
>>>>>>>>> Any thoughts?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Wanlong Gao
>>>>>>>> Thanks for the testing, which steps or cases did you met this assertion,
>>>>>>>> migration, reboot or just changing the number of virtqueues?
>>>>>>> I use the 3.8-rc2 to test it again, I saw this tag has the multi-tap support.
>>>>>>>
>>>>>>> I just can't start the QEMU use  -netdev tap,id=hostnet0,queues=2,fd=%d,fd=%d -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3
>>>>>>>
>>>>>>> I pre-opened two tap fds, did I missing something?
>>>>>> Nothing missed :) It should work.
>>>>>>
>>>>>> Could you please try not use fd=X and let qemu to create the file
>>>>>> descriptors by itself? Btw, how did you create the two tap fds?
>>>>> Can it create descriptors itself? I get 
>>>>> qemu-system-x86_64: -netdev tap,id=hostnet0,queues=2: Device 'tap' could not be initialized
>>>> You need prepare an ifup script which default at /etc/qemu-ifup (like
>>>> following). Or you may try to add a script=no after:
>>>>
>>>> #!/bin/sh
>>>>
>>>> switch=kvmbr0
>>>>
>>>> /sbin/ifconfig $1 0.0.0.0 up
>>>> /usr/sbin/brctl addif $switch $1
>>>> /usr/sbin/brctl stp $switch off
>>>>
>>>> This will let qemu create a tap fd itself and make it to be connected to
>>>> a port of the bridge caled kvmbr0.
>>> But how to support multi-queue in this way?
>> Qemu will create the necessary multiqueue tap by itself, see patch 0/12.
>>> I got guest kernel panic when using this way and set queues=4.
>> Does it happens w/o or w/ a fd parameter? What's the qemu command line?
>> Did you meet it during boot time?
> The QEMU command line is 
>
> /work/git/qemu/x86_64-softmmu/qemu-system-x86_64 -name f17 -M pc-0.15 -enable-kvm -m 3096 \
> -smp 4,sockets=4,cores=1,threads=1 \
> -uuid c31a9f3e-4161-c53a-339c-5dc36d0497cb -no-user-config -nodefaults \
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f17.monitor,server,nowait \
> -mon chardev=charmonitor,id=monitor,mode=control \
> -rtc base=utc -no-shutdown \
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xb,num_queues=4,hotplug=on \
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
> -drive file=/vm/f17.img,if=none,id=drive-virtio-disk0,format=qcow2 \
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
> -drive file=/vm2/f17-kernel.img,if=none,id=drive-virtio-disk1,format=qcow2 \
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 \
> -drive file=/vm/virtio-scsi/scsi3.img,if=none,id=drive-scsi0-0-2-0,format=raw \
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-2-0,id=scsi0-0-2-0,removable=on \
> -drive file=/vm/virtio-scsi/scsi4.img,if=none,id=drive-scsi0-0-3-0,format=raw \
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi0-0-3-0,id=scsi0-0-3-0 \
> -drive file=/vm/virtio-scsi/scsi1.img,if=none,id=drive-scsi0-0-0-0,format=raw \
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
> -drive file=/vm/virtio-scsi/scsi2.img,if=none,id=drive-scsi0-0-1-0,format=raw \
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 \
> -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
> -chardev file,id=charserial1,path=/vm/f17.log \
> -device isa-serial,chardev=charserial1,id=serial1 \
> -device usb-tablet,id=input0 -vga std \
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
> -netdev tap,id=hostnet0,vhost=on,queues=4 \
> -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3 \
> -monitor stdio
>
> I got panic just after booting the system, did nothing,  waited for a while, the guest panicked.
>
> [   28.053004] BUG: soft lockup - CPU#1 stuck for 23s! [ip:592]
> [   28.053004] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables uinput joydev microcode virtio_balloon pcspkr virtio_net i2c_piix4 i2c_core virtio_scsi virtio_blk floppy
> [   28.053004] CPU 1 
> [   28.053004] Pid: 592, comm: ip Not tainted 3.8.0-rc1-net+ #3 Bochs Bochs
> [   28.053004] RIP: 0010:[<ffffffff8137a9ab>]  [<ffffffff8137a9ab>] virtqueue_get_buf+0xb/0x120
> [   28.053004] RSP: 0018:ffff8800bc913550  EFLAGS: 00000246
> [   28.053004] RAX: 0000000000000000 RBX: ffff8800bc49c000 RCX: ffff8800bc49e000
> [   28.053004] RDX: 0000000000000000 RSI: ffff8800bc913584 RDI: ffff8800bcfd4000
> [   28.053004] RBP: ffff8800bc913558 R08: ffff8800bcfd0800 R09: 0000000000000000
> [   28.053004] R10: ffff8800bc49c000 R11: ffff880036cc4de0 R12: ffff8800bcfd4000
> [   28.053004] R13: ffff8800bc913558 R14: ffffffff8137ad73 R15: 00000000000200d0
> [   28.053004] FS:  00007fb27a589740(0000) GS:ffff8800c1480000(0000) knlGS:0000000000000000
> [   28.053004] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   28.053004] CR2: 0000000000640530 CR3: 00000000baeff000 CR4: 00000000000006e0
> [   28.053004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   28.053004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   28.053004] Process ip (pid: 592, threadinfo ffff8800bc912000, task ffff880036da2e20)
> [   28.053004] Stack:
> [   28.053004]  ffff8800bcfd0800 ffff8800bc913638 ffffffffa003e9bb ffff8800bc913656
> [   28.053004]  0000000100000002 ffff8800c17ebb08 000000500000ff10 ffffea0002f244c0
> [   28.053004]  0000000200000582 0000000000000000 0000000000000000 ffffea0002f244c0
> [   28.053004] Call Trace:
> [   28.053004]  [<ffffffffa003e9bb>] virtnet_send_command.constprop.26+0x24b/0x270 [virtio_net]
> [   28.053004]  [<ffffffff812ed963>] ? sg_init_table+0x23/0x50
> [   28.053004]  [<ffffffffa0040629>] virtnet_set_rx_mode+0x99/0x300 [virtio_net]
> [   28.053004]  [<ffffffff8152306f>] __dev_set_rx_mode+0x5f/0xb0
> [   28.053004]  [<ffffffff815230ef>] dev_set_rx_mode+0x2f/0x50
> [   28.053004]  [<ffffffff815231b7>] __dev_open+0xa7/0xf0
> [   28.053004]  [<ffffffff81523461>] __dev_change_flags+0xa1/0x180
> [   28.053004]  [<ffffffff815235f8>] dev_change_flags+0x28/0x70
> [   28.053004]  [<ffffffff8152ff20>] do_setlink+0x3b0/0xa50
> [   28.053004]  [<ffffffff812fb6b1>] ? nla_parse+0x31/0xe0
> [   28.053004]  [<ffffffff815325de>] rtnl_newlink+0x36e/0x580
> [   28.053004]  [<ffffffff811355cc>] ? get_page_from_freelist+0x37c/0x730
> [   28.053004]  [<ffffffff81531e13>] rtnetlink_rcv_msg+0x113/0x2f0
> [   28.053004]  [<ffffffff8117d973>] ? __kmalloc_node_track_caller+0x63/0x1c0
> [   28.053004]  [<ffffffff8151526b>] ? __alloc_skb+0x8b/0x2a0
> [   28.053004]  [<ffffffff81531d00>] ? __rtnl_unlock+0x20/0x20
> [   28.053004]  [<ffffffff8154b571>] netlink_rcv_skb+0xb1/0xc0
> [   28.053004]  [<ffffffff8152ea05>] rtnetlink_rcv+0x25/0x40
> [   28.053004]  [<ffffffff8154ae91>] netlink_unicast+0x1a1/0x220
> [   28.053004]  [<ffffffff8154b211>] netlink_sendmsg+0x301/0x3c0
> [   28.053004]  [<ffffffff81508530>] sock_sendmsg+0xb0/0xe0
> [   28.053004]  [<ffffffff8113a45b>] ? lru_cache_add_lru+0x3b/0x60
> [   28.053004]  [<ffffffff811608b7>] ? page_add_new_anon_rmap+0xc7/0x180
> [   28.053004]  [<ffffffff81509efc>] __sys_sendmsg+0x3ac/0x3c0
> [   28.053004]  [<ffffffff8162e47c>] ? __do_page_fault+0x23c/0x4d0
> [   28.053004]  [<ffffffff8115c9ef>] ? do_brk+0x1ff/0x370
> [   28.053004]  [<ffffffff8150bec9>] sys_sendmsg+0x49/0x90
> [   28.053004]  [<ffffffff81632d59>] system_call_fastpath+0x16/0x1b
> [   28.053004] Code: 04 0f ae f0 48 8b 47 50 5d 0f b7 50 02 66 39 57 64 0f 94 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 <53> 80 7f 59 00 48 89 fb 0f 85 90 00 00 00 48 8b 47 50 0f b7 50 
>
>
> The QEMU tree I used is git://github.com/jasowang/qemu.git

Thanks a lot, will try to reproduce my self tomorrow. From the
calltrace, looks like we send a command to a rx/tx queue.
> Thanks,
> Wanlong Gao
>
>> Thanks
>>> Thanks,
>>> Wanlong Gao
>>>
>>>>> I create the tap fd like this, and dup create the second fd, third fd, right?
>>>> The second and third fd should be created with TUNSETIFF with the same
>>>> tap_name also. Btw, you need to specify a IFF_MULTI_QUEUE flag to tell
>>>> the kernel you want to create a multiqueue tap device, otherwise the
>>>> second and third calling of TUNSETIFF will fail.
>>>>
>>>> Thanks
>>>>> 	int tap_fd = open("/dev/net/tun", O_RDWR);
>>>>> 	int vhost_fd = open("/dev/vhost-net", O_RDWR);
>>>>> 	char *tap_name = "tap";
>>>>> 	char cmd[2048];
>>>>> 	char brctl[256];
>>>>> 	char netup[256];
>>>>> 	struct ifreq ifr;
>>>>> 	if (tap_fd < 0) {
>>>>> 		printf("open tun device failed\n");
>>>>> 		return -1;
>>>>> 	}
>>>>> 	if (vhost_fd < 0) {
>>>>> 		printf("open vhost-net device failed\n");
>>>>> 		return -1;
>>>>> 	}
>>>>> 	memset(&ifr, 0, sizeof(ifr));
>>>>> 	memcpy(ifr.ifr_name, tap_name, sizeof(tap_name));
>>>>> 	ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
>>>>>
>>>>> 	/*
>>>>> 	 * setup tap net device
>>>>> 	 */
>>>>> 	if (ioctl(tap_fd, TUNSETIFF, &ifr) < 0) {
>>>>> 		printf("setup tap net device failed\n");
>>>>> 		return -1;
>>>>> 	}
>>>>>
>>>>> 	sprintf(brctl, "brctl addif virbr0 %s", tap_name);
>>>>> 	sprintf(netup, "ifconfig %s up", tap_name);
>>>>> 	system(brctl);
>>>>> 	system(netup);
>>>>>
>>>>> Thanks,
>>>>> Wanlong Gao
>>>>>
>>>>>
>>>>>> Thanks
>>>>>>> Thanks,
>>>>>>> Wanlong Gao
>>>>>>>
>>>>>>>>>> +        }
>>>>>>>>>> +    }
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue, int ctrl);
>>>>>>>>>> +
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>