Re: vhost-net patches

Re: vhost-net patches

[Sridhar Samudrala]

On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
> > On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
> > > Hi Michael,
> > > 
> > > We are trying out your vhost-net patches from your git trees on kernel.org.
> > > I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.
> > > 
> > > I am using the following qemu script to start the guest using userspace tap backend.
> > > 
> > > home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/sridhar/kvm_images/fedora10-1-vm -m 512 -drive file=/home/sridhar/kvm_images/fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:35:e3:73,model=virtio -net tap,ifname=vnet0,script=no,downscript=no
> > > 
> > > Now that i got the default backend to work, i wanted to try vhost in kernel. But
> > > could not figure out the right -net option to pass to qemu.
> > > 
> > > Can you let me know the right syntax to start a guest using vhost.
> > > 
> > > Thanks
> > > Sridhar
> > 
> > Here's an example with raw socket:
> > 
> > /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
> > /boot/vmlinuz-$release -append \
> > 'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
> > console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
> > $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
> > -balloon none -redir tcp:8023::22
> > 
> > As you see, I changed the command line.
> > You now simply add ",vhost" after model, and it will locate
> > host network interface specified earlier and attach to it.
> > This should have been clear from running  qemu with -help
> > flag. Could you please suggest how can that text
> > be clarified?

I updated to your latest git trees and the default user-space tap backend using the
following -net options worked fine.
-net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio

But i could not get vhost to work with either raw or tap backends.
I tried the following combinations.
1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost

They all failed with the following error
    vhost_net_init returned -7
This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
vhost_net_start() fails. It looks like dev->binding->irqfd() is failing in
vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
enabled in the host kernel.

Are all the above -net options supposed to work?

In your descriptions, you say that checksum/tso offload is not supported. Isn't it
possible to send/receive large packets without checksum using AF_PACKET sockets if
the attached interface supports these offloads.
Do you see the same offload issue even when using tap backend via vhost?

Thanks
Sridhar

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 19, 2009 at 03:56:54PM -0700, Sridhar Samudrala wrote:
> On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> > On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
> > > On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
> > > > Hi Michael,
> > > > 
> > > > We are trying out your vhost-net patches from your git trees on kernel.org.
> > > > I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.
> > > > 
> > > > I am using the following qemu script to start the guest using userspace tap backend.
> > > > 
> > > > home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/sridhar/kvm_images/fedora10-1-vm -m 512 -drive file=/home/sridhar/kvm_images/fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:35:e3:73,model=virtio -net tap,ifname=vnet0,script=no,downscript=no
> > > > 
> > > > Now that i got the default backend to work, i wanted to try vhost in kernel. But
> > > > could not figure out the right -net option to pass to qemu.
> > > > 
> > > > Can you let me know the right syntax to start a guest using vhost.
> > > > 
> > > > Thanks
> > > > Sridhar
> > > 
> > > Here's an example with raw socket:
> > > 
> > > /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
> > > /boot/vmlinuz-$release -append \
> > > 'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
> > > console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
> > > $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
> > > -balloon none -redir tcp:8023::22
> > > 
> > > As you see, I changed the command line.
> > > You now simply add ",vhost" after model, and it will locate
> > > host network interface specified earlier and attach to it.
> > > This should have been clear from running  qemu with -help
> > > flag. Could you please suggest how can that text
> > > be clarified?
> 
> I updated to your latest git trees and the default user-space tap backend using the
> following -net options worked fine.
> -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
> 
> But i could not get vhost to work with either raw or tap backends.
> I tried the following combinations.
> 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
> 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
> 3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost
> 
> They all failed with the following error
>     vhost_net_init returned -7
> This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
> vhost_net_start() fails. It looks like dev->binding->irqfd() is failing in
> vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
> enabled in the host kernel.
> 
> Are all the above -net options supposed to work?
> 
> In your descriptions, you say that checksum/tso offload is not supported.

They should work with tap but not raw sockets yet.

> Isn't it
> possible to send/receive large packets without checksum using AF_PACKET sockets if
> the attached interface supports these offloads.
> Do you see the same offload issue even when using tap backend via vhost?
> 
> Thanks
> Sridhar
> 
> 
> 
> 
> 
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> >They all failed with the following error
> >    vhost_net_init returned -7
> >This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
> >vhost_net_start() fails. It looks like dev->binding->irqfd() is failing in
> >vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
> >enabled in the host kernel.
> 
> >From the debug output, looks like the vnet->vector is not defined,

what is vnet->vector?
And what do you mean by not defined?

> and the
> default msix_entries_nr is 3, so it returned EINVAL from virtio_pci_irqfd.
> Looks we need to either disable QEMU_PCI_CAP_MSIX or define vector in QEMU
> configuration?

You shouldn't have to do anything.

> I am not familiar with MSIX stuffs.
> 
> Thanks
> Shirley
> 
> 
> Inactive hide details for sri---10/19/2009 03:56:57 PM---On Sun, 2009-10-18 at
> 19:32 +0200, Michael S. Tsirkin wrote:sri---10/19/2009 03:56:57 PM---On Sun,
> 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> 
>                 sri@linux.vnet.ibm.com [cid]   *
>                                             To "Michael S. Tsirkin" <mst@redhat.com>,
>                 10/19/2009 03:56 PM            kvm@vger.kernel.org
>                                        [cid]   *
>                                             cc David Stevens/Beaverton/IBM@IBMUS, Shirley Ma/
>                                                Beaverton/IBM@IBMUS
>                                        [cid]   *
>                                        Subject Re: vhost-net patches
>                                        *        *
> 
> On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> > On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
> > > On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
> > > > Hi Michael,
> > > >
> > > > We are trying out your vhost-net patches from your git trees on
> kernel.org.
> > > > I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.
> > > >
> > > > I am using the following qemu script to start the guest using userspace
> tap backend.
> > > >
> > > > home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/
> sridhar/kvm_images/fedora10-1-vm -m 512 -drive file=/home/sridhar/kvm_images/
> fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=
> 54:52:00:35:e3:73,model=virtio -net tap,ifname=vnet0,script=no,downscript=no
> > > >
> > > > Now that i got the default backend to work, i wanted to try vhost in
> kernel. But
> > > > could not figure out the right -net option to pass to qemu.
> > > >
> > > > Can you let me know the right syntax to start a guest using vhost.
> > > >
> > > > Thanks
> > > > Sridhar
> > >
> > > Here's an example with raw socket:
> > >
> > > /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
> > > /boot/vmlinuz-$release -append \
> > > 'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
> > > console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
> > > $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
> > > -balloon none -redir tcp:8023::22
> > >
> > > As you see, I changed the command line.
> > > You now simply add ",vhost" after model, and it will locate
> > > host network interface specified earlier and attach to it.
> > > This should have been clear from running  qemu with -help
> > > flag. Could you please suggest how can that text
> > > be clarified?
> 
> I updated to your latest git trees and the default user-space tap backend using
> the
> following -net options worked fine.
> -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
> 
> But i could not get vhost to work with either raw or tap backends.
> I tried the following combinations.
> 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
> 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
> 3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost

Yes, should work.

> 
> They all failed with the following error
>    vhost_net_init returned -7
> This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
> vhost_net_start() fails. It looks like dev->binding->irqfd() is failing in
> vhost_virtqueue_init().

what parameters are passed in?

> Haven't yet debugged further.

this calls into virtio_pci_irqfd.

> I have CONFIG_EVENTFD
> enabled in the host kernel.

Note you need to also enable eventfd support under kvm menu.

> Are all the above -net options supposed to work?
> 
> In your descriptions, you say that checksum/tso offload is not supported. Isn't
> it
> possible to send/receive large packets without checksum using AF_PACKET sockets
> if
> the attached interface supports these offloads.
> Do you see the same offload issue even when using tap backend via vhost?
> 
> Thanks
> Sridhar
> 
> 
> 
> 
> 
> 
> 
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 20, 2009 at 10:14:55AM -0700, Shirley Ma wrote:
> 
> Hello Michael,
> 
> >what is vnet->vector?
> >And what do you mean by not defined?
> 
> In funcation:
> 
> static int vhost_virtqueue_init()
> {
> ......
>               r = vdev->binding->irqfd(vdev->binding_opaque, q->vector,
> vq->call);
> ......
> }.....
> 
> q->vector is 65535,

Thanks for debugging this.
I think this means that guest does not use MSI-X.

You can verify this by booting guest without vhost,
and performing the following command:
	 cat /proc/interrupts

Please note that you currently need recent kernel in guest,
so that it uses MSI-X. I plan on implementing regular IRQ,
but not yet, and it will be slower anyway.

> in   static int virtio_pci_irqfd()
> {
> ......
>     if (vector >= proxy->pci_dev.msix_entries_nr) {
>         fprintf(stderr, " pci irq fd returned vector %d, msix_entries_nr %d
> \n",
>                 vector, proxy->pci_dev.msix_entries_nr); <------- I added
> one output line here.
>         return -EINVAL;
> ....
> }...
> The output is:
> 
>        pci irq fd returned vector 65535, msix_entries_nr 3, EINVAL is
> returned.
> 
> thanks
> Shirley Ma
> IBM Linux Technology Center
> 15300 SW Koll Parkway
> Beaverton, OR 97006-6063
> Phone(Fax): (503) 578-7638
> 
> 
> 
> 
>                                                                        
>              "Michael S.                                               
>              Tsirkin"                                                  
>              <mst@redhat.com>                                           To
>                                        Shirley Ma/Beaverton/IBM@IBMUS  
>              10/20/2009 04:34                                           cc
>              AM                        sri@linux.vnet.ibm.com, David   
>                                        Stevens/Beaverton/IBM@IBMUS,    
>                                        kvm@vger.kernel.org             
>                                                                    Subject
>                                        Re: vhost-net patches           
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
> 
> 
> 
> 
> On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
> > Hello Michael,
> >
> > >They all failed with the following error
> > >    vhost_net_init returned -7
> > >This is an error message from hw/virtio-net.c:virtio_net_driver_ok()
> when
> > >vhost_net_start() fails. It looks like dev->binding->irqfd() is failing
> in
> > >vhost_virtqueue_init(). Haven't yet debugged further. I have
> CONFIG_EVENTFD
> > >enabled in the host kernel.
> >
> > >From the debug output, looks like the vnet->vector is not defined,
> 
> what is vnet->vector?
> And what do you mean by not defined?
> 
> > and the
> > default msix_entries_nr is 3, so it returned EINVAL from
> virtio_pci_irqfd.
> > Looks we need to either disable QEMU_PCI_CAP_MSIX or define vector in
> QEMU
> > configuration?
> 
> You shouldn't have to do anything.
> 
> > I am not familiar with MSIX stuffs.
> >
> > Thanks
> > Shirley
> >
> >
> > Inactive hide details for sri---10/19/2009 03:56:57 PM---On Sun,
> 2009-10-18 at
> > 19:32 +0200, Michael S. Tsirkin wrote:sri---10/19/2009 03:56:57 PM---On
> Sun,
> > 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> >
> >                 sri@linux.vnet.ibm.com [cid]   *
> >                                             To "Michael S. Tsirkin"
> <mst@redhat.com>,
> >                 10/19/2009 03:56 PM            kvm@vger.kernel.org
> >                                        [cid]   *
> >                                             cc David
> Stevens/Beaverton/IBM@IBMUS, Shirley Ma/
> >                                                Beaverton/IBM@IBMUS
> >                                        [cid]   *
> >                                        Subject Re: vhost-net patches
> >                                        *        *
> >
> > On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> > > On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
> > > > On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
> > > > > Hi Michael,
> > > > >
> > > > > We are trying out your vhost-net patches from your git trees on
> > kernel.org.
> > > > > I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for
> qemu.
> > > > >
> > > > > I am using the following qemu script to start the guest using
> userspace
> > tap backend.
> > > > >
> > > > >
> home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/
> > sridhar/kvm_images/fedora10-1-vm -m 512 -drive
> file=/home/sridhar/kvm_images/
> > fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=
> > 54:52:00:35:e3:73,model=virtio -net
> tap,ifname=vnet0,script=no,downscript=no
> > > > >
> > > > > Now that i got the default backend to work, i wanted to try vhost
> in
> > kernel. But
> > > > > could not figure out the right -net option to pass to qemu.
> > > > >
> > > > > Can you let me know the right syntax to start a guest using vhost.
> > > > >
> > > > > Thanks
> > > > > Sridhar
> > > >
> > > > Here's an example with raw socket:
> > > >
> > > > /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
> > > > /boot/vmlinuz-$release -append \
> > > > 'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
> > > > console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
> > > > $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost
> \
> > > > -balloon none -redir tcp:8023::22
> > > >
> > > > As you see, I changed the command line.
> > > > You now simply add ",vhost" after model, and it will locate
> > > > host network interface specified earlier and attach to it.
> > > > This should have been clear from running  qemu with -help
> > > > flag. Could you please suggest how can that text
> > > > be clarified?
> >
> > I updated to your latest git trees and the default user-space tap backend
> using
> > the
> > following -net options worked fine.
> > -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
> >
> > But i could not get vhost to work with either raw or tap backends.
> > I tried the following combinations.
> > 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
> > 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
> > 3) -net tap,ifname=vnet0,script=no,downscript=no -net
> nic,model=virtio,vhost
> 
> Yes, should work.
> 
> >
> > They all failed with the following error
> >    vhost_net_init returned -7
> > This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
> > vhost_net_start() fails. It looks like dev->binding->irqfd() is failing
> in
> > vhost_virtqueue_init().
> 
> what parameters are passed in?
> 
> > Haven't yet debugged further.
> 
> this calls into virtio_pci_irqfd.
> 
> > I have CONFIG_EVENTFD
> > enabled in the host kernel.
> 
> Note you need to also enable eventfd support under kvm menu.
> 
> > Are all the above -net options supposed to work?
> >
> > In your descriptions, you say that checksum/tso offload is not supported.
> Isn't
> > it
> > possible to send/receive large packets without checksum using AF_PACKET
> sockets
> > if
> > the attached interface supports these offloads.
> > Do you see the same offload issue even when using tap backend via vhost?
> >
> > Thanks
> > Sridhar
> >
> >
> >
> >
> >
> >
> >
> >
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 20, 2009 at 10:27:38AM -0700, Shirley Ma wrote:
> 
> Hello Michael,
> 
> Here are the output, I am using guest 2.6.32-rc3 kernel. It doesn't use
> MSIX. So which guest kernel I should use??
> 
> [xma@localhost ~]$ cat /proc/interrupts.
>            CPU0
>   0:        299   IO-APIC-edge      timer.
>   1:          2   IO-APIC-edge      i8042.
>   2:          0    XT-PIC-XT        cascade
>   4:         76   IO-APIC-edge      serial
>  11:       2126   IO-APIC-edge      virtio1, virtio0 <--------- here is the
> virtio for both disk and network i/o??

Yes, this is regular shared IRQ, no good.
I think your guest is too old, please use kernel 2.6.31 and up in guest.
I will work to improve the error message as well.

>  12:         89   IO-APIC-edge      i8042
> NMI:          0   Non-maskable interrupts
> LOC:       5146   Local timer interrupts
> SPU:          0   Spurious interrupts
> CNT:          0   Performance counter interrupts
> PND:          0   Performance pending work
> RES:          0   Rescheduling interrupts
> CAL:          0   Function call interrupts
> TLB:          0   TLB shootdowns
> TRM:          0   Thermal event interrupts
> MCE:          0   Machine check exceptions
> MCP:          1   Machine check polls
> ERR:          0
> MIS:          0
> [xma@localhost ~]$ uname -r
> 2.6.32-rc3
> 
> 
> Shirley Ma
> IBM Linux Technology Center
> 15300 SW Koll Parkway
> Beaverton, OR 97006-6063
> Phone(Fax): (503) 578-7638
> 
> 
> 
> 
>                                                                        
>              "Michael S.                                               
>              Tsirkin"                                                  
>              <mst@redhat.com>                                           To
>                                        Shirley Ma/Beaverton/IBM@IBMUS  
>              10/20/2009 10:18                                           cc
>              AM                        David Stevens/Beaverton/IBM@IBMUS,
>                                        kvm@vger.kernel.org,            
>                                        sri@linux.vnet.ibm.com          
>                                                                    Subject
>                                        Re: vhost-net patches           
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
> 
> 
> 
> 
> On Tue, Oct 20, 2009 at 10:14:55AM -0700, Shirley Ma wrote:
> >
> > Hello Michael,
> >
> > >what is vnet->vector?
> > >And what do you mean by not defined?
> >
> > In funcation:
> >
> > static int vhost_virtqueue_init()
> > {
> > ......
> >               r = vdev->binding->irqfd(vdev->binding_opaque, q->vector,
> > vq->call);
> > ......
> > }.....
> >
> > q->vector is 65535,
> 
> Thanks for debugging this.
> I think this means that guest does not use MSI-X.
> 
> You can verify this by booting guest without vhost,
> and performing the following command:
>               cat /proc/interrupts
> 
> Please note that you currently need recent kernel in guest,
> so that it uses MSI-X. I plan on implementing regular IRQ,
> but not yet, and it will be slower anyway.
> 
> > in   static int virtio_pci_irqfd()
> > {
> > ......
> >     if (vector >= proxy->pci_dev.msix_entries_nr) {
> >         fprintf(stderr, " pci irq fd returned vector %d, msix_entries_nr
> %d
> > \n",
> >                 vector, proxy->pci_dev.msix_entries_nr); <------- I added
> > one output line here.
> >         return -EINVAL;
> > ....
> > }...
> > The output is:
> >
> >        pci irq fd returned vector 65535, msix_entries_nr 3, EINVAL is
> > returned.
> >
> > thanks
> > Shirley Ma
> > IBM Linux Technology Center
> > 15300 SW Koll Parkway
> > Beaverton, OR 97006-6063
> > Phone(Fax): (503) 578-7638
> >
> >
> >
> >
> >
> >              "Michael S.
> >              Tsirkin"
> >              <mst@redhat.com>
> To
> >                                        Shirley Ma/Beaverton/IBM@IBMUS
> >              10/20/2009 04:34
> cc
> >              AM                        sri@linux.vnet.ibm.com, David
> >                                        Stevens/Beaverton/IBM@IBMUS,
> >                                        kvm@vger.kernel.org
> >
> Subject
> >                                        Re: vhost-net patches
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
> > > Hello Michael,
> > >
> > > >They all failed with the following error
> > > >    vhost_net_init returned -7
> > > >This is an error message from hw/virtio-net.c:virtio_net_driver_ok()
> > when
> > > >vhost_net_start() fails. It looks like dev->binding->irqfd() is
> failing
> > in
> > > >vhost_virtqueue_init(). Haven't yet debugged further. I have
> > CONFIG_EVENTFD
> > > >enabled in the host kernel.
> > >
> > > >From the debug output, looks like the vnet->vector is not defined,
> >
> > what is vnet->vector?
> > And what do you mean by not defined?
> >
> > > and the
> > > default msix_entries_nr is 3, so it returned EINVAL from
> > virtio_pci_irqfd.
> > > Looks we need to either disable QEMU_PCI_CAP_MSIX or define vector in
> > QEMU
> > > configuration?
> >
> > You shouldn't have to do anything.
> >
> > > I am not familiar with MSIX stuffs.
> > >
> > > Thanks
> > > Shirley
> > >
> > >
> > > Inactive hide details for sri---10/19/2009 03:56:57 PM---On Sun,
> > 2009-10-18 at
> > > 19:32 +0200, Michael S. Tsirkin wrote:sri---10/19/2009 03:56:57 PM---On
> > Sun,
> > > 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> > >
> > >                 sri@linux.vnet.ibm.com [cid]   *
> > >                                             To "Michael S. Tsirkin"
> > <mst@redhat.com>,
> > >                 10/19/2009 03:56 PM            kvm@vger.kernel.org
> > >                                        [cid]   *
> > >                                             cc David
> > Stevens/Beaverton/IBM@IBMUS, Shirley Ma/
> > >                                                Beaverton/IBM@IBMUS
> > >                                        [cid]   *
> > >                                        Subject Re: vhost-net patches
> > >                                        *        *
> > >
> > > On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
> > > > On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
> > > > > On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
> > > > > > Hi Michael,
> > > > > >
> > > > > > We are trying out your vhost-net patches from your git trees on
> > > kernel.org.
> > > > > > I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for
> > qemu.
> > > > > >
> > > > > > I am using the following qemu script to start the guest using
> > userspace
> > > tap backend.
> > > > > >
> > > > > >
> > home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/
> > > sridhar/kvm_images/fedora10-1-vm -m 512 -drive
> > file=/home/sridhar/kvm_images/
> > > fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=
> > > 54:52:00:35:e3:73,model=virtio -net
> > tap,ifname=vnet0,script=no,downscript=no
> > > > > >
> > > > > > Now that i got the default backend to work, i wanted to try vhost
> > in
> > > kernel. But
> > > > > > could not figure out the right -net option to pass to qemu.
> > > > > >
> > > > > > Can you let me know the right syntax to start a guest using
> vhost.
> > > > > >
> > > > > > Thanks
> > > > > > Sridhar
> > > > >
> > > > > Here's an example with raw socket:
> > > > >
> > > > > /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
> > > > > /boot/vmlinuz-$release -append \
> > > > > 'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
> > > > > console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
> > > > > $HOME/disk.raw.copy -net raw,ifname=eth3 -net
> nic,model=virtio,vhost
> > \
> > > > > -balloon none -redir tcp:8023::22
> > > > >
> > > > > As you see, I changed the command line.
> > > > > You now simply add ",vhost" after model, and it will locate
> > > > > host network interface specified earlier and attach to it.
> > > > > This should have been clear from running  qemu with -help
> > > > > flag. Could you please suggest how can that text
> > > > > be clarified?
> > >
> > > I updated to your latest git trees and the default user-space tap
> backend
> > using
> > > the
> > > following -net options worked fine.
> > > -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
> > >
> > > But i could not get vhost to work with either raw or tap backends.
> > > I tried the following combinations.
> > > 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
> > > 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
> > > 3) -net tap,ifname=vnet0,script=no,downscript=no -net
> > nic,model=virtio,vhost
> >
> > Yes, should work.
> >
> > >
> > > They all failed with the following error
> > >    vhost_net_init returned -7
> > > This is an error message from hw/virtio-net.c:virtio_net_driver_ok()
> when
> > > vhost_net_start() fails. It looks like dev->binding->irqfd() is failing
> > in
> > > vhost_virtqueue_init().
> >
> > what parameters are passed in?
> >
> > > Haven't yet debugged further.
> >
> > this calls into virtio_pci_irqfd.
> >
> > > I have CONFIG_EVENTFD
> > > enabled in the host kernel.
> >
> > Note you need to also enable eventfd support under kvm menu.
> >
> > > Are all the above -net options supposed to work?
> > >
> > > In your descriptions, you say that checksum/tso offload is not
> supported.
> > Isn't
> > > it
> > > possible to send/receive large packets without checksum using AF_PACKET
> > sockets
> > > if
> > > the attached interface supports these offloads.
> > > Do you see the same offload issue even when using tap backend via
> vhost?
> > >
> > > Thanks
> > > Sridhar
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
> Hello Micahel,
> 
> I have set up guest kernel 2.6.32-rc5 with MSI configured. Here are errors what
> I have got:
> 
> 1. First, qemu complained extboot.bin not found, I copied the file from
> optionrom/ dir to pc-bios/ dir, this problem is gone.
> 
> 2. Second, when guest boot up, it has lots of errors as below. Without vhost
> support, I still saw same errors but the guest interface can communicate with
> host, but with vhost, it doesn't work.

There was a ecent bugfix in qemu-kvm I pushed.
Could you please verify that you have cec75e39151e49cc90c849eab5d0d729667c9e68 ?


> I am posting the errors from /var/log/
> messages here:
> 
> virtio-pci 0000:00:03.0: can't find IRQ for PCI INT A; probably buggy MP table
> virtio-pci 0000:00:04.0: can't find IRQ for PCI INT A; probably buggy MP table
> virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
> IRQ handler type mismatch for IRQ 1
> current handler: i8042
> Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
> Call Trace:
> __setup_irq+0x24c/0x2ac
> request_threaded_irq+0x113/0x179
> ? vring_interrupt+0x0/0x2f
> vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> ? blk_done+0x0/0xa7 [virtio_blk]
> vp_find_vqs+0x1b/0x62 [virtio_pci]
> virtblk_probe+0xbd/0x3d0 [virtio_blk]
> ? sysfs_do_create_link+0xbb/0xfd
> ? blk_done+0x0/0xa7 [virtio_blk]
> ? add_status+0x1f/0x24
> virtio_dev_probe+0x91/0xb0
> driver_probe_device+0x79/0x105
> __driver_attach+0x43/0x5f
> bus_for_each_dev+0x3d/0x67
> driver_attach+0x14/0x16
> ? __driver_attach+0x0/0x5f
> bus_add_driver+0xa2/0x1c9
> driver_register+0x8b/0xeb
> ? init+0x0/0x24 [virtio_blk]
> register_virtio_driver+0x1f/0x22
> init+0x22/0x24 [virtio_blk]
> do_one_initcall+0x4c/0x13a
> sys_init_module+0xa7/0x1db
> syscall_call+0x7/0xb
> virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X

This was recently reported without vhost, did not
reproduce it here yet.
And you say you do not see the above without vhost?

> vda: vda1 vda2
> EXT3-fs: INFO: recovery required on readonly filesystem.
> EXT3-fs: write access will be enabled during recovery.
> kjournald starting. Commit interval 5 seconds
> EXT3-fs: recovery complete.
> EXT3-fs: mounted filesystem with writeback data mode.
> udevd version 127 started
> virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> virtio-pci 0000:00:03.0: irq 28 for MSI/MSI-X
> IRQ handler type mismatch for IRQ 1
> current handler: i8042
> Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
> Call Trace:
> __setup_irq+0x24c/0x2ac
> request_threaded_irq+0x113/0x179
> ? vring_interrupt+0x0/0x2f
> vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> ? skb_recv_done+0x0/0x36 [virtio_net]
> vp_find_vqs+0x1b/0x62 [virtio_pci]
> virtnet_probe+0x265/0x347 [virtio_net]
> ? skb_recv_done+0x0/0x36 [virtio_net]
> ? skb_xmit_done+0x0/0x1e [virtio_net]
> ? add_status+0x1f/0x24
> virtio_dev_probe+0x91/0xb0
> driver_probe_device+0x79/0x105
> __driver_attach+0x43/0x5f
> bus_for_each_dev+0x3d/0x67
> driver_attach+0x14/0x16
> ? __driver_attach+0x0/0x5f
> bus_add_driver+0xa2/0x1c9
> driver_register+0x8b/0xeb
> ? init+0x0/0xf [virtio_net]
> register_virtio_driver+0x1f/0x22
> init+0xd/0xf [virtio_net]
> do_one_initcall+0x4c/0x13a
> sys_init_module+0xa7/0x1db
> syscall_call+0x7/0xb
> virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> 
> 3. The guest interface is up, and cat /proc/interrupts outputs:
> 
> 24: 0 PCI-MSI-edge virtio1-config
> 25: 2571 PCI-MSI-edge virtio1-virtqueues
> 26: 0 PCI-MSI-edge virtio0-config
> 27: 0 PCI-MSI-edge virtio0-virtqueues
> 
> Thanks
> Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> >There was a recent bugfix in qemu-kvm I pushed.
> >Could you please verify that you have cec75e39151e49cc90c849eab5d0d729667c9e68
> ?
> 
> Yes, I cloned your qemu-kvm and kernel git. Here is "git log" output:
> 
> commit cec75e39151e49cc90c849eab5d0d729667c9e68
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Wed Oct 21 12:34:36 2009 +0200
> 
>     vhost: use acked features, not all features
> 
> >This was recently reported without vhost, did not
> >reproduce it here yet.
> >And you say you do not see the above without vhost?
> 
> I saw above with and without vhost, but the network worked with the errors
> without vhost.

Could you please supply more information?
What is the qemu command line you use?
What guest do you use?
How is the network configured?

Please attach tcpdump to interfaces in host and guest,
and run ping from guest.
Do you see any traffic? Does it look sane?



> Thanks
> Shirley
> 
> 
> Inactive hide details for "Michael S. Tsirkin" ---10/21/2009 01:24:58 PM---On
> Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma"Michael S. Tsirkin" ---10/21/
> 2009 01:24:58 PM---On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
> 
>                 "Michael S.        [cid]   *
>                 Tsirkin"                To Shirley Ma/Beaverton/IBM@IBMUS
>                 <mst@redhat.com>   [cid]   *
>                                         cc David Stevens/Beaverton/IBM@IBMUS,
>                 10/21/2009 01:19           kvm@vger.kernel.org, sri@linux.vnet.ibm.com
>                 PM                 [cid]   *
>                                    Subject Re: vhost-net patches
>                                    *        *
> 
> On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
> > Hello Micahel,
> >
> > I have set up guest kernel 2.6.32-rc5 with MSI configured. Here are errors
> what
> > I have got:
> >
> > 1. First, qemu complained extboot.bin not found, I copied the file from
> > optionrom/ dir to pc-bios/ dir, this problem is gone.
> >
> > 2. Second, when guest boot up, it has lots of errors as below. Without vhost
> > support, I still saw same errors but the guest interface can communicate with
> > host, but with vhost, it doesn't work.
> 
> There was a ecent bugfix in qemu-kvm I pushed.
> Could you please verify that you have cec75e39151e49cc90c849eab5d0d729667c9e68
> ?
> 
> 
> > I am posting the errors from /var/log/
> > messages here:
> >
> > virtio-pci 0000:00:03.0: can't find IRQ for PCI INT A; probably buggy MP
> table
> > virtio-pci 0000:00:04.0: can't find IRQ for PCI INT A; probably buggy MP
> table
> > virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> > virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
> > IRQ handler type mismatch for IRQ 1
> > current handler: i8042
> > Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
> > Call Trace:
> > __setup_irq+0x24c/0x2ac
> > request_threaded_irq+0x113/0x179
> > ? vring_interrupt+0x0/0x2f
> > vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> > ? blk_done+0x0/0xa7 [virtio_blk]
> > vp_find_vqs+0x1b/0x62 [virtio_pci]
> > virtblk_probe+0xbd/0x3d0 [virtio_blk]
> > ? sysfs_do_create_link+0xbb/0xfd
> > ? blk_done+0x0/0xa7 [virtio_blk]
> > ? add_status+0x1f/0x24
> > virtio_dev_probe+0x91/0xb0
> > driver_probe_device+0x79/0x105
> > __driver_attach+0x43/0x5f
> > bus_for_each_dev+0x3d/0x67
> > driver_attach+0x14/0x16
> > ? __driver_attach+0x0/0x5f
> > bus_add_driver+0xa2/0x1c9
> > driver_register+0x8b/0xeb
> > ? init+0x0/0x24 [virtio_blk]
> > register_virtio_driver+0x1f/0x22
> > init+0x22/0x24 [virtio_blk]
> > do_one_initcall+0x4c/0x13a
> > sys_init_module+0xa7/0x1db
> > syscall_call+0x7/0xb
> > virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> > virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
> 
> > vda: vda1 vda2
> > EXT3-fs: INFO: recovery required on readonly filesystem.
> > EXT3-fs: write access will be enabled during recovery.
> > kjournald starting. Commit interval 5 seconds
> > EXT3-fs: recovery complete.
> > EXT3-fs: mounted filesystem with writeback data mode.
> > udevd version 127 started
> > virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 28 for MSI/MSI-X
> > IRQ handler type mismatch for IRQ 1
> > current handler: i8042
> > Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
> > Call Trace:
> > __setup_irq+0x24c/0x2ac
> > request_threaded_irq+0x113/0x179
> > ? vring_interrupt+0x0/0x2f
> > vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> > ? skb_recv_done+0x0/0x36 [virtio_net]
> > vp_find_vqs+0x1b/0x62 [virtio_pci]
> > virtnet_probe+0x265/0x347 [virtio_net]
> > ? skb_recv_done+0x0/0x36 [virtio_net]
> > ? skb_xmit_done+0x0/0x1e [virtio_net]
> > ? add_status+0x1f/0x24
> > virtio_dev_probe+0x91/0xb0
> > driver_probe_device+0x79/0x105
> > __driver_attach+0x43/0x5f
> > bus_for_each_dev+0x3d/0x67
> > driver_attach+0x14/0x16
> > ? __driver_attach+0x0/0x5f
> > bus_add_driver+0xa2/0x1c9
> > driver_register+0x8b/0xeb
> > ? init+0x0/0xf [virtio_net]
> > register_virtio_driver+0x1f/0x22
> > init+0xd/0xf [virtio_net]
> > do_one_initcall+0x4c/0x13a
> > sys_init_module+0xa7/0x1db
> > syscall_call+0x7/0xb
> > virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> >
> > 3. The guest interface is up, and cat /proc/interrupts outputs:
> >
> > 24: 0 PCI-MSI-edge virtio1-config
> > 25: 2571 PCI-MSI-edge virtio1-virtqueues
> > 26: 0 PCI-MSI-edge virtio0-config
> > 27: 0 PCI-MSI-edge virtio0-virtqueues
> >
> > Thanks
> > Shirley
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
> 
> Hello Michael,
> 
> >There was a recent bugfix in qemu-kvm I pushed.
> >Could you please verify that you have
> cec75e39151e49cc90c849eab5d0d729667c9e68 ?
> 
> Yes, I cloned your qemu-kvm and kernel git.
> > I am posting the errors from /var/log/
> > messages here:
> >
> > virtio-pci 0000:00:03.0: can't find IRQ for PCI INT A; probably buggy MP
> table
> > virtio-pci 0000:00:04.0: can't find IRQ for PCI INT A; probably buggy MP
> table
> > virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> > virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
> > IRQ handler type mismatch for IRQ 1


So I think these are not related.

> > current handler: i8042
> > Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
> > Call Trace:
> > __setup_irq+0x24c/0x2ac
> > request_threaded_irq+0x113/0x179
> > ? vring_interrupt+0x0/0x2f
> > vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> > ? blk_done+0x0/0xa7 [virtio_blk]
> > vp_find_vqs+0x1b/0x62 [virtio_pci]
> > virtblk_probe+0xbd/0x3d0 [virtio_blk]
> > ? sysfs_do_create_link+0xbb/0xfd
> > ? blk_done+0x0/0xa7 [virtio_blk]
> > ? add_status+0x1f/0x24
> > virtio_dev_probe+0x91/0xb0
> > driver_probe_device+0x79/0x105
> > __driver_attach+0x43/0x5f
> > bus_for_each_dev+0x3d/0x67
> > driver_attach+0x14/0x16
> > ? __driver_attach+0x0/0x5f
> > bus_add_driver+0xa2/0x1c9
> > driver_register+0x8b/0xeb
> > ? init+0x0/0x24 [virtio_blk]
> > register_virtio_driver+0x1f/0x22
> > init+0x22/0x24 [virtio_blk]
> > do_one_initcall+0x4c/0x13a
> > sys_init_module+0xa7/0x1db
> > syscall_call+0x7/0xb
> > virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
> > virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
> 
> > vda: vda1 vda2
> > EXT3-fs: INFO: recovery required on readonly filesystem.
> > EXT3-fs: write access will be enabled during recovery.
> > kjournald starting. Commit interval 5 seconds
> > EXT3-fs: recovery complete.
> > EXT3-fs: mounted filesystem with writeback data mode.
> > udevd version 127 started
> > virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 28 for MSI/MSI-X
> > IRQ handler type mismatch for IRQ 1
> > current handler: i8042
> > Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
> > Call Trace:
> > __setup_irq+0x24c/0x2ac
> > request_threaded_irq+0x113/0x179
> > ? vring_interrupt+0x0/0x2f
> > vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
> > ? skb_recv_done+0x0/0x36 [virtio_net]
> > vp_find_vqs+0x1b/0x62 [virtio_pci]
> > virtnet_probe+0x265/0x347 [virtio_net]
> > ? skb_recv_done+0x0/0x36 [virtio_net]
> > ? skb_xmit_done+0x0/0x1e [virtio_net]
> > ? add_status+0x1f/0x24
> > virtio_dev_probe+0x91/0xb0
> > driver_probe_device+0x79/0x105
> > __driver_attach+0x43/0x5f
> > bus_for_each_dev+0x3d/0x67
> > driver_attach+0x14/0x16
> > ? __driver_attach+0x0/0x5f
> > bus_add_driver+0xa2/0x1c9
> > driver_register+0x8b/0xeb
> > ? init+0x0/0xf [virtio_net]
> > register_virtio_driver+0x1f/0x22
> > init+0xd/0xf [virtio_net]
> > do_one_initcall+0x4c/0x13a
> > sys_init_module+0xa7/0x1db
> > syscall_call+0x7/0xb
> > virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X
> > virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X
> >
> > 3. The guest interface is up, and cat /proc/interrupts outputs:
> >
> > 24: 0 PCI-MSI-edge virtio1-config
> > 25: 2571 PCI-MSI-edge virtio1-virtqueues
> > 26: 0 PCI-MSI-edge virtio0-config
> > 27: 0 PCI-MSI-edge virtio0-virtqueues

Aha. I notice that we failed to allocate 3 vectors, and have fallen back
on 2 vectors here.  This also should work, but is not the configuration
that I am testing, and will be slower anyway.  Why are there 2 virtio
devices here? Is the second one for storage? Can you try switching to
another hard drive model please?


> > Thanks
> > Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
> 
> Hello Michael,
> 
> >There was a recent bugfix in qemu-kvm I pushed.
> >Could you please verify that you have
> cec75e39151e49cc90c849eab5d0d729667c9e68 ?
> 
> Yes, I cloned your qemu-kvm and kernel git.


It seems that the errors you observe are a result of changes
in guest virtio. I work to fix these, for now please
revert this commit f68d24082e22ccee3077d11aeb6dc5354f0ca7f1
in guest kernel.

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 02:34:56PM +0200, Michael S. Tsirkin wrote:
> On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
> > 
> > Hello Michael,
> > 
> > >There was a recent bugfix in qemu-kvm I pushed.
> > >Could you please verify that you have
> > cec75e39151e49cc90c849eab5d0d729667c9e68 ?
> > 
> > Yes, I cloned your qemu-kvm and kernel git.
> 
> 
> It seems that the errors you observe are a result of changes
> in guest virtio. I work to fix these, for now please
> revert this commit f68d24082e22ccee3077d11aeb6dc5354f0ca7f1
> in guest kernel.

OK, I sent a patch that should fix the errors for you.
Could you please confirm, preferably on-list, whether
the patch makes the errors go away for you with
userspace virtio?

However, as I said previously, it's good to fix them but I think
they are unrelated to the fact that vhost does not work
for you.

Shirley, if you have the time, please send details
on your setup and observations on tcpdump output
in both guest and host with vhost enabled.

Thanks!

-- 
MST

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 15:13 +0200, Michael S. Tsirkin wrote:
> OK, I sent a patch that should fix the errors for you.
> Could you please confirm, preferably on-list, whether
> the patch makes the errors go away for you with
> userspace virtio?

Confirmed, your patch has fixed irq handler mismatch errors.

> However, as I said previously, it's good to fix them but I think
> they are unrelated to the fact that vhost does not work
> for you.

Yes, agreed. One observation is when I enable PCI MSI in guest kernel, I
found that even without vhost supportin host kernel the network doesn't
work either. So I think this is nothing related to vhost. I need to find
why PCI MSI doesn't work for me.

> Shirley, if you have the time, please send details
> on your setup and observations on tcpdump output
> in both guest and host with vhost enabled.

Yes, here are details:

HW and distros:
T61 laptop, cpu is 
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
stepping	: 11
cpu MHz		: 2393.902
cache size	: 4096 KB

Guest installed Fedora 10, Host installed Fedora 10 and updated to
Fedora11, both doesn't work with guest kernel MSI enabled w/i, w/o vhost
support.


Source trees:
clone your qemu-kvm and vhost git. host kernel is vhost git, guest
kernel 2.6.32-rc5

Compile option:
qemu: ./configure (default target-list = x86_64-softmmu)
guest kernel: PCI MSI, virio enabled
host kernel: vhost, kvm, evenfd all enabled

I tried tap, raw, vnet0, eth0, here are some qemu commandline example:

mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
-s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
1 -monitor pty -drive
file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
raw,ifname=eth0 -net nic,macaddr=54:52:00:46:26:80,model=virtio,vhost

mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
-s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
1 -monitor pty -drive
file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
raw,ifname=vnet0 -net nic,model,macaddr=54:52:00:46:26:80=virtio,vhost 

Please let me know what I can do next. I am eager to test the vhost
performance for both latency and bw.

tcpdump output:
When I ping from guest to host, I didn't see any packets from any
interfaces, not even an arp request.

When I ping from host to guest, guest interface didn't see any packets
either, not even an arp request.

Thanks for your help!
Shirley

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 10:23 -0700, Shirley Ma wrote:
> Yes, agreed. One observation is when I enable PCI MSI in guest kernel,
> I
> found that even without vhost supportin host kernel the network
> doesn't
> work either. So I think this is nothing related to vhost. I need to
> find
> why PCI MSI doesn't work for me.

Let me clarify this, here is the table:

1. upstream qemu + guest kernel MSI support --- doesn't work
2. upstream qemu + guest kernel no MSI support --- work
3. mst qemu git tree + guest kernel MSI support - vhost --- work
4. mst qemu git tree + guest kernel MSI support + vhost -- doesn't work

I hope it's clear.

thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:32:55AM -0700, Shirley Ma wrote:
> On Thu, 2009-10-22 at 10:23 -0700, Shirley Ma wrote:
> > Yes, agreed. One observation is when I enable PCI MSI in guest kernel,
> > I
> > found that even without vhost supportin host kernel the network
> > doesn't
> > work either. So I think this is nothing related to vhost. I need to
> > find
> > why PCI MSI doesn't work for me.
> 
> Let me clarify this, here is the table:
> 
> 1. upstream qemu + guest kernel MSI support --- doesn't work
> 2. upstream qemu + guest kernel no MSI support --- work
> 3. mst qemu git tree + guest kernel MSI support - vhost --- work
> 4. mst qemu git tree + guest kernel MSI support + vhost -- doesn't work

Upstream is Avi's qemu-kvm.git?
So, for a moment taking vhost out of the equation, it seems that MSI was
broken in Avi's tree again, after I forked my tree?


> 
> I hope it's clear.
> 
> thanks
> Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:23:44AM -0700, Shirley Ma wrote:
> On Thu, 2009-10-22 at 15:13 +0200, Michael S. Tsirkin wrote:
> > OK, I sent a patch that should fix the errors for you.
> > Could you please confirm, preferably on-list, whether
> > the patch makes the errors go away for you with
> > userspace virtio?
> 
> Confirmed, your patch has fixed irq handler mismatch errors.
> 
> > However, as I said previously, it's good to fix them but I think
> > they are unrelated to the fact that vhost does not work
> > for you.
> 
> Yes, agreed. One observation is when I enable PCI MSI in guest kernel, I
> found that even without vhost supportin host kernel the network doesn't
> work either.

Yes, but that's on Avi's qemu-kvm.git, so probably merge problems again.

> So I think this is nothing related to vhost. I need to find
> why PCI MSI doesn't work for me.

Probably just bisect until you find the bad commit.

> > Shirley, if you have the time, please send details
> > on your setup and observations on tcpdump output
> > in both guest and host with vhost enabled.
> 
> Yes, here are details:
> 
> HW and distros:
> T61 laptop, cpu is 
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 15
> model name	: Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
> stepping	: 11
> cpu MHz		: 2393.902
> cache size	: 4096 KB
> 
> Guest installed Fedora 10, Host installed Fedora 10 and updated to
> Fedora11, both doesn't work with guest kernel MSI enabled w/i, w/o vhost
> support.

So, IIUC, my tree with MSI without vhost works?

> 
> Source trees:
> clone your qemu-kvm and vhost git. host kernel is vhost git, guest
> kernel 2.6.32-rc5
> 
> Compile option:
> qemu: ./configure (default target-list = x86_64-softmmu)
> guest kernel: PCI MSI, virio enabled
> host kernel: vhost, kvm, evenfd all enabled
> 
> I tried tap, raw, vnet0, eth0,

what are vnet0 and eth0?
Do you have a bridge in host?
What do brctl show and ifconfig -a show in host?

> here are some qemu commandline example:
> 
> mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
> -s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
> 1 -monitor pty -drive
> file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
> raw,ifname=eth0 -net nic,macaddr=54:52:00:46:26:80,model=virtio,vhost
> 
> mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
> -s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
> 1 -monitor pty -drive
> file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
> raw,ifname=vnet0 -net nic,model,macaddr=54:52:00:46:26:80=virtio,vhost 
> 
> Please let me know what I can do next. I am eager to test the vhost
> performance for both latency and bw.
> tcpdump output:
> When I ping from guest to host, I didn't see any packets from any
> interfaces, not even an arp request.
> 
> When I ping from host to guest, guest interface didn't see any packets
> either, not even an arp request.

Did you try tcpdump on the tap interface in host?

> Thanks for your help!
> Shirley
> 
> 
> 
> 

Possibly we'll have to debug this in vhost in host kernel.
I would debug this directly, it's just that my setup is somehow
different and I do not see this issue, otherwise I would not
waste your time.

Can we add some printks?
handle_tx has this at the top:

        if (!sock || !sock_writeable(sock->sk))
                return;

Could you please add

 printk(KERN_ERR "%s: !sock = %d, !sock || !sock_writeable(sock->sk) =
%d",
        __func__, !sock , !sock || !sock_writeable(sock->sk));

*Before* these checks?
Then make modules modules_install
rmmod vhost_net
insmod vhost_net
and re-run.
If you want me to send a patch that does this, let me know please.

Thanks!
-- 
MST

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 19:36 +0200, Michael S. Tsirkin wrote:
> Upstream is Avi's qemu-kvm.git?
> So, for a moment taking vhost out of the equation, it seems that MSI
> was
> broken in Avi's tree again, after I forked my tree?

The upper stream qemu git tree never worked for me w/i MSI, the boot
hung on booting. I tried to debug this, hasn't found any hint. But
Sridhar has same configure, it works for him.

But with your git tree, guest kernel works without vhost; it doesn't
work with vhost. So the vhost might have some issues.

Thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:44:29AM -0700, Shirley Ma wrote:
> On Thu, 2009-10-22 at 19:36 +0200, Michael S. Tsirkin wrote:
> > Upstream is Avi's qemu-kvm.git?
> > So, for a moment taking vhost out of the equation, it seems that MSI
> > was
> > broken in Avi's tree again, after I forked my tree?
> 
> The upper stream qemu git tree never worked for me w/i MSI, the boot
> hung on booting. I tried to debug this, hasn't found any hint. But
> Sridhar has same configure, it works for him.
> 
> But with your git tree, guest kernel works without vhost;

What happens if you reset my tree to commit
47e465f031fc43c53ea8f08fa55cc3482c6435c8?

> it doesn't work with vhost.
> So the vhost might have some issues.

Looks like there are 2 issues:
- upstream issue
- vhost issue
Let's try to debug both.

> Thanks
> Shirley

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 19:47 +0200, Michael S. Tsirkin wrote:
> What happens if you reset my tree to commit
> 47e465f031fc43c53ea8f08fa55cc3482c6435c8?

I am going to clean up my upperstream git tree and retest first. Then I
will try back up this commit.

> Looks like there are 2 issues:
> - upstream issue
> - vhost issue
> Let's try to debug both.

Sounds good.

Thanks
Shirley

Re: vhost-net patches

[Sridhar Samudrala]

On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:

> 
> Possibly we'll have to debug this in vhost in host kernel.
> I would debug this directly, it's just that my setup is somehow
> different and I do not see this issue, otherwise I would not
> waste your time.
> 
> Can we add some printks?
> handle_tx has this at the top:
> 
>         if (!sock || !sock_writeable(sock->sk))
>                 return;

I added some debug printks in handle_rx and handle_tx
get_user() calls are failing with EFAULT.

Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_rx: net:f5494800
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to enable notification: -14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to disable notification: -14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:51:48 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:48 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:51:49 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:49 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:51:53 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:53 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:52:03 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:03 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:52:22 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:22 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:52:23 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:23 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:53:17 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:53:17 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
Oct 21 10:56:56 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:56:56 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14

Thanks
Sridhar
> Could you please add
> 
>  printk(KERN_ERR "%s: !sock = %d, !sock || !sock_writeable(sock->sk) =
> %d",
>         __func__, !sock , !sock || !sock_writeable(sock->sk));
> 
> *Before* these checks?
> Then make modules modules_install
> rmmod vhost_net
> insmod vhost_net
> and re-run.
> If you want me to send a patch that does this, let me know please.
> 
> Thanks!

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 10:56 -0700, Shirley Ma wrote:
> On Thu, 2009-10-22 at 19:47 +0200, Michael S. Tsirkin wrote:
> > What happens if you reset my tree to commit
> > 47e465f031fc43c53ea8f08fa55cc3482c6435c8?
> 
> I am going to clean up my upperstream git tree and retest first. Then
> I
> will try back up this commit.

Cleaned my local qemu upperstream git tree and guest kernel works with
MSI enabled.

So only vhost does seem to have some issues in my configuration. 

Thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 11:00:20AM -0700, Sridhar Samudrala wrote:
> On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:
> 
> > 
> > Possibly we'll have to debug this in vhost in host kernel.
> > I would debug this directly, it's just that my setup is somehow
> > different and I do not see this issue, otherwise I would not
> > waste your time.
> > 
> > Can we add some printks?
> > handle_tx has this at the top:
> > 
> >         if (!sock || !sock_writeable(sock->sk))
> >                 return;
> 
> I added some debug printks in handle_rx and handle_tx
> get_user() calls are failing with EFAULT.
> 
> Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_rx: net:f5494800
> Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to enable notification: -14
> Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14
> Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to disable notification: -14
> Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx at 00000002 ret:-14

Aha, we are getting somewhere!
Thanks very much for looking into this.  This suggests strongly that
available index pointer is not set up correctly in vhost.
I would like to verify this and understand how it came to be
incorrect.

Sridhar, Shirley, could you please try this:
        - make sure to enable CONFIG_DYNAMIC_DEBUG=y in .config
          (if it's not there, you'll have to rebuild kernel, or
           hack drivers/vhost/vhost.h to use pr_err instead of
           pr_debug).
        - echo 'module vhost_net +p' > /sys/kernel/debug/dynamic_debug/control

the above will make vhost print address which we tried to access, on error,
and they will apear in the output of the dmesg command.

In addition, please apply the following debugging patch to qemu.
this will make it output data about available index on startup.
Thanks!


diff --git a/hw/vhost_net.c b/hw/vhost_net.c
index bc179ab..d651504 100644
--- a/hw/vhost_net.c
+++ b/hw/vhost_net.c
@@ -66,6 +66,11 @@ static int vhost_virtqueue_init(struct vhost_dev *dev,
 	r = ioctl(dev->control, VHOST_SET_VRING_AVAIL, &addr);
 	if (r < 0)
 		return -errno;
+
+	fprintf(stderr, "VQ %d, avail va %p(=%llx), pa %llx, num %x size %llx\n",
+		idx, vq->avail, (unsigned long long)addr.user_addr,
+		(unsigned long long)q->vring.avail, (unsigned)q->vring.num, s);
+
 	s = l = offsetof(struct vring_used, ring) +
 		sizeof(struct vring_used_elem) * q->vring.num;
 	vq->used = cpu_physical_memory_map(q->vring.used, &l, 1);

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 11:00:20AM -0700, Sridhar Samudrala wrote:
> On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:
> 
> > 
> > Possibly we'll have to debug this in vhost in host kernel.
> > I would debug this directly, it's just that my setup is somehow
> > different and I do not see this issue, otherwise I would not
> > waste your time.
> > 
> > Can we add some printks?
> > handle_tx has this at the top:
> > 
> >         if (!sock || !sock_writeable(sock->sk))
> >                 return;
> 
> I added some debug printks in handle_rx and handle_tx
> get_user() calls are failing with EFAULT.

Sridhar, Shirley,
Could you please test the following patch?
It should fix a bug on 32 bit hosts - is this what you have?


diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 5b57697..9d856e9 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -286,11 +286,11 @@ static long vhost_set_vring(struct vhost_dev *d, int ioctl, void __user *argp)
 			r = -EOPNOTSUPP;
 			break;
 		}
-		if ((u64)(long)a.user_addr != a.user_addr) {
+		if ((u64)(unsigned long)a.user_addr != a.user_addr) {
 			r = -EFAULT;
 			break;
 		}
-		vq->desc = (void __user *)(long)a.user_addr;
+		vq->desc = (void __user *)(unsigned long)a.user_addr;
 		break;
 	case VHOST_SET_VRING_AVAIL:
 		r = copy_from_user(&a, argp, sizeof a);
@@ -300,11 +300,11 @@ static long vhost_set_vring(struct vhost_dev *d, int ioctl, void __user *argp)
 			r = -EOPNOTSUPP;
 			break;
 		}
-		if ((u64)(long)a.user_addr != a.user_addr) {
+		if ((u64)(unsigned long)a.user_addr != a.user_addr) {
 			r = -EFAULT;
 			break;
 		}
-		vq->avail = (void __user *)(long)a.user_addr;
+		vq->avail = (void __user *)(unsigned long)a.user_addr;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
 		break;
@@ -316,11 +316,11 @@ static long vhost_set_vring(struct vhost_dev *d, int ioctl, void __user *argp)
 			r = -EOPNOTSUPP;
 			break;
 		}
-		if ((u64)(long)a.user_addr != a.user_addr) {
+		if ((u64)(unsigned long)a.user_addr != a.user_addr) {
 			r = -EFAULT;
 			break;
 		}
-		vq->used = (void __user *)(long)a.user_addr;
+		vq->used = (void __user *)(unsigned long)a.user_addr;
 		r = init_used(vq);
 		if (r)
 			break;
@@ -389,7 +389,7 @@ static long vhost_set_vring(struct vhost_dev *d, int ioctl, void __user *argp)
 
 	if (pollstop && vq->handle_kick)
 		vhost_poll_flush(&vq->poll);
-	return 0;
+	return r;
 }
 
 long vhost_dev_ioctl(struct vhost_dev *d, unsigned int ioctl, unsigned long arg)

Re: vhost-net patches

[Shirley Ma]

On Fri, 2009-10-23 at 13:04 +0200, Michael S. Tsirkin wrote:
> Sridhar, Shirley,
> Could you please test the following patch?
> It should fix a bug on 32 bit hosts - is this what you have?

Yes, it's 32 bit host. I checked out your recent git tree. Looks like
the patch is already there, but vhost doesn't work. 

I will apply the debug one you sent out in previous email.

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Tested raw packet, it didn't work; switching to tap device, it is
working. Qemu command is:

x86_64-softmmu/qemu-system-x86_64 -s /home/xma/images/fedora10-2-vm -m
512 -drive file=/home/xma/images/fedora10-2-vm,if=virtio,index=0,boot=on
-net tap,ifname=vnet0,script=no,downscript=no -net
nic,macaddr=54:52:00:35:e3:73,model=virtio,vhost -no-acpi

Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Some initial vhost test netperf results on my T61 laptop from the
working tap device are here, latency has been significant decreased, but
throughput from guest to host has huge regression. I also hit guest
skb_xmit panic.

netperf TCP_STREAM, default setup, 60 secs run

guest->host drops from 3XXXMb/s to 1XXXMb/s (regression)
host->guest increases from 3XXXMb/s to 4XXXXMb/s 

TCP_RR, 60 secs run (very impressive)

guest->host trans/s increases from 2XXX/s to 13XXX/s
host->guest trans/s increases from 2XXX/s to 13XXX/s

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Some update,

On Fri, 2009-10-23 at 08:12 -0700, Shirley Ma wrote:
> Tested raw packet, it didn't work; 

Tested option -net raw,ifname=eth0, attached to a real device, raw works
to remote node. I was expecting raw worked to local host. 

Does this option "-net raw,ifname=vnet0" support?

Thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 12:24:15PM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> Some update,
> 
> On Fri, 2009-10-23 at 08:12 -0700, Shirley Ma wrote:
> > Tested raw packet, it didn't work; 
> 
> Tested option -net raw,ifname=eth0, attached to a real device, raw works
> to remote node. I was expecting raw worked to local host. 
> 
> Does this option "-net raw,ifname=vnet0" support?

What is vnet0?

> Thanks
> Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> Some initial vhost test netperf results on my T61 laptop from the
> working tap device are here, latency has been significant decreased, but
> throughput from guest to host has huge regression.

Could you please try adding ,sndbuf=0 to tap options, and re-testing?
This would make the -net option look like something like this:
-net tap,vlan=1,ifname=msttap0,script=/home/mst/ifup,downscript=no,sndbuf=0

> I also hit guest skb_xmit panic.

I have not seen panics in a while.
Could you provide more information please?
Is the panic reproducible?
Do you see one of these messages:
                 id %u out of range
                 id %u is not a head!


> 
> netperf TCP_STREAM, default setup, 60 secs run
> 
> guest->host drops from 3XXXMb/s to 1XXXMb/s (regression)
> host->guest increases from 3XXXMb/s to 4XXXXMb/s 
> 
> TCP_RR, 60 secs run (very impressive)
> 
> guest->host trans/s increases from 2XXX/s to 13XXX/s
> host->guest trans/s increases from 2XXX/s to 13XXX/s
> 
> Thanks
> Shirley
> 

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
> I also hit guest skb_xmit panic.

OK, I have fixed a couple of reasons for panic. Will push and
look at host to guest performance soon.

-- 
MST

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
> I also hit guest skb_xmit panic.

If these are the same panics I have seen myself,
they are probably fixed with recent virtio patches
I sent to Rusty. I put them on my vhost.git tree to make
it easier for you to test.
If you see any more crashes, please holler, preferably
with a backtrace.

-- 
MST

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> Some initial vhost test netperf results on my T61 laptop from the
> working tap device are here, latency has been significant decreased, but
> throughput from guest to host has huge regression. I also hit guest
> skb_xmit panic.
> 
> netperf TCP_STREAM, default setup, 60 secs run
> 
> guest->host drops from 3XXXMb/s to 1XXXMb/s (regression)
> host->guest increases from 3XXXMb/s to 4XXXXMb/s 
> 
> TCP_RR, 60 secs run (very impressive)
> 
> guest->host trans/s increases from 2XXX/s to 13XXX/s
> host->guest trans/s increases from 2XXX/s to 13XXX/s
> 
> Thanks
> Shirley

Shirley, could you please test the following patch?
It is surprising to me that it should improve
performance, but seems to do this in my setup.
Please comment.


diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 30708c6..67bfc08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -775,7 +775,7 @@ void vhost_no_notify(struct vhost_virtqueue *vq)
 
 int vhost_init(void)
 {
-	vhost_workqueue = create_workqueue("vhost");
+	vhost_workqueue = create_singlethread_workqueue("vhost");
 	if (!vhost_workqueue)
 		return -ENOMEM;
 	return 0;
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index a140dad..49026bb 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -106,10 +106,14 @@ static void handle_tx(struct vhost_net *net)
 		.msg_flags = MSG_DONTWAIT,
 	};
 	size_t len, total_len = 0;
-	int err;
+	int err, wmem;
 	size_t hdr_size;
 	struct socket *sock = rcu_dereference(vq->private_data);
-	if (!sock || !sock_writeable(sock->sk))
+	if (!sock)
+		return;
+
+	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
+	if (wmem >= sock->sk->sk_sndbuf)
 		return;
 
 	use_mm(net->dev.mm);

Re: vhost-net patches

[Shirley Ma]

Hello Miachel,

On Mon, 2009-10-26 at 22:05 +0200, Michael S. Tsirkin wrote:
> Shirley, could you please test the following patch?

With this patch, the performance has gained from 1xxx to 2xxx Mb/s,
still has some performance gap compared to without vhost. It was
3xxxMb/s before from guest to host on my set up.

Looks like your git tree virtio_net has fixed the skb_xmit panic I have
seen before as well, good news.

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

Pulled your git tree, didn't see the panic.

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

On Sun, 2009-10-25 at 11:11 +0200, Michael S. Tsirkin wrote:
> What is vnet0?

That's a tap interface. I am binding raw socket to a tap interface and
it doesn't work. Does it support?

Thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 26, 2009 at 02:37:47PM -0700, Shirley Ma wrote:
> On Sun, 2009-10-25 at 11:11 +0200, Michael S. Tsirkin wrote:
> > What is vnet0?
> 
> That's a tap interface. I am binding raw socket to a tap interface and
> it doesn't work. Does it support?

Yes but you need to make host send packets out to tap as well,
somehow. One way to do this is to assign IP address in
a separate subnet to tap in host and to eth device in guest.

> Thanks
> Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 26, 2009 at 02:34:49PM -0700, Shirley Ma wrote:
> Hello Miachel,
> 
> On Mon, 2009-10-26 at 22:05 +0200, Michael S. Tsirkin wrote:
> > Shirley, could you please test the following patch?
> 
> With this patch, the performance has gained from 1xxx to 2xxx Mb/s,
> still has some performance gap compared to without vhost. It was
> 3xxxMb/s before from guest to host on my set up.

At some point my guest had a runaway nash-hotplug process
consuming 100% CPU. Could you please verify this
does not happen to you?

> Looks like your git tree virtio_net has fixed the skb_xmit panic I have
> seen before as well, good news.
> 
> Thanks
> Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

On Tue, 2009-10-27 at 08:43 +0200, Michael S. Tsirkin wrote:
> At some point my guest had a runaway nash-hotplug process
> consuming 100% CPU. Could you please verify this
> does not happen to you?

What I have found that the start_xmit stopped and restarted too often.
There is no vring descriptor available for adding the new buf. The buf
release is not able to keep up after vhost patch?

I saw lots of "Unexpected full queue" from dmesg. I haven't had time to
debug it yet. You might have some idea here?

        /* This can happen with OOM and indirect buffers. */
        if (unlikely(capacity < 0)) {
                netif_stop_queue(dev);
                dev_warn(&dev->dev, "Unexpected full queue\n");
                if (unlikely(!vi->svq->vq_ops->enable_cb(vi->svq))) {
                        vi->svq->vq_ops->disable_cb(vi->svq);
                        netif_start_queue(dev);
                        goto again;
                }
                return NETDEV_TX_BUSY;
        }

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

On Tue, 2009-10-27 at 08:38 +0200, Michael S. Tsirkin wrote:
> Yes but you need to make host send packets out to tap as well,
> somehow. One way to do this is to assign IP address in
> a separate subnet to tap in host and to eth device in guest.

Thanks for the hint, I will make a try.

Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 07:46:59AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> On Tue, 2009-10-27 at 08:43 +0200, Michael S. Tsirkin wrote:
> > At some point my guest had a runaway nash-hotplug process
> > consuming 100% CPU. Could you please verify this
> > does not happen to you?
> 
> What I have found that the start_xmit stopped and restarted too often.
> There is no vring descriptor available for adding the new buf. The buf
> release is not able to keep up after vhost patch?

Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
Is there a chance you are tight on guest memory for some reason?
with vhost, virtio does currently consume a bit more memory than
with userspace backend.

> I saw lots of "Unexpected full queue" from dmesg. I haven't had time to
> debug it yet. You might have some idea here?
> 
>         /* This can happen with OOM and indirect buffers. */
>         if (unlikely(capacity < 0)) {
>                 netif_stop_queue(dev);
>                 dev_warn(&dev->dev, "Unexpected full queue\n");
>                 if (unlikely(!vi->svq->vq_ops->enable_cb(vi->svq))) {
>                         vi->svq->vq_ops->disable_cb(vi->svq);
>                         netif_start_queue(dev);
>                         goto again;
>                 }
>                 return NETDEV_TX_BUSY;
>         }
> 
> Thanks
> Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
> Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
> Is there a chance you are tight on guest memory for some reason?
> with vhost, virtio does currently consume a bit more memory than
> with userspace backend.

I did see memory leak on host every time after exiting guest. I don't
know where. Do you see it?

Anyway after I reboot host and restart guest with large memory
allocation, I do see performance improves to 3xxxMb/s, and occasionally
reaches 40xxMb/s. But "queue full" still exists, I can avoid the problem
by increasing send queue size from qemu.

I will apply deferring skb allocation patch on guest to see any
performance gain after your vhost patch.

Thanks
Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
> > Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
> > Is there a chance you are tight on guest memory for some reason?
> > with vhost, virtio does currently consume a bit more memory than
> > with userspace backend.
> 
> I did see memory leak on host every time after exiting guest. I don't
> know where. Do you see it?

I didn't notice. I'll check this.

> Anyway after I reboot host and restart guest with large memory
> allocation,

How large is large here? I usually allocate 1G.

> I do see performance improves to 3xxxMb/s, and occasionally
> reaches 40xxMb/s.

This is same as userspace, isn't it?

> But "queue full" still exists, I can avoid the problem
> by increasing send queue size from qemu.

And what performance do you get then?

> I will apply deferring skb allocation patch on guest to see any
> performance gain after your vhost patch.
> 
> Thanks
> Shirley

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
> > Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
> > Is there a chance you are tight on guest memory for some reason?
> > with vhost, virtio does currently consume a bit more memory than
> > with userspace backend.
> 
> I did see memory leak on host every time after exiting guest. I don't
> know where. Do you see it?
> 
> Anyway after I reboot host and restart guest with large memory
> allocation, I do see performance improves to 3xxxMb/s, and occasionally
> reaches 40xxMb/s. But "queue full" still exists, I can avoid the problem
> by increasing send queue size from qemu.

Here's another hack to try. It will break raw sockets,
but just as a test:

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index a140dad..e830b30 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -106,22 +106,41 @@ static void handle_tx(struct vhost_net *net)
 		.msg_flags = MSG_DONTWAIT,
 	};
 	size_t len, total_len = 0;
-	int err;
+	int err, wmem;
 	size_t hdr_size;
 	struct socket *sock = rcu_dereference(vq->private_data);
-	if (!sock || !sock_writeable(sock->sk))
+	if (!sock)
+		return;
+
+	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
+	if (wmem >= sock->sk->sk_sndbuf)
 		return;
 
 	use_mm(net->dev.mm);
 	mutex_lock(&vq->mutex);
-	tx_poll_stop(net);
+	vhost_no_notify(vq);
+	
+	if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
+		//tx_poll_start(net);
+	} else {
+		//tx_poll_stop(net);
+	}
 	hdr_size = vq->hdr_size;
 
 	for (;;) {
 		head = vhost_get_vq_desc(&net->dev, vq, vq->iov, &out, &in);
 		/* Nothing new?  Wait for eventfd to tell us they refilled. */
-		if (head == vq->num)
+		if (head == vq->num) {
+			wmem = atomic_read(&sock->sk->sk_wmem_alloc);
+			if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
+				set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
+				break;
+			}
+			if (vhost_notify(vq)) {
+				continue;
+			}
 			break;
+		}
 		if (in) {
 			vq_err(vq, "Unexpected descriptor format for TX: "
 			       "out %d, int %d\n", out, in);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 30708c6..67bfc08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -775,7 +775,7 @@ void vhost_no_notify(struct vhost_virtqueue *vq)
 
 int vhost_init(void)
 {
-	vhost_workqueue = create_workqueue("vhost");
+	vhost_workqueue = create_singlethread_workqueue("vhost");
 	if (!vhost_workqueue)
 		return -ENOMEM;
 	return 0;

Re: vhost-net patches

[Shirley Ma]

On Tue, 2009-10-27 at 22:58 +0200, Michael S. Tsirkin wrote:
> How large is large here? I usually allocate 1G.

I used to have 512, for this run I allocated 1G.

> > I do see performance improves to 3xxxMb/s, and occasionally
> > reaches 40xxMb/s.
> 
> This is same as userspace, isn't it?

A little bit better than userspace.

> > But "queue full" still exists, I can avoid the problem
> > by increasing send queue size from qemu.
> 
> And what performance do you get then?

5-10% better than userspace.

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

On Wed, 2009-10-28 at 17:39 +0200, Michael S. Tsirkin wrote:
> Here's another hack to try. It will break raw sockets,
> but just as a test:

This patch looks better than previous one for guest to host TCP_STREAM
performance. The transmission queue full still exists, but TCP_STREAM
results is 43xxMb/s (userspace is about 3500Mb/s). When I increase the
transmission queue size to 1K, The performance burst up to 53xxMb/s. But
some runs I still hit several transmission queue full in 60 secs run
even with 1K queue size.

Thanks
Shirley

kvm memory leak (Was Re: vhost-net patches)

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
> > Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
> > Is there a chance you are tight on guest memory for some reason?
> > with vhost, virtio does currently consume a bit more memory than
> > with userspace backend.
> 
> I did see memory leak on host every time after exiting guest. I don't
> know where. Do you see it?

I don't see a leak specific to vhost. However, even without vhost,
starting up guest increases kmalloc-128 counter in /proc/slabinfo
and this fails to go down when qemu is killed.

Anyone else seeing this?

-- 
MST

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 28, 2009 at 09:45:37AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> On Wed, 2009-10-28 at 17:39 +0200, Michael S. Tsirkin wrote:
> > Here's another hack to try. It will break raw sockets,
> > but just as a test:
> 
> This patch looks better than previous one for guest to host TCP_STREAM
> performance. The transmission queue full still exists, but TCP_STREAM
> results is 43xxMb/s (userspace is about 3500Mb/s). When I increase the
> transmission queue size to 1K,
> The performance burst up to 53xxMb/s. But
> some runs I still hit several transmission queue full in 60 secs run
> even with 1K queue size.

what exactly do you mean by transmission queue size?
tx_queue_len?
I think what should help with transmission queue full is
actually sndbuf parameter for tap in qemu.

> 
> Thanks
> Shirley

Re: vhost-net patches

[Arnd Bergmann]

On Monday 26 October 2009, Shirley Ma wrote:
> On Sun, 2009-10-25 at 11:11 +0200, Michael S. Tsirkin wrote:
> > What is vnet0?
> 
> That's a tap interface. I am binding raw socket to a tap interface and
> it doesn't work. Does it support?


Is the tap device connected to a bridge as you'd normally do
with qemu? That won't work because then the data you send to
the socket will be queued at the /dev/tun chardev.

You can probably connect it like this:

qemu - vhost_net - vnet0 == /dev/tun - qemu

To connect two guests.

I've also used a bidirectional pipe before, to connect two tap
interfaces to each other. However, if you want to connect to
a bridge, the easier interface would be to use a veth pair,
with one end on the bridge and the other end used for the packet
socket.

	Arnd <><

Re: vhost-net patches

[Shirley Ma]

Hello Miachel,

On Wed, 2009-10-28 at 18:53 +0200, Michael S. Tsirkin wrote:
> what exactly do you mean by transmission queue size?
> tx_queue_len?
> I think what should help with transmission queue full is
> actually sndbuf parameter for tap in qemu.

I didn't see my email out, I resend the response here.
What I meant is the number of descriptors in transmission queue. To be
more specific, it is 256->1024 in hw/virtio-net.c file

   n->tx_vq = virtio_add_queue(&n->vdev, 256, virtio_net_handle_tx);

I had a patch to have trans/recv descriptors configurable rather than
fixed before. But the test results doens't show any difference for both
TCP and UDP, so I didn't submit the patch. I think it's better to make
it configurable based on how much memory virtio-net can be used. 

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Arnd,

On Wed, 2009-10-28 at 18:46 +0100, Arnd Bergmann wrote:
> You can probably connect it like this:
> 
> qemu - vhost_net - vnet0 == /dev/tun - qemu
> 
> To connect two guests.
> 
> I've also used a bidirectional pipe before, to connect two tap
> interfaces to each other. However, if you want to connect to
> a bridge, the easier interface would be to use a veth pair,
> with one end on the bridge and the other end used for the packet
> socket.

Thanks for the tip, I will make a try.

Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

When I am testing deferring skb allocation patch, I found this problem.
Simply removing and reloading guest virtio_net module would cause guest
exit with errors. It is easy to reproduce it:

[root@localhost ~]# rmmod virtio_net

[root@localhost ~]# modprobe virtio_net

vhost_net_init returned -17

Any idea?

Thanks
Shirley

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

I am able to get 63xxMb/s throughput with 10% less cpu utilization when
I apply deferring skb patch on top of your most recent vhost patch. The
userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 

After applying your recent vhost patch, it goes up to 53xxMb/s. Now it
can reach 63xxMb/s, pretty good news. (this test with transmission
descriptors = 1K)

netperf output with recent vhost patch:

[root@localhost ~]# netperf -H 192.168.122.1 -c -C -l60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.1
(192.168.122.1) port 0 AF_INET
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
us/KB

 87380  16384  16384    60.03      5325.82   99.58    83.33    1.532
2.564

netperf output with vhost + deferring skb allocation patch

[root@localhost linux-2.6.32-rc5]# netperf -H 192.168.122.1 -c -C -l60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.1
(192.168.122.1) port 0 AF_INET
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
us/KB

 87380  16384  16384    60.02      6332.38   99.33    76.16    1.285
1.970

Thanks
Shirley

Re: kvm memory leak (Was Re: vhost-net patches)

[Avi Kivity]

On 10/28/2009 06:46 PM, Michael S. Tsirkin wrote:
> On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
>    
>> Hello Michael,
>>
>> On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
>>      
>>> Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
>>> Is there a chance you are tight on guest memory for some reason?
>>> with vhost, virtio does currently consume a bit more memory than
>>> with userspace backend.
>>>        
>> I did see memory leak on host every time after exiting guest. I don't
>> know where. Do you see it?
>>      
> I don't see a leak specific to vhost. However, even without vhost,
> starting up guest increases kmalloc-128 counter in /proc/slabinfo
> and this fails to go down when qemu is killed.
>
> Anyone else seeing this?
>    

I'm seeing it, but starting a new guest seems to reclaim this memory.  
I'll enable debug and see what's the cause.

-- 
error compiling committee.c: too many arguments to function

Re: kvm memory leak (Was Re: vhost-net patches)

[Avi Kivity]

On 10/29/2009 02:21 PM, Avi Kivity wrote:
> On 10/28/2009 06:46 PM, Michael S. Tsirkin wrote:
>> On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
>>> Hello Michael,
>>>
>>> On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
>>>> Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
>>>> Is there a chance you are tight on guest memory for some reason?
>>>> with vhost, virtio does currently consume a bit more memory than
>>>> with userspace backend.
>>> I did see memory leak on host every time after exiting guest. I don't
>>> know where. Do you see it?
>> I don't see a leak specific to vhost. However, even without vhost,
>> starting up guest increases kmalloc-128 counter in /proc/slabinfo
>> and this fails to go down when qemu is killed.
>>
>> Anyone else seeing this?
>
> I'm seeing it, but starting a new guest seems to reclaim this memory.  
> I'll enable debug and see what's the cause.
>

kmemleak doesn't see any kvm memory leaks.

-- 
error compiling committee.c: too many arguments to function

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 29, 2009 at 12:11:33AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> I am able to get 63xxMb/s throughput with 10% less cpu utilization when
> I apply deferring skb patch on top of your most recent vhost patch. The
> userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 
> 
> After applying your recent vhost patch, it goes up to 53xxMb/s.

Thanks for the report!  I have pushed this patch (after cleaning it up
and fixing raw sockets), and updated the numbers in the wiki here
http://www.linux-kvm.org/page/VhostNet

-- 
MST

RE: vhost-net patches

[Xin, Xiaohui]

Hi, Michael,
What's your deferring skb allocation patch mentioned here, may you elaborate it a little more detailed?

Thanks
Xiaohui

-----Original Message-----
From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Michael S. Tsirkin
Sent: Tuesday, November 03, 2009 3:47 AM
To: Shirley Ma
Cc: Sridhar Samudrala; Shirley Ma; David Stevens; kvm@vger.kernel.org; sri@linux.vnet.ibm.com; mashirle@linux.vnet.ibm.com
Subject: Re: vhost-net patches

On Thu, Oct 29, 2009 at 12:11:33AM -0700, Shirley Ma wrote:
> Hello Michael,
> 
> I am able to get 63xxMb/s throughput with 10% less cpu utilization when
> I apply deferring skb patch on top of your most recent vhost patch. The
> userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 
> 
> After applying your recent vhost patch, it goes up to 53xxMb/s.

Thanks for the report!  I have pushed this patch (after cleaning it up
and fixing raw sockets), and updated the numbers in the wiki here
http://www.linux-kvm.org/page/VhostNet

-- 
MST
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: vhost-net patches

[Shirley Ma]

Hello Xiaohui,

On Tue, 2009-11-03 at 09:06 +0800, Xin, Xiaohui wrote:
> Hi, Michael,
> What's your deferring skb allocation patch mentioned here, may you
> elaborate it a little more detailed?

That's my patch. It was submitted a few month ago. Here is the link to
this RFC patch:
http://www.mail-archive.com/kvm@vger.kernel.org/msg20777.html

It is a patch for guest receiving. Right now, kvm guest did pre-skb
allocation, the worse case when receiving large packet for mergable
buffers, 15/16 skbs need to be freed. Avi and Michale gave me some
comments. I will post the updated patch after running a few more test in
a few days.

Thanks
Shirley