SR-IOV and KVM?

SR-IOV and KVM?

[Chris Friesen]

Is anyone working on SR-IOV for KVM, specifically for networking?  Or is 
virtio good enough that it's not a pressing concern?

Thanks,
Chris


-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: SR-IOV and KVM?

[lidong chen]

I have tested the performance of sr-iov for kvm, it better than virtio-net.

2011/6/25 Chris Friesen <chris.friesen@genband.com>:
>
> Is anyone working on SR-IOV for KVM, specifically for networking?  Or is
> virtio good enough that it's not a pressing concern?
>
> Thanks,
> Chris
>
>
> --
> Chris Friesen
> Software Developer
> GENBAND
> chris.friesen@genband.com
> www.genband.com
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: SR-IOV and KVM?

[Sasha Levin]

On Fri, 2011-06-24 at 15:10 -0600, Chris Friesen wrote:
> Is anyone working on SR-IOV for KVM, specifically for networking?  Or is 
> virtio good enough that it's not a pressing concern?
> 

What exactly are you missing now?

> Thanks,
> Chris
> 
> 

-- 

Sasha.

Re: SR-IOV and KVM?

[Troels Arvin]

Hello,

Chris Friesen wrote:
> Is anyone working on SR-IOV for KVM, specifically for networking?

KVM is one of the few virtualization systems which supports SR-IOV.

But my experience is that it's hard to find a hardware combination which 
really lets you use it:
http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/40981

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/

Re: SR-IOV and KVM?

[David Ahern]

On 06/25/2011 02:53 AM, Troels Arvin wrote:
> Hello,
> 
> Chris Friesen wrote:
>> Is anyone working on SR-IOV for KVM, specifically for networking?
> 
> KVM is one of the few virtualization systems which supports SR-IOV.

82576EB and a Dell R410 are working fine for me, but my host OS is
Fedora 14, not RHEL6.
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

David


> 
> But my experience is that it's hard to find a hardware combination which 
> really lets you use it:
> http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/40981
> 

Re: SR-IOV and KVM?

[Troels Arvin]

David Ahern wrote:
> 82576EB and a Dell R410 are working fine for me, but my host OS is
> Fedora 14, not RHEL6.

82576EB is a dual-port embedded NIC?

(The 82576 which is in "my" R710 is a quad-port PCIe card.)

Does your setup allow for SR-IOV without having to turn off ACS? (ACS is 
a security feature.)


> http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Interesting thread, thanks.

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/

Re: SR-IOV and KVM?

[Chris Friesen]

On 06/24/2011 07:13 PM, Sasha Levin wrote:
> On Fri, 2011-06-24 at 15:10 -0600, Chris Friesen wrote:
>> Is anyone working on SR-IOV for KVM, specifically for networking?  Or is
>> virtio good enough that it's not a pressing concern?
>>
>
> What exactly are you missing now?

Maybe it's just a documentation thing.  I'm just getting into KVM and 
trying to figure everything out, and I don't see a step-by-step guide to 
configuring a host/guest system with either SR-IOV or Intel's VMDq.

For SR-IOV I assume I need to configure the PF in the host, enable some 
number of VFs, pass through a VF to each guest and load the vf driver. 
Is that right?

Do I need to configure a software bridge to allow the guests to talk to 
each other?  I assume the passthrough require VT-d support in the 
hardware and host kernel--is that correct?

Thanks,
Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: SR-IOV and KVM?

[Avi Kivity]

On 06/27/2011 05:15 PM, Chris Friesen wrote:
>
> Maybe it's just a documentation thing.  I'm just getting into KVM and 
> trying to figure everything out, and I don't see a step-by-step guide 
> to configuring a host/guest system with either SR-IOV or Intel's VMDq.

You can use virt-manager; it has a GUI for that.  Once you're 
comfortable with it you can drop down to virsh command line and from 
there to raw qemu command line / monitor.

>
> For SR-IOV I assume I need to configure the PF in the host, enable 
> some number of VFs, pass through a VF to each guest and load the vf 
> driver. Is that right?

I think all that is needed is to prevent the host from binding to the 
guest - not sure.

> Do I need to configure a software bridge to allow the guests to talk 
> to each other? 

S/R-IOV networking is beyond my knowledge - I think it depends on your 
switch, sometimes it will loop back packets to other guests, other times 
the NIC itself will do that.  It may just work, so try it out.

> I assume the passthrough require VT-d support in the hardware and host 
> kernel--is that correct?
>

Yes.  Recent kernels have that.

-- 
error compiling committee.c: too many arguments to function

Re: SR-IOV and KVM?

[David Ahern]

On 06/27/2011 02:18 AM, Troels Arvin wrote:
> David Ahern wrote:
>> 82576EB and a Dell R410 are working fine for me, but my host OS is
>> Fedora 14, not RHEL6.
> 
> 82576EB is a dual-port embedded NIC?

I have a quad-port PCIe card.

> 
> (The 82576 which is in "my" R710 is a quad-port PCIe card.)
> 
> Does your setup allow for SR-IOV without having to turn off ACS? (ACS is 
> a security feature.)

I recall having to twiddle some BIOS settings - and doing so in
iterations. I forget which settings exactly. I'll see if Dell has a tool
to dump BIOS settings to a file that works with Fedora 14.

David

> 
> 
>> http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686
> 
> Interesting thread, thanks.
> 

Re: SR-IOV and KVM?

[David Ahern]

On 06/27/2011 08:40 AM, Avi Kivity wrote:
> On 06/27/2011 05:15 PM, Chris Friesen wrote:
>>
>> Maybe it's just a documentation thing.  I'm just getting into KVM and
>> trying to figure everything out, and I don't see a step-by-step guide
>> to configuring a host/guest system with either SR-IOV or Intel's VMDq.
> 
> You can use virt-manager; it has a GUI for that.  Once you're
> comfortable with it you can drop down to virsh command line and from
> there to raw qemu command line / monitor.

To some virt-manager is more painful than the command line is not too
painful. I have a set of home grown scripts. This is the code I added
for PCI devices:

function claim_pci_dev
{
        local pciid=$1
        local domain=0000   # TO-DO: determine this automatically
        local id

        [ -z "$pciid" ] && return 1

        local drv=$(lspci -s ${pciid} -k | awk '$0 ~ /Kernel driver in
use:/ {print $NF}')
        [ "$drv" = "pci-stub" ] && return 0

        id=$(lspci -n | awk  '{if ($1 == "'${pciid}'") {print $3}}')
        id=${id/:/ }
        if [ -z "$id" ]
        then
                err "failed to find vendor-product id for PCI id \"$pciid\""
                return 1
        fi

        echo "$id" > /sys/bus/pci/drivers/pci-stub/new_id
        if [ $? -ne 0 ]
        then
                err "Failed to tell pci-stub about id \"$id\""
                return 1
        fi

        echo "${domain}:${pciid}" >
/sys/bus/pci/devices/${domain}:${pciid}/driver/unbind
        if [ $? -ne 0 ]
        then
                err "Failed to unbind PCI device \"${domain}:${pciid}\""
                return 1
        fi

        # pci-stub claims device so it can be assigned guest
        echo "${domain}:${pciid}" > /sys/bus/pci/drivers/pci-stub/bind
        if [ $? -ne 0 ]
        then
                err "Failed to bind pci-stub to device
\"${domain}:${pciid}\""
                return 1
        fi

        return 0
}

e.g., claim_pci_dev "06:10.0"

PF-VF correlations. Example using eth2
INTFC=eth2

# pci id to VF correlation:
DEVPATH=$(readlink -f /sys/class/net/${INTFC})
ls -l ${DEVPATH}/device/virtfn*

# map a virtual function to a netdevice - in this case find the PCI id
for VF 0
VFN=0
VFID=$(readlink  ${DEVPATH}/device/virtfn${VFN} | sed -e 's,^../,,')

--> VFID is the PCI device ID for VF 0 on netdevice eth2

qemu-kvm command line addition is:
-device pci-assign,host=${PCIID}


> 
>>
>> For SR-IOV I assume I need to configure the PF in the host, enable
>> some number of VFs, pass through a VF to each guest and load the vf
>> driver. Is that right?
> 
> I think all that is needed is to prevent the host from binding to the
> guest - not sure.

See above.

> 
>> Do I need to configure a software bridge to allow the guests to talk
>> to each other? 
> 
> S/R-IOV networking is beyond my knowledge - I think it depends on your
> switch, sometimes it will loop back packets to other guests, other times
> the NIC itself will do that.  It may just work, so try it out.

PF for the NIC is attached to the network. VFs within the nic are
attached by extension. VF-VF packet flows (ie., VM to VM on the same
host) work more efficiently than hitting the wire, but not as well as
virtio connected to a bridge:
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Very good descriptions in the companion guide at
http://www.intel.com/products/ethernet/resource.htm#s1=Gigabit%20Ethernet&s2=82576EB&s3=all

David

> 
>> I assume the passthrough require VT-d support in the hardware and host
>> kernel--is that correct?
>>
> 
> Yes.  Recent kernels have that.
> 

Re: SR-IOV and KVM?

[Steve Dobbelstein]

Chris Friesen <chris.friesen@genband.com> wrote on 06/27/2011 09:15:05 AM:

> > On Fri, 2011-06-24 at 15:10 -0600, Chris Friesen wrote:
> >> Is anyone working on SR-IOV for KVM, specifically for networking?  Or
is
> >> virtio good enough that it's not a pressing concern?

In IBM's recent postings of results for the SPECvirt benchmark on RHEL 6.0
[1] we used SR-IOV for the network adapter for most of the guests.  The
adapters in the host were the Intel x520 10 Gigabit Server adapters (lscpi
says Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev
22) ).

virtio may be good enough for lighter workloads, but there was no way we
could get the higher guest counts for SPECvirt with the performance of
virtio.  We needed the performance of SR-IOV.

> For SR-IOV I assume I need to configure the PF in the host, enable some
> number of VFs, pass through a VF to each guest and load the vf driver.
> Is that right?

That is correct.  You also need to make sure your guest has the driver for
the virtual function.

As an example, on my qemu command line I have the options:
-pcidevice host=8b:10.0\
-net none\

I use the "-net none" option because the qemu I am running likes to do you
a favor and create a network device for you if you didn't specify one (it
doesn't know that the PCI device is a network device).  Without "-net
none", what can happen is that qemu will create an emulated network device
for you (e.g., e1000), and the guest will most likely have a driver for it
that it will load.  If you don't have the driver in the guest for the
virtual function, you can still get network connectivity to the guest and
may think you are using SR-IOV when you are not.

> Do I need to configure a software bridge to allow the guests to talk to
> each other?

You do not need to configure a software bridge.  The adapter on the host
behaves like a switch and routes traffic between the virtual functions.  (I
don't know offhand if the traffic between the virtual functions is looped
back by the adapter or if it actually goes out the adapter port and comes
back in.)

Steve D.

[1] http://www.spec.org/virt_sc2010/results/specvirt_sc2010_perf.html

Re: SR-IOV and KVM?

[Troels Arvin]

David Ahern wrote:
>> Does your setup allow for SR-IOV without having to turn off ACS? (ACS
>> is a security feature.)
> 
> I recall having to twiddle some BIOS settings

I believe that if you turn off ACS, it's through a kernel module option.

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/