SR-IOV problems - HVM cannot access network

SR-IOV problems - HVM cannot access network

[David White]

Hi all,

I am having problems getting SR-IOV functions to work in my HVMs.  My 
hardware has VT-d support, and pci passthrough works fine for physical 
functions.  In a nutshell here is the current state:

Dom0 :  NICs are 2-port 82576.  I can get full network access using 
either PF or VF interfaces.
HVM : PCI passthrough of physical functions work -- full network access
HVM : PCI passthrough of virtual functions fail -- can send pkts but 
cannot receive.

The best lead I have right now is evident from the qemu logs.

when PF (04:00.0) is assigned to HVM:

pt_msix_init: get MSI-X table bar base fafbc000
pt_msix_init: table_off = 0, total_entries = 10
pt_msix_init: errno = 2
pt_msix_init: mapping physical MSI-X table to 7f23a03d5000
pt_msi_setup: msi mapped with pirq 37
pci_intx: intx=1
register_real_device: Real physical device 04:00.0 registered successfuly!
IRQ type = MSI-INTx

when VF (04:10.2) is assigned to HVM:

pt_msix_init: get MSI-X table bar base fae24000
pt_msix_init: table_off = 0, total_entries = 3
pt_msix_init: errno = 2
pt_msix_init: mapping physical MSI-X table to 7fc918846000
register_real_device: Real physical device 04:10.2 registered successfuly!
IRQ type = INTx

VFs don't seem to be using MSI/MSI-X interupts.  Does this indicate a 
problem?

Can anyone give some insight on this?
thanks  - david

RE: SR-IOV problems - HVM cannot access network

[Rose, Gregory V]

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of David White
> Sent: Tuesday, March 01, 2011 11:32 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] SR-IOV problems - HVM cannot access network
> 
> Hi all,
> 
> I am having problems getting SR-IOV functions to work in my HVMs.  My
> hardware has VT-d support, and pci passthrough works fine for physical
> functions.  In a nutshell here is the current state:
> 
> Dom0 :  NICs are 2-port 82576.  I can get full network access using
> either PF or VF interfaces.
> HVM : PCI passthrough of physical functions work -- full network access
> HVM : PCI passthrough of virtual functions fail -- can send pkts but
> cannot receive.
> 
> The best lead I have right now is evident from the qemu logs.
> 
> when PF (04:00.0) is assigned to HVM:
> 
> pt_msix_init: get MSI-X table bar base fafbc000
> pt_msix_init: table_off = 0, total_entries = 10
> pt_msix_init: errno = 2
> pt_msix_init: mapping physical MSI-X table to 7f23a03d5000
> pt_msi_setup: msi mapped with pirq 37
> pci_intx: intx=1
> register_real_device: Real physical device 04:00.0 registered successfuly!
> IRQ type = MSI-INTx
> 
> when VF (04:10.2) is assigned to HVM:
> 
> pt_msix_init: get MSI-X table bar base fae24000
> pt_msix_init: table_off = 0, total_entries = 3
> pt_msix_init: errno = 2
> pt_msix_init: mapping physical MSI-X table to 7fc918846000
> register_real_device: Real physical device 04:10.2 registered successfuly!
> IRQ type = INTx
> 
> VFs don't seem to be using MSI/MSI-X interupts.  Does this indicate a
> problem?

Yes, this is absolutely a problem.  82576 virtual functions require MSI-X interrupt support to function properly.  You didn't mention what your guest OS is but the guest OS must support MSI-X interrupts.  Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason.  If that happens then the VF will not function correctly.

- Greg Rose
LAN Access Division
Intel Corp.

Re: SR-IOV problems - HVM cannot access network

[David White]

On 03/01/2011 12:50 PM, Rose, Gregory V wrote:
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of David White
>> Sent: Tuesday, March 01, 2011 11:32 AM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] SR-IOV problems - HVM cannot access network
>>
>> Hi all,
>>
>> I am having problems getting SR-IOV functions to work in my HVMs.  My
>> hardware has VT-d support, and pci passthrough works fine for physical
>> functions.  In a nutshell here is the current state:
>>
>> Dom0 :  NICs are 2-port 82576.  I can get full network access using
>> either PF or VF interfaces.
>> HVM : PCI passthrough of physical functions work -- full network access
>> HVM : PCI passthrough of virtual functions fail -- can send pkts but
>> cannot receive.
>>
>> The best lead I have right now is evident from the qemu logs.
>>
>> when PF (04:00.0) is assigned to HVM:
>>
>> pt_msix_init: get MSI-X table bar base fafbc000
>> pt_msix_init: table_off = 0, total_entries = 10
>> pt_msix_init: errno = 2
>> pt_msix_init: mapping physical MSI-X table to 7f23a03d5000
>> pt_msi_setup: msi mapped with pirq 37
>> pci_intx: intx=1
>> register_real_device: Real physical device 04:00.0 registered successfuly!
>> IRQ type = MSI-INTx
>>
>> when VF (04:10.2) is assigned to HVM:
>>
>> pt_msix_init: get MSI-X table bar base fae24000
>> pt_msix_init: table_off = 0, total_entries = 3
>> pt_msix_init: errno = 2
>> pt_msix_init: mapping physical MSI-X table to 7fc918846000
>> register_real_device: Real physical device 04:10.2 registered successfuly!
>> IRQ type = INTx
>>
>> VFs don't seem to be using MSI/MSI-X interupts.  Does this indicate a
>> problem?
> Yes, this is absolutely a problem.  82576 virtual functions require MSI-X interrupt support to function properly.  You didn't mention what your guest OS is but the guest OS must support MSI-X interrupts.  Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason.  If that happens then the VF will not function correctly.
>
> - Greg Rose
> LAN Access Division
> Intel Corp.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

The guest (Ubuntu Maverick 64-bit) supports MSI-X, as is indicated by the fact
that PF passthrough works fine (am I interpreting the above output correctly,
for the PF/HVM case?).

The only thing different between the two above cases is that the first one is
assigned a PF, the second on a VF.  Is there something in the dom0 igb/igbvf
drivers that affect the MSI-X capabilities in the HVM?  igb/igbvf drivers on
dom0 are from Intel site:

root@dom0:~# modinfo igb | grep ^version:
version:        2.4.12
root@dom0:~# modinfo igbvf | grep ^version:
version:        1.0.7

The guest driver does not seem to be the source of this difference,
since these qemu logs are logged before HVM grub menu appears (and hence
before guest kernel is loaded)

Why would a physical function invoke MSI-X but not a virtual function?
Is it something that pciback is doing?

When I bind a PF to pciback, the xen-pciback driver shows:
[ 1390.090693] pciback 0000:04:00.1: seizing device
[ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17
[ 1390.101021] Already setup the GSI :17
[ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
[ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled

but when I bind a VF to pciback, the driver shows:
[ 1439.411763] pciback 0000:04:10.0: seizing device
[ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002)

(note no INT or GSI messages)

could pciback be the source of the HVM INTx problem?

-David

RE: SR-IOV problems - HVM cannot access network

[Rose, Gregory V]

> -----Original Message-----
> From: David White [mailto:dwhite@speakeasy.net]
> Sent: Tuesday, March 01, 2011 2:05 PM
> To: Rose, Gregory V
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network
> 
 [snip}

> 
> The guest driver does not seem to be the source of this difference,
> since these qemu logs are logged before HVM grub menu appears (and hence
> before guest kernel is loaded)
> 
> Why would a physical function invoke MSI-X but not a virtual function?

A lot of reasons actually :-).  SR-IOV is a complete platform solution and feature.  It requires full support from all PCIe devices from the root complex to the endpoint device as well as proper BIOS programming and OS support.

I don't know that Ubuntu actually supports SR-IOV and I don't know which platform you're using and whether it supports SR-IOV.

> Is it something that pciback is doing?
> 
> When I bind a PF to pciback, the xen-pciback driver shows:
> [ 1390.090693] pciback 0000:04:00.1: seizing device
> [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17
> [ 1390.101021] Already setup the GSI :17
> [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) ->
> IRQ 17
> [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled
> 
> but when I bind a VF to pciback, the driver shows:
> [ 1439.411763] pciback 0000:04:10.0: seizing device
> [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002)
> 
> (note no INT or GSI messages)
> 
> could pciback be the source of the HVM INTx problem?

Well it's certainly possible but another good possibility is that the machine doesn't actually implement the SR-IOV feature correctly in the BIOS.  What machine are you using?  Is there a BIOS setting to enable SR-IOV and if so are you sure you've enabled it?

Also, would it be possible to get a register dump of the VF device from both the Dom0 host domain and the DomU guest domain using the Intel ethregs utility?  You can get that from source forge.

- Greg

Re: SR-IOV problems - HVM cannot access network

[David White]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3140 bytes --]


On 03/01/2011 02:41 PM, Rose, Gregory V wrote:
>> -----Original Message-----
>> From: David White [mailto:dwhite@speakeasy.net]
>> Sent: Tuesday, March 01, 2011 2:05 PM
>> To: Rose, Gregory V
>> Cc:xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network
>>
>   [snip}
>
>> The guest driver does not seem to be the source of this difference,
>> since these qemu logs are logged before HVM grub menu appears (and hence
>> before guest kernel is loaded)
>>
>> Why would a physical function invoke MSI-X but not a virtual function?
> A lot of reasons actually :-).  SR-IOV is a complete platform solution and
> feature.  It requires full support from all PCIe devices from the root complex
> to the endpoint device as well as proper BIOS programming and OS support.
>
> I don't know that Ubuntu actually supports SR-IOV and I don't know which
> platform you're using and whether it supports SR-IOV.
>
>> Is it something that pciback is doing?
>>
>> When I bind a PF to pciback, the xen-pciback driver shows:
>> [ 1390.090693] pciback 0000:04:00.1: seizing device
>> [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17
>> [ 1390.101021] Already setup the GSI :17
>> [ 1390.104717] pciback 0000:04:00.1: PCI INT B ->  GSI 17 (level, low) ->
>> IRQ 17
>> [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled
>>
>> but when I bind a VF to pciback, the driver shows:
>> [ 1439.411763] pciback 0000:04:10.0: seizing device
>> [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 ->  0002)
>>
>> (note no INT or GSI messages)
>>
>> could pciback be the source of the HVM INTx problem?
> Well it's certainly possible but another good possibility is that the machine
> doesn't actually implement the SR-IOV feature correctly in the BIOS.  What
> machine are you using?  Is there a BIOS setting to enable SR-IOV and if so are
> you sure you've enabled it?
>
> Also, would it be possible to get a register dump of the VF device from both
> the Dom0 host domain and the DomU guest domain using the Intel ethregs
> utility?  You can get that from source forge.
>
> - Greg
>

Thanks for your help so far.

Ubuntu Maverick is only running in the guest HVM domains.

I have two boards I am testing with, both with Intel 5520 chipset.  I'm using
two 82576 nics from two different vendors, one attached to either board.  Both
systems have AMI bios.  One system (call it systemA) has bios options for both
VT-d and SR-IOV, and both are enabled, as is I/OAT.  The other system (systemB)
has bios options for VT-d and "Coherency support" (which seemed to help me
enumerate the VFs), and both are enabled (no mention of I/OAT).

SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro.
SystemB runs Fedora14 with the well-known "jeremy" kernel
(http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x),
and xen-4.0.1 hypervisor built from source (unaltered).

Both systems are experiencing the same behavior.

attached are dom0 and hvm ethregs output for the VF using '-o -m'
flags.  This one was taken from the Debian host ("systemA")

-dave

[-- Attachment #2: Type: TEXT/PLAIN, Size: 4009 bytes --]

01:10.0 (8086:10ca)
Intel Corporation 82576 Virtual Function
	Offset    Name                  Value
	~~~~~~    ~~~~                  ~~~~~
	(0x00000) CTRL                  00000000
	(0x00008) STATUS                000c8743
	(0x00800) VMBMEM0               a0020003
	(0x00c40) V2PMAILBOX0           00000000
	(0x01048) FRTIMER               b5c5c2df
	(0x01520) EICS                  00000005
	(0x01524) EIMS                  00000000
	(0x01528) EIMC                  00000000
	(0x0152c) EIAC                  00000007
	(0x01530) EIAM                  00000007
name: EITR  len: 2
	(0x01680) EITR[0]               00e007a0
	(0x01684) EITR[1]               000007a0
	(0x01700) IVAR                  0a0a8081
	(0x01740) IVAR_MISC             00000082
	(0x02800) RDBAL[0]              1df7c000
	(0x02804) RDBAH[0]              00000000
	(0x02808) RDLEN[0]              00004000
	(0x0280c) SRRCTL[0]             f20b0002
	(0x0290c) SRRCTL[1]      80000400
	(0x02810) RDH[0]                00000012
	(0x02814) DCA_RXCTL[0]          0000a200
	(0x02914) DCA_RXCTL[1]   0000a200
	(0x02818) RDT[0]                000003fe
	(0x02828) RXDCTL[0]             02010810
	(0x02928) RXDCTL[1]      00010000
	(0x02830) RQDPC[0]              00000000
	(0x03800) TDBAL[0]              1dbc8000
	(0x03804) TDBAH[0]              00000000
	(0x03808) TDLEN[0]              00004000
	(0x03810) TDH[0]                00000006
	(0x03814) DCA_TXCTRL[0]         00002200
	(0x03914) DCA_TXCTRL[1]  00002a00
	(0x03818) TDT[0]                00000006
	(0x03828) TXDCTL[0]             02000000
	(0x03838) TDWBAL[0]             00000000
	(0x0383c) TDWBAH[0]             00000000


	Vector Control 	Message Data 	Message Address
	0x00000000 	0x00000000 	0x00000000FEE003F8
	0x00000000 	0x00000000 	0x00000000FEE00418
	0x00000000 	0x00000000 	0x00000000FEE00498

	Pending Bit Array
	0x00000000


\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0

[-- Attachment #3: Type: TEXT/PLAIN, Size: 4009 bytes --]

00:04.0 (8086:10ca)
Intel Corporation 82576 Virtual Function
	Offset    Name                  Value
	~~~~~~    ~~~~                  ~~~~~
	(0x00000) CTRL                  00000000
	(0x00008) STATUS                000c8743
	(0x00800) VMBMEM0               a0020003
	(0x00c40) V2PMAILBOX0           00000000
	(0x01048) FRTIMER               8ea2b3a9
	(0x01520) EICS                  00000005
	(0x01524) EIMS                  00000000
	(0x01528) EIMC                  00000000
	(0x0152c) EIAC                  00000007
	(0x01530) EIAM                  00000007
name: EITR  len: 2
	(0x01680) EITR[0]               062007a0
	(0x01684) EITR[1]               000007a0
	(0x01700) IVAR                  0a0a8081
	(0x01740) IVAR_MISC             00000082
	(0x02800) RDBAL[0]              1e8a8000
	(0x02804) RDBAH[0]              00000000
	(0x02808) RDLEN[0]              00004000
	(0x0280c) SRRCTL[0]             92088002
	(0x0290c) SRRCTL[1]      80000400
	(0x02810) RDH[0]                0000019c
	(0x02814) DCA_RXCTL[0]          0000a200
	(0x02914) DCA_RXCTL[1]   0000a200
	(0x02818) RDT[0]                000003fe
	(0x02828) RXDCTL[0]             02010810
	(0x02928) RXDCTL[1]      00010000
	(0x02830) RQDPC[0]              00000000
	(0x03800) TDBAL[0]              1d124000
	(0x03804) TDBAH[0]              00000000
	(0x03808) TDLEN[0]              00004000
	(0x03810) TDH[0]                00000030
	(0x03814) DCA_TXCTRL[0]         00002200
	(0x03914) DCA_TXCTRL[1]  00002a00
	(0x03818) TDT[0]                00000030
	(0x03828) TXDCTL[0]             02000000
	(0x03838) TDWBAL[0]             00000000
	(0x0383c) TDWBAH[0]             00000000


	Vector Control 	Message Data 	Message Address
	0x00000000 	0x00000000 	0x0000000000000000
	0x00000000 	0x00000000 	0x0000000000000000
	0x00000000 	0x00000000 	0x0000000000000000

	Pending Bit Array
	0x00000000


\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: SR-IOV problems - HVM cannot access network

[Stefano Stabellini]

On Wed, 2 Mar 2011, David White wrote:
> Thanks for your help so far.
> 
> Ubuntu Maverick is only running in the guest HVM domains.
> 
> I have two boards I am testing with, both with Intel 5520 chipset.  I'm using
> two 82576 nics from two different vendors, one attached to either board.  Both
> systems have AMI bios.  One system (call it systemA) has bios options for both
> VT-d and SR-IOV, and both are enabled, as is I/OAT.  The other system (systemB)
> has bios options for VT-d and "Coherency support" (which seemed to help me
> enumerate the VFs), and both are enabled (no mention of I/OAT).
> 
> SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro.
> SystemB runs Fedora14 with the well-known "jeremy" kernel
> (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x),
> and xen-4.0.1 hypervisor built from source (unaltered).
> 
> Both systems are experiencing the same behavior.
> 
> attached are dom0 and hvm ethregs output for the VF using '-o -m'
> flags.  This one was taken from the Debian host ("systemA")
 
could you please try adding

acpi=0

in your VM config file?

Re: SR-IOV problems - HVM cannot access network

[David white]

On Wed, 2 Mar 2011, Stefano Stabellini wrote:

> On Wed, 2 Mar 2011, David White wrote:
> > Thanks for your help so far.
> >
> > Ubuntu Maverick is only running in the guest HVM domains.
> >
> > I have two boards I am testing with, both with Intel 5520 chipset.  I'm using
> > two 82576 nics from two different vendors, one attached to either board.  Both
> > systems have AMI bios.  One system (call it systemA) has bios options for both
> > VT-d and SR-IOV, and both are enabled, as is I/OAT.  The other system (systemB)
> > has bios options for VT-d and "Coherency support" (which seemed to help me
> > enumerate the VFs), and both are enabled (no mention of I/OAT).
> >
> > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro.
> > SystemB runs Fedora14 with the well-known "jeremy" kernel
> > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x),
> > and xen-4.0.1 hypervisor built from source (unaltered).
> >
> > Both systems are experiencing the same behavior.
> >
> > attached are dom0 and hvm ethregs output for the VF using '-o -m'
> > flags.  This one was taken from the Debian host ("systemA")
>
> could you please try adding
>
> acpi=0
>
> in your VM config file?
>

Hm, ok that seemed to do the trick, as far as getting the VFs up on the network.

Is this by design or is it a workaround? Must I sacrifice acpi in my guest to
enable sr-iov?  Does this come with a cost to I/O performance, since the guest
isn't using MSI?

-David

Re: SR-IOV problems - HVM cannot access network

[Stefano Stabellini]

On Wed, 2 Mar 2011, David white wrote:
> 
> On Wed, 2 Mar 2011, Stefano Stabellini wrote:
> 
> > On Wed, 2 Mar 2011, David White wrote:
> > > Thanks for your help so far.
> > >
> > > Ubuntu Maverick is only running in the guest HVM domains.
> > >
> > > I have two boards I am testing with, both with Intel 5520 chipset.  I'm using
> > > two 82576 nics from two different vendors, one attached to either board.  Both
> > > systems have AMI bios.  One system (call it systemA) has bios options for both
> > > VT-d and SR-IOV, and both are enabled, as is I/OAT.  The other system (systemB)
> > > has bios options for VT-d and "Coherency support" (which seemed to help me
> > > enumerate the VFs), and both are enabled (no mention of I/OAT).
> > >
> > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro.
> > > SystemB runs Fedora14 with the well-known "jeremy" kernel
> > > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x),
> > > and xen-4.0.1 hypervisor built from source (unaltered).
> > >
> > > Both systems are experiencing the same behavior.
> > >
> > > attached are dom0 and hvm ethregs output for the VF using '-o -m'
> > > flags.  This one was taken from the Debian host ("systemA")
> >
> > could you please try adding
> >
> > acpi=0
> >
> > in your VM config file?
> >
> 
> Hm, ok that seemed to do the trick, as far as getting the VFs up on the network.
> 

Thank you very much for testing this, it proves that the recent problems
with VF passthrough and HVM Linux guests are the same issue reported
here:

http://marc.info/?l=xen-devel&m=129847075105008&w=2

and here:

http://marc.info/?l=xen-devel&m=129891005422870&w=2


> Is this by design or is it a workaround? Must I sacrifice acpi in my guest to
> enable sr-iov?  Does this come with a cost to I/O performance, since the guest
> isn't using MSI?
 
It is just a workaround.
It shouldn't cost you any performance at all, but you won't be able to
hot-plug or hot-unplug PCI devices to your guests.