* SR-IOV problems - HVM cannot access network @ 2011-03-01 19:31 David White 2011-03-01 20:50 ` Rose, Gregory V 0 siblings, 1 reply; 8+ messages in thread From: David White @ 2011-03-01 19:31 UTC (permalink / raw) To: xen-devel Hi all, I am having problems getting SR-IOV functions to work in my HVMs. My hardware has VT-d support, and pci passthrough works fine for physical functions. In a nutshell here is the current state: Dom0 : NICs are 2-port 82576. I can get full network access using either PF or VF interfaces. HVM : PCI passthrough of physical functions work -- full network access HVM : PCI passthrough of virtual functions fail -- can send pkts but cannot receive. The best lead I have right now is evident from the qemu logs. when PF (04:00.0) is assigned to HVM: pt_msix_init: get MSI-X table bar base fafbc000 pt_msix_init: table_off = 0, total_entries = 10 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 pt_msi_setup: msi mapped with pirq 37 pci_intx: intx=1 register_real_device: Real physical device 04:00.0 registered successfuly! IRQ type = MSI-INTx when VF (04:10.2) is assigned to HVM: pt_msix_init: get MSI-X table bar base fae24000 pt_msix_init: table_off = 0, total_entries = 3 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7fc918846000 register_real_device: Real physical device 04:10.2 registered successfuly! IRQ type = INTx VFs don't seem to be using MSI/MSI-X interupts. Does this indicate a problem? Can anyone give some insight on this? thanks - david ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: SR-IOV problems - HVM cannot access network 2011-03-01 19:31 SR-IOV problems - HVM cannot access network David White @ 2011-03-01 20:50 ` Rose, Gregory V 2011-03-01 22:04 ` David White 0 siblings, 1 reply; 8+ messages in thread From: Rose, Gregory V @ 2011-03-01 20:50 UTC (permalink / raw) To: David White, xen-devel > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of David White > Sent: Tuesday, March 01, 2011 11:32 AM > To: xen-devel@lists.xensource.com > Subject: [Xen-devel] SR-IOV problems - HVM cannot access network > > Hi all, > > I am having problems getting SR-IOV functions to work in my HVMs. My > hardware has VT-d support, and pci passthrough works fine for physical > functions. In a nutshell here is the current state: > > Dom0 : NICs are 2-port 82576. I can get full network access using > either PF or VF interfaces. > HVM : PCI passthrough of physical functions work -- full network access > HVM : PCI passthrough of virtual functions fail -- can send pkts but > cannot receive. > > The best lead I have right now is evident from the qemu logs. > > when PF (04:00.0) is assigned to HVM: > > pt_msix_init: get MSI-X table bar base fafbc000 > pt_msix_init: table_off = 0, total_entries = 10 > pt_msix_init: errno = 2 > pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 > pt_msi_setup: msi mapped with pirq 37 > pci_intx: intx=1 > register_real_device: Real physical device 04:00.0 registered successfuly! > IRQ type = MSI-INTx > > when VF (04:10.2) is assigned to HVM: > > pt_msix_init: get MSI-X table bar base fae24000 > pt_msix_init: table_off = 0, total_entries = 3 > pt_msix_init: errno = 2 > pt_msix_init: mapping physical MSI-X table to 7fc918846000 > register_real_device: Real physical device 04:10.2 registered successfuly! > IRQ type = INTx > > VFs don't seem to be using MSI/MSI-X interupts. Does this indicate a > problem? Yes, this is absolutely a problem. 82576 virtual functions require MSI-X interrupt support to function properly. You didn't mention what your guest OS is but the guest OS must support MSI-X interrupts. Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason. If that happens then the VF will not function correctly. - Greg Rose LAN Access Division Intel Corp. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SR-IOV problems - HVM cannot access network 2011-03-01 20:50 ` Rose, Gregory V @ 2011-03-01 22:04 ` David White 2011-03-01 22:41 ` Rose, Gregory V 0 siblings, 1 reply; 8+ messages in thread From: David White @ 2011-03-01 22:04 UTC (permalink / raw) To: Rose, Gregory V; +Cc: xen-devel On 03/01/2011 12:50 PM, Rose, Gregory V wrote: >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of David White >> Sent: Tuesday, March 01, 2011 11:32 AM >> To: xen-devel@lists.xensource.com >> Subject: [Xen-devel] SR-IOV problems - HVM cannot access network >> >> Hi all, >> >> I am having problems getting SR-IOV functions to work in my HVMs. My >> hardware has VT-d support, and pci passthrough works fine for physical >> functions. In a nutshell here is the current state: >> >> Dom0 : NICs are 2-port 82576. I can get full network access using >> either PF or VF interfaces. >> HVM : PCI passthrough of physical functions work -- full network access >> HVM : PCI passthrough of virtual functions fail -- can send pkts but >> cannot receive. >> >> The best lead I have right now is evident from the qemu logs. >> >> when PF (04:00.0) is assigned to HVM: >> >> pt_msix_init: get MSI-X table bar base fafbc000 >> pt_msix_init: table_off = 0, total_entries = 10 >> pt_msix_init: errno = 2 >> pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 >> pt_msi_setup: msi mapped with pirq 37 >> pci_intx: intx=1 >> register_real_device: Real physical device 04:00.0 registered successfuly! >> IRQ type = MSI-INTx >> >> when VF (04:10.2) is assigned to HVM: >> >> pt_msix_init: get MSI-X table bar base fae24000 >> pt_msix_init: table_off = 0, total_entries = 3 >> pt_msix_init: errno = 2 >> pt_msix_init: mapping physical MSI-X table to 7fc918846000 >> register_real_device: Real physical device 04:10.2 registered successfuly! >> IRQ type = INTx >> >> VFs don't seem to be using MSI/MSI-X interupts. Does this indicate a >> problem? > Yes, this is absolutely a problem. 82576 virtual functions require MSI-X interrupt support to function properly. You didn't mention what your guest OS is but the guest OS must support MSI-X interrupts. Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason. If that happens then the VF will not function correctly. > > - Greg Rose > LAN Access Division > Intel Corp. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel The guest (Ubuntu Maverick 64-bit) supports MSI-X, as is indicated by the fact that PF passthrough works fine (am I interpreting the above output correctly, for the PF/HVM case?). The only thing different between the two above cases is that the first one is assigned a PF, the second on a VF. Is there something in the dom0 igb/igbvf drivers that affect the MSI-X capabilities in the HVM? igb/igbvf drivers on dom0 are from Intel site: root@dom0:~# modinfo igb | grep ^version: version: 2.4.12 root@dom0:~# modinfo igbvf | grep ^version: version: 1.0.7 The guest driver does not seem to be the source of this difference, since these qemu logs are logged before HVM grub menu appears (and hence before guest kernel is loaded) Why would a physical function invoke MSI-X but not a virtual function? Is it something that pciback is doing? When I bind a PF to pciback, the xen-pciback driver shows: [ 1390.090693] pciback 0000:04:00.1: seizing device [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 [ 1390.101021] Already setup the GSI :17 [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17 [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled but when I bind a VF to pciback, the driver shows: [ 1439.411763] pciback 0000:04:10.0: seizing device [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) (note no INT or GSI messages) could pciback be the source of the HVM INTx problem? -David ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: SR-IOV problems - HVM cannot access network 2011-03-01 22:04 ` David White @ 2011-03-01 22:41 ` Rose, Gregory V 2011-03-02 2:41 ` David White 0 siblings, 1 reply; 8+ messages in thread From: Rose, Gregory V @ 2011-03-01 22:41 UTC (permalink / raw) To: David White; +Cc: xen-devel > -----Original Message----- > From: David White [mailto:dwhite@speakeasy.net] > Sent: Tuesday, March 01, 2011 2:05 PM > To: Rose, Gregory V > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network > [snip} > > The guest driver does not seem to be the source of this difference, > since these qemu logs are logged before HVM grub menu appears (and hence > before guest kernel is loaded) > > Why would a physical function invoke MSI-X but not a virtual function? A lot of reasons actually :-). SR-IOV is a complete platform solution and feature. It requires full support from all PCIe devices from the root complex to the endpoint device as well as proper BIOS programming and OS support. I don't know that Ubuntu actually supports SR-IOV and I don't know which platform you're using and whether it supports SR-IOV. > Is it something that pciback is doing? > > When I bind a PF to pciback, the xen-pciback driver shows: > [ 1390.090693] pciback 0000:04:00.1: seizing device > [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 > [ 1390.101021] Already setup the GSI :17 > [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> > IRQ 17 > [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled > > but when I bind a VF to pciback, the driver shows: > [ 1439.411763] pciback 0000:04:10.0: seizing device > [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) > > (note no INT or GSI messages) > > could pciback be the source of the HVM INTx problem? Well it's certainly possible but another good possibility is that the machine doesn't actually implement the SR-IOV feature correctly in the BIOS. What machine are you using? Is there a BIOS setting to enable SR-IOV and if so are you sure you've enabled it? Also, would it be possible to get a register dump of the VF device from both the Dom0 host domain and the DomU guest domain using the Intel ethregs utility? You can get that from source forge. - Greg ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SR-IOV problems - HVM cannot access network 2011-03-01 22:41 ` Rose, Gregory V @ 2011-03-02 2:41 ` David White 2011-03-02 11:06 ` Stefano Stabellini 0 siblings, 1 reply; 8+ messages in thread From: David White @ 2011-03-02 2:41 UTC (permalink / raw) To: Rose, Gregory V; +Cc: xen-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 3140 bytes --] On 03/01/2011 02:41 PM, Rose, Gregory V wrote: >> -----Original Message----- >> From: David White [mailto:dwhite@speakeasy.net] >> Sent: Tuesday, March 01, 2011 2:05 PM >> To: Rose, Gregory V >> Cc:xen-devel@lists.xensource.com >> Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network >> > [snip} > >> The guest driver does not seem to be the source of this difference, >> since these qemu logs are logged before HVM grub menu appears (and hence >> before guest kernel is loaded) >> >> Why would a physical function invoke MSI-X but not a virtual function? > A lot of reasons actually :-). SR-IOV is a complete platform solution and > feature. It requires full support from all PCIe devices from the root complex > to the endpoint device as well as proper BIOS programming and OS support. > > I don't know that Ubuntu actually supports SR-IOV and I don't know which > platform you're using and whether it supports SR-IOV. > >> Is it something that pciback is doing? >> >> When I bind a PF to pciback, the xen-pciback driver shows: >> [ 1390.090693] pciback 0000:04:00.1: seizing device >> [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 >> [ 1390.101021] Already setup the GSI :17 >> [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> >> IRQ 17 >> [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled >> >> but when I bind a VF to pciback, the driver shows: >> [ 1439.411763] pciback 0000:04:10.0: seizing device >> [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) >> >> (note no INT or GSI messages) >> >> could pciback be the source of the HVM INTx problem? > Well it's certainly possible but another good possibility is that the machine > doesn't actually implement the SR-IOV feature correctly in the BIOS. What > machine are you using? Is there a BIOS setting to enable SR-IOV and if so are > you sure you've enabled it? > > Also, would it be possible to get a register dump of the VF device from both > the Dom0 host domain and the DomU guest domain using the Intel ethregs > utility? You can get that from source forge. > > - Greg > Thanks for your help so far. Ubuntu Maverick is only running in the guest HVM domains. I have two boards I am testing with, both with Intel 5520 chipset. I'm using two 82576 nics from two different vendors, one attached to either board. Both systems have AMI bios. One system (call it systemA) has bios options for both VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) has bios options for VT-d and "Coherency support" (which seemed to help me enumerate the VFs), and both are enabled (no mention of I/OAT). SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. SystemB runs Fedora14 with the well-known "jeremy" kernel (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), and xen-4.0.1 hypervisor built from source (unaltered). Both systems are experiencing the same behavior. attached are dom0 and hvm ethregs output for the VF using '-o -m' flags. This one was taken from the Debian host ("systemA") -dave [-- Attachment #2: Type: TEXT/PLAIN, Size: 4009 bytes --] 01:10.0 (8086:10ca) Intel Corporation 82576 Virtual Function Offset Name Value ~~~~~~ ~~~~ ~~~~~ (0x00000) CTRL 00000000 (0x00008) STATUS 000c8743 (0x00800) VMBMEM0 a0020003 (0x00c40) V2PMAILBOX0 00000000 (0x01048) FRTIMER b5c5c2df (0x01520) EICS 00000005 (0x01524) EIMS 00000000 (0x01528) EIMC 00000000 (0x0152c) EIAC 00000007 (0x01530) EIAM 00000007 name: EITR len: 2 (0x01680) EITR[0] 00e007a0 (0x01684) EITR[1] 000007a0 (0x01700) IVAR 0a0a8081 (0x01740) IVAR_MISC 00000082 (0x02800) RDBAL[0] 1df7c000 (0x02804) RDBAH[0] 00000000 (0x02808) RDLEN[0] 00004000 (0x0280c) SRRCTL[0] f20b0002 (0x0290c) SRRCTL[1] 80000400 (0x02810) RDH[0] 00000012 (0x02814) DCA_RXCTL[0] 0000a200 (0x02914) DCA_RXCTL[1] 0000a200 (0x02818) RDT[0] 000003fe (0x02828) RXDCTL[0] 02010810 (0x02928) RXDCTL[1] 00010000 (0x02830) RQDPC[0] 00000000 (0x03800) TDBAL[0] 1dbc8000 (0x03804) TDBAH[0] 00000000 (0x03808) TDLEN[0] 00004000 (0x03810) TDH[0] 00000006 (0x03814) DCA_TXCTRL[0] 00002200 (0x03914) DCA_TXCTRL[1] 00002a00 (0x03818) TDT[0] 00000006 (0x03828) TXDCTL[0] 02000000 (0x03838) TDWBAL[0] 00000000 (0x0383c) TDWBAH[0] 00000000 Vector Control Message Data Message Address 0x00000000 0x00000000 0x00000000FEE003F8 0x00000000 0x00000000 0x00000000FEE00418 0x00000000 0x00000000 0x00000000FEE00498 Pending Bit Array 0x00000000 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 [-- Attachment #3: Type: TEXT/PLAIN, Size: 4009 bytes --] 00:04.0 (8086:10ca) Intel Corporation 82576 Virtual Function Offset Name Value ~~~~~~ ~~~~ ~~~~~ (0x00000) CTRL 00000000 (0x00008) STATUS 000c8743 (0x00800) VMBMEM0 a0020003 (0x00c40) V2PMAILBOX0 00000000 (0x01048) FRTIMER 8ea2b3a9 (0x01520) EICS 00000005 (0x01524) EIMS 00000000 (0x01528) EIMC 00000000 (0x0152c) EIAC 00000007 (0x01530) EIAM 00000007 name: EITR len: 2 (0x01680) EITR[0] 062007a0 (0x01684) EITR[1] 000007a0 (0x01700) IVAR 0a0a8081 (0x01740) IVAR_MISC 00000082 (0x02800) RDBAL[0] 1e8a8000 (0x02804) RDBAH[0] 00000000 (0x02808) RDLEN[0] 00004000 (0x0280c) SRRCTL[0] 92088002 (0x0290c) SRRCTL[1] 80000400 (0x02810) RDH[0] 0000019c (0x02814) DCA_RXCTL[0] 0000a200 (0x02914) DCA_RXCTL[1] 0000a200 (0x02818) RDT[0] 000003fe (0x02828) RXDCTL[0] 02010810 (0x02928) RXDCTL[1] 00010000 (0x02830) RQDPC[0] 00000000 (0x03800) TDBAL[0] 1d124000 (0x03804) TDBAH[0] 00000000 (0x03808) TDLEN[0] 00004000 (0x03810) TDH[0] 00000030 (0x03814) DCA_TXCTRL[0] 00002200 (0x03914) DCA_TXCTRL[1] 00002a00 (0x03818) TDT[0] 00000030 (0x03828) TXDCTL[0] 02000000 (0x03838) TDWBAL[0] 00000000 (0x0383c) TDWBAH[0] 00000000 Vector Control Message Data Message Address 0x00000000 0x00000000 0x0000000000000000 0x00000000 0x00000000 0x0000000000000000 0x00000000 0x00000000 0x0000000000000000 Pending Bit Array 0x00000000 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 [-- Attachment #4: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SR-IOV problems - HVM cannot access network 2011-03-02 2:41 ` David White @ 2011-03-02 11:06 ` Stefano Stabellini 2011-03-02 20:55 ` David white 0 siblings, 1 reply; 8+ messages in thread From: Stefano Stabellini @ 2011-03-02 11:06 UTC (permalink / raw) To: David White; +Cc: Rose, Gregory V, xen-devel On Wed, 2 Mar 2011, David White wrote: > Thanks for your help so far. > > Ubuntu Maverick is only running in the guest HVM domains. > > I have two boards I am testing with, both with Intel 5520 chipset. I'm using > two 82576 nics from two different vendors, one attached to either board. Both > systems have AMI bios. One system (call it systemA) has bios options for both > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > has bios options for VT-d and "Coherency support" (which seemed to help me > enumerate the VFs), and both are enabled (no mention of I/OAT). > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > SystemB runs Fedora14 with the well-known "jeremy" kernel > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > and xen-4.0.1 hypervisor built from source (unaltered). > > Both systems are experiencing the same behavior. > > attached are dom0 and hvm ethregs output for the VF using '-o -m' > flags. This one was taken from the Debian host ("systemA") could you please try adding acpi=0 in your VM config file? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SR-IOV problems - HVM cannot access network 2011-03-02 11:06 ` Stefano Stabellini @ 2011-03-02 20:55 ` David white 2011-03-02 21:58 ` Stefano Stabellini 0 siblings, 1 reply; 8+ messages in thread From: David white @ 2011-03-02 20:55 UTC (permalink / raw) To: Stefano Stabellini; +Cc: Rose, Gregory V, xen-devel On Wed, 2 Mar 2011, Stefano Stabellini wrote: > On Wed, 2 Mar 2011, David White wrote: > > Thanks for your help so far. > > > > Ubuntu Maverick is only running in the guest HVM domains. > > > > I have two boards I am testing with, both with Intel 5520 chipset. I'm using > > two 82576 nics from two different vendors, one attached to either board. Both > > systems have AMI bios. One system (call it systemA) has bios options for both > > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > > has bios options for VT-d and "Coherency support" (which seemed to help me > > enumerate the VFs), and both are enabled (no mention of I/OAT). > > > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > > SystemB runs Fedora14 with the well-known "jeremy" kernel > > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > > and xen-4.0.1 hypervisor built from source (unaltered). > > > > Both systems are experiencing the same behavior. > > > > attached are dom0 and hvm ethregs output for the VF using '-o -m' > > flags. This one was taken from the Debian host ("systemA") > > could you please try adding > > acpi=0 > > in your VM config file? > Hm, ok that seemed to do the trick, as far as getting the VFs up on the network. Is this by design or is it a workaround? Must I sacrifice acpi in my guest to enable sr-iov? Does this come with a cost to I/O performance, since the guest isn't using MSI? -David ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SR-IOV problems - HVM cannot access network 2011-03-02 20:55 ` David white @ 2011-03-02 21:58 ` Stefano Stabellini 0 siblings, 0 replies; 8+ messages in thread From: Stefano Stabellini @ 2011-03-02 21:58 UTC (permalink / raw) To: David white; +Cc: Rose, Gregory V, xen-devel, Allen Kay, Stefano Stabellini On Wed, 2 Mar 2011, David white wrote: > > On Wed, 2 Mar 2011, Stefano Stabellini wrote: > > > On Wed, 2 Mar 2011, David White wrote: > > > Thanks for your help so far. > > > > > > Ubuntu Maverick is only running in the guest HVM domains. > > > > > > I have two boards I am testing with, both with Intel 5520 chipset. I'm using > > > two 82576 nics from two different vendors, one attached to either board. Both > > > systems have AMI bios. One system (call it systemA) has bios options for both > > > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > > > has bios options for VT-d and "Coherency support" (which seemed to help me > > > enumerate the VFs), and both are enabled (no mention of I/OAT). > > > > > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > > > SystemB runs Fedora14 with the well-known "jeremy" kernel > > > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > > > and xen-4.0.1 hypervisor built from source (unaltered). > > > > > > Both systems are experiencing the same behavior. > > > > > > attached are dom0 and hvm ethregs output for the VF using '-o -m' > > > flags. This one was taken from the Debian host ("systemA") > > > > could you please try adding > > > > acpi=0 > > > > in your VM config file? > > > > Hm, ok that seemed to do the trick, as far as getting the VFs up on the network. > Thank you very much for testing this, it proves that the recent problems with VF passthrough and HVM Linux guests are the same issue reported here: http://marc.info/?l=xen-devel&m=129847075105008&w=2 and here: http://marc.info/?l=xen-devel&m=129891005422870&w=2 > Is this by design or is it a workaround? Must I sacrifice acpi in my guest to > enable sr-iov? Does this come with a cost to I/O performance, since the guest > isn't using MSI? It is just a workaround. It shouldn't cost you any performance at all, but you won't be able to hot-plug or hot-unplug PCI devices to your guests. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-03-02 21:58 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-03-01 19:31 SR-IOV problems - HVM cannot access network David White 2011-03-01 20:50 ` Rose, Gregory V 2011-03-01 22:04 ` David White 2011-03-01 22:41 ` Rose, Gregory V 2011-03-02 2:41 ` David White 2011-03-02 11:06 ` Stefano Stabellini 2011-03-02 20:55 ` David white 2011-03-02 21:58 ` Stefano Stabellini
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.