From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sander Eikelenboom Subject: Re: [Xen-users] Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails. Date: Wed, 8 Dec 2010 17:44:39 +0100 Message-ID: <843495883.20101208174439@eikelenboom.it> References: <20101115174413.GA8227@dumpdata.com> <20101115175626.GA31636@campbell-lange.net> <20101124175926.GA17565@campbell-lange.net> <20101124202842.GA28222@dumpdata.com> <20101126111520.GA17221@campbell-lange.net> <20101129163635.GA20417@dumpdata.com> <20101208125855.GA26222@campbell-lange.net> <675512317.20101208143715@eikelenboom.it> <20101208134848.GA27543@campbell-lange.net> <1416172450.20101208150550@eikelenboom.it> <20101208154857.GA30254@campbell-lange.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101208154857.GA30254@campbell-lange.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Mark Adams Cc: "xen-devel@lists.xensource.com" , "xen-users@lists.xensource.com" , Konrad Rzeszutek Wilk , Stefano Stabellini List-Id: xen-devel@lists.xenproject.org Wednesday, December 8, 2010, 4:48:57 PM, you wrote: > On Wed, Dec 08, 2010 at 03:05:50PM +0100, Sander Eikelenboom wrote: >>=20 >> Wednesday, December 8, 2010, 2:48:48 PM, you wrote: >>=20 >> > On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote: >> >> Hello Mark, >>=20 >> > Hi >>=20 >> >>=20 >> >> Just a recap: >> >> you pass through: >> >> - 3 physical nics/IGB >> >> - 1 ISDN pci ISDN box >>=20 >> > The redfone box runs on 1 of the nics - its not seperate. It converts >> > ISDN to TDMoE see here.. http://www.red-fone.com/ >>=20 >> So the problem is probably with the igb's. >> Searching showed http://forums.virtualbox.org/viewtopic.php?f=3D7&t=3D32= 171 , perhaps worth a try ? > Tried this - doesn't help. >>=20 >> Have you tried with just 1 IGB, and/or another simple 1gb NIC (non intel= ) to see if it's due to any of the special offload features ? > Haven't got any other NIC's to try unfortunately. Even if it did work > with 1, it would be no use to me as I need 3. I understand, but simplifying the setup and trying to isolate the problem, = could clarify things. I also read you previous thread, and i saw you hide the 02:00.0 and 03:00.0= with xen-pciback (e1000e driver) there, but now you seem to be passing thr= ough 08:00.0 and 08:00.1 (igb) ? So i assume you have already tried 2 different NIC's http://download.intel.com/design/network/specupdt/82574.pdf though shows so= me errata regarding msi-x interrupts and timing issues and workarounds on t= he 82574 (02:00.0 and 03:00.0) nics. -- Sander >>=20 >>=20 >> >> - all using msi/msi-x interrupts ? >>=20 >> > I tried using msi/msi-x interrupts, but it caused the raid card to drop >> > off (after some use) and provided seemingly even worse performance than >> > pegging everything back to legacy. >>=20 >> >>=20 >> >> Have you tried using a PV domU instead of a HVM domU ? >>=20 >> > I initially tried PV but had issues with the igb NIC's. There was >> > another thread somewhere about my issues with that. >>=20 >>=20 >> >> Have you tried passing through only the ISDN box, and let the network= run with the xen backend/frontend to rule out the IGB/network stuff ? >> >>=20 >> >>=20 >> >> -- >> >> Sander >> >>=20 >> >>=20 >> >>=20 >> >> Wednesday, December 8, 2010, 1:58:55 PM, you wrote: >> >>=20 >> >> > Hi - Apologies to top post this, but after alot of testing, I belie= ve >> >> > there must be an issue with IRQ's going missing between domU and do= m0. >> >> > Unfortunately I have no data to prove this! >> >>=20 >> >> > With msitranslate=3D0 as detailed below, and pci=3Dnomsi in the gue= st kernel >> >> > grub config, all 3 NIC's appear OK in the domU however I still had >> >> > issues with the red-fone ISDN box. The interrupts were showing corr= ectly >> >> > (2000/s) in the domU but communication to the device via the NIC was >> >> > still being interrupted (as shown in the asterisk console)Note that= to >> >> > get the igb driver to allow this many interrupts, the >> >> > InterruptThrottleRate was set to 0. The same config (red-fone box, >> >> > asterisk etc) works fine with a physical server. >> >>=20 >> >> > There is also the additional issue that I could not get the passthr= ough >> >> > NIC's to show correctly when I also had a bridge setup. >> >>=20 >> >> > Throughout my testing however, I could not get the machine to crash. >> >>=20 >> >> > Not sure where to go with this one. For now we are keeping our VoIP >> >> > servers physical when ISDN connections are required. >> >>=20 >> >> > Regards, >> >> > Mark >> >>=20 >> >> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wro= te: >> >> >> >=20 >> >> >> > In my new test setup, I have seen some strange behaviour. 1 of t= he HVM's >> >> >> > (with identical config in dom0 and domU) suddenly would not allo= w the >> >> >> > igb driver to be loaded in domU, even though the device was visi= ble in >> >> >>=20 >> >> >> Let's create a new thread for this other issue. >> >> >>=20 >> >> >> > lspci. Shutting the machine down, removing the power cord, waiti= ng 5 >> >> >> > seconds then plugging it in again corrected that issue - Is this >> >> >> > possibly a motherboard bug? I have also disabled the SR-IOV >> >> >> > functionality in the BIOS incase this is causing any issues. >> >> >> >=20 >> >> >> > In addition, to try to correct the MSI issue noted above, I have= changed >> >> >> > my pci=3D line to the following: >> >> >> >=20 >> >> >> > pci=3D[ '08:00.0,msitranslate=3D0', '08:00.1,msitranslate=3D0' ] >> >> >>=20 >> >> >> With the msi_translate=3D1 turned on the DomU HVM guests did work,= right? >> >> >>=20 >> >> >> >=20 >> >> >> > This has stopped the "already in use on device" log, and the dev= ices >> >> >> > appear to show correctly in the domU. Is it safe to disable >> >> >> > msitranslate? as I understand it, its for allowing multifunction= devices >> >> >> > to be seen as such in domU. Is that correct? >> >> >> >=20 >> >> >> > I haven't been able to reproduce the dropped raid issue yet, but= I am >> >> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem t= o cause >> >> >> > this due to their very high interrupt usage (2000 per second). >> >> >>=20 >> >> >> OK. >> >> >> >=20 >> >> >> > In the mean time, I can see the following in the qemu-dm logs no= w with >> >> >> > the msitranslate=3D0 enabled. Is it anything to worry about? >> >> >>=20 >> >> >> Well, the "Error" ones are pretty bad, thought I am having a hard= time >> >> >> understanding what it means. Lets copy some of the QEMU folks on t= his. >> >> >>=20 >> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to un= used Base Address Register. [00:05.0][Offset:14h][Length:4] >> >> >> > pt_ioport_map: e_phys=3Dffff pio_base=3De880 len=3D32 index=3D2 = first_map=3D0 >> >> >> > pt_ioport_map: e_phys=3Dc220 pio_base=3De880 len=3D32 index=3D2 = first_map=3D0 >> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to un= used Base Address Register. [00:06.0][Offset:14h][Length:4] >> >> >> > pt_ioport_map: e_phys=3Dffff pio_base=3Dec00 len=3D32 index=3D2 = first_map=3D0 >> >> >> > pt_ioport_map: e_phys=3Dc240 pio_base=3Dec00 len=3D32 index=3D2 = first_map=3D0 >> >> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59 >> >> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61 >> >> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69 >> >> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71 >> >> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79 >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is= already function. >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is= already function. >> >> >> >=20 >> >> >> > >=20 >> >> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hy= pervisor when your >> >> >> > > machine is toast. I mentioned in the previous email the key se= quences - look on Google >> >> >> > > on how to pass in SysRQ if you are using a serial concentrator. >> >> >> >=20 >> >> >> > I will do this when I can get the machine to crash. >> >> >> >=20 >> >> >> > Best Regards, >> >> >> > Mark >> >> >> >=20 >> >> >> > _______________________________________________ >> >> >> > Xen-devel mailing list >> >> >> > Xen-devel@lists.xensource.com >> >> >> > http://lists.xensource.com/xen-devel >> >>=20 >> >>=20 >> >>=20 >> >>=20 >> >>=20 >> >> --=20 >> >> Best regards, >> >> Sander mailto:linux@eikelenboom.it >> >>=20 >>=20 >>=20 >>=20 >> --=20 >> Best regards, >> Sander mailto:linux@eikelenboom.it >>=20 >>=20 >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users --=20 Best regards, Sander mailto:linux@eikelenboom.it