Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Dietmar Hahn <dietmar.hahn@ts.fujitsu.com>]

[-- Attachment #1.1: Type: text/plain, Size: 6279 bytes --]

Hi,

Am 08.12.2010 schrieb "Mark Adams <mark@campbell-lange.net>":
> Hi - Apologies to top post this, but after alot of testing, I believe
> there must be an issue with IRQ's going missing between domU and dom0.
> Unfortunately I have no data to prove this!
> 
> With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> grub config, all 3 NIC's appear OK in the domU however I still had
> issues with the red-fone ISDN box. The interrupts were showing correctly
> (2000/s) in the domU but communication to the device via the NIC was
> still being interrupted (as shown in the asterisk console)Note that to
> get the igb driver to allow this many interrupts, the
> InterruptThrottleRate was set to 0. The same config (red-fone box,
> asterisk etc) works fine with a physical server.
> 
> There is also the additional issue that I could not get the passthrough
> NIC's to show correctly when I also had a bridge setup.
> 
> Throughout my testing however, I could not get the machine to crash.
> 
> Not sure where to go with this one. For now we are keeping our VoIP
> servers physical when ISDN connections are required.

Today I did some tests with xen-unstable and found these problems too.
I tried to passthrough 2 pci cards and got some error messages on the xen
xonsole and in the qemu logs.
With msitranslate=0 and pci=nomsi I got the soundcard working in a domU linux
but it doesn't help on windows.
I attached the logs from the xen serial console and the qemu logs.
Thanks!

Dietmar.

> 
> Regards,
> Mark
> 
> On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> > > 
> > > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> > > (with identical config in dom0 and domU) suddenly would not allow the
> > > igb driver to be loaded in domU, even though the device was visible in
> > 
> > Let's create a new thread for this other issue.
> > 
> > > lspci. Shutting the machine down, removing the power cord, waiting 5
> > > seconds then plugging it in again corrected that issue - Is this
> > > possibly a motherboard bug? I have also disabled the SR-IOV
> > > functionality in the BIOS incase this is causing any issues.
> > > 
> > > In addition, to try to correct the MSI issue noted above, I have changed
> > > my pci= line to the following:
> > > 
> > > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> > 
> > With the msi_translate=1 turned on the DomU HVM guests did work, right?
> > 
> > > 
> > > This has stopped the "already in use on device" log, and the devices
> > > appear to show correctly in the domU. Is it safe to disable
> > > msitranslate? as I understand it, its for allowing multifunction devices
> > > to be seen as such in domU. Is that correct?
> > > 
> > > I haven't been able to reproduce the dropped raid issue yet, but I am
> > > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> > > this due to their very high interrupt usage (2000 per second).
> > 
> > OK.
> > > 
> > > In the mean time, I can see the following in the qemu-dm logs now with
> > > the msitranslate=0 enabled. Is it anything to worry about?
> > 
> > Well, the  "Error" ones are pretty bad, thought I am having a hard time
> > understanding what it means. Lets copy some of the QEMU folks on this.
> > 
> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> > > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> > > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> > > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> > > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> > > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> > > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> > > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> > > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> > > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > 
> > > > 
> > > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> > > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> > > > on how to pass in SysRQ if you are using a serial concentrator.
> > > 
> > > I will do this when I can get the machine to crash.
> > > 
> > > Best Regards,
> > > Mark
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
-- 
Company details: http://ts.fujitsu.com/imprint.html

[-- Attachment #1.2: Type: text/html, Size: 25276 bytes --]

[-- Attachment #2: xen-serial-log.txt --]
[-- Type: text/plain, Size: 7666 bytes --]


(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=18 memflags=0 (3 of 4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=18 memflags=0 (0 of 3)
(XEN) [VT-D]iommu.c:1514: d0:PCIe: unmap bdf = 0:1b.0
(XEN) [VT-D]iommu.c:1387: d18:PCIe: map bdf = 0:1b.0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=0 g_gsi=32 device=4 intx=0
(XEN) [VT-D]iommu.c:1514: d0:PCIe: unmap bdf = 10:0.0
(XEN) HAHN-me_wifi_quirk: id 422c8086
(XEN) [VT-D]iommu.c:1387: d18:PCIe: map bdf = 10:0.0
(XEN) HAHN-me_wifi_quirk: id 422c8086
(XEN) irq.c:1511: dom18: pirq 0 or irq 40 already mapped
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=0 g_gsi=36 device=5 intx=0
(XEN) HVM18: HVM Loader
(XEN) HVM18: Detected Xen v4.1-unstable
(XEN) HVM18: CPU speed is 2394 MHz
(XEN) HVM18: Xenbus rings @0xfeffc000, event channel 2
(XEN) irq.c:258: Dom18 PCI link 0 changed 0 -> 5
(XEN) HVM18: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:258: Dom18 PCI link 1 changed 0 -> 10
(XEN) HVM18: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:258: Dom18 PCI link 2 changed 0 -> 11
(XEN) HVM18: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:258: Dom18 PCI link 3 changed 0 -> 5
(XEN) HVM18: PCI-ISA link 3 routed to IRQ5
(XEN) HVM18: pci dev 01:2 INTD->IRQ5
(XEN) HVM18: pci dev 01:3 INTA->IRQ10
(XEN) HVM18: pci dev 03:0 INTA->IRQ5
(XEN) HVM18: pci dev 04:0 INTA->IRQ5
(XEN) HVM18: pci dev 05:0 INTA->IRQ10
(XEN) HVM18: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM18: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM18: pci dev 04:0 bar 10 size 00004000: f3000004
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=f2720 nr_mfns=4
(XEN) HVM18: pci dev 05:0 bar 10 size 00002000: f3004004
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3004 mfn=f2400 nr_mfns=2
(XEN) HVM18: pci dev 02:0 bar 14 size 00001000: f3006000
(XEN) HVM18: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) HVM18: pci dev 01:2 bar 20 size 00000020: 0000c101
(XEN) HVM18: pci dev 01:1 bar 20 size 00000010: 0000c121
(XEN) HVM18: Multiprocessor initialisation:
(XEN) HVM18:  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(XEN) HVM18: Testing HVM environment:
(XEN) HVM18:  - REP INSB across page boundaries ... passed
(XEN) HVM18:  - GS base MSRs and SWAPGS ... passed
(XEN) HVM18: Passed 2 of 2 tests
(XEN) HVM18: Writing SMBIOS tables ...
(XEN) HVM18: Loading ROMBIOS ...
(XEN) HVM18: 9660 bytes of ROMBIOS high-memory extensions:
(XEN) HVM18:   Relocating to 0xfc000000-0xfc0025bc ... done
(XEN) HVM18: Creating MP tables ...
(XEN) HVM18: Loading Cirrus VGABIOS ...
(XEN) HVM18: Loading ACPI ...
(XEN) HVM18:  - Lo data: 000ea020-000ea04f
(XEN) HVM18:  - Hi data: fc002800-fc01291f
(XEN) HVM18: vm86 TSS at fc012c00
(XEN) HVM18: BIOS map:
(XEN) HVM18:  c0000-c8fff: VGA BIOS
(XEN) HVM18:  eb000-eb158: SMBIOS tables
(XEN) HVM18:  f0000-fffff: Main BIOS
(XEN) HVM18: E820 table:
(XEN) HVM18:  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(XEN) HVM18:  [01]: 00000000:0009e000 - 00000000:0009fc00: RESERVED
(XEN) HVM18:  [02]: 00000000:0009fc00 - 00000000:000a0000: RESERVED
(XEN) HVM18:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM18:  [03]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM18:  [04]: 00000000:00100000 - 00000000:40000000: RAM
(XEN) HVM18:  HOLE: 00000000:40000000 - 00000000:fc000000
(XEN) HVM18:  [05]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM18: Invoking ROMBIOS ...
(XEN) HVM18: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) stdvga.c:147:d18 entering stdvga and caching modes
(XEN) HVM18: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $
(XEN) HVM18: Bochs BIOS - build: 06/23/99
(XEN) HVM18: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) HVM18: Options: apmbios pcibios eltorito PMM 
(XEN) HVM18: 
(XEN) HVM18: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM18: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (8192 MBytes)
(XEN) HVM18: IDE time out
(XEN) HVM18: 
(XEN) HVM18: 
(XEN) HVM18: 
(XEN) HVM18: Press F12 for boot menu.
(XEN) HVM18: 
(XEN) HVM18: Booting from Hard Disk...
(XEN) HVM18: Booting from 0000:7c00
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 08, unmapped device for ELDL=81
(XEN) HVM18: *** int 15h function AX=00c0, BX=0000 not yet supported!
(XEN) HVM18: *** int 15h function AX=ec00, BX=0002 not yet supported!
(XEN) HVM18: KBD: unsupported int 16h function 03
(XEN) HVM18: *** int 15h function AX=e980, BX=0000 not yet supported!
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=82
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=82
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=83
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=83
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=84
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=84
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=85
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=85
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=86
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=86
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=87
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=87
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 88
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 88
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 89
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 89
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8a
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8a
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8b
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8b
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8c
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8c
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8d
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8d
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8e
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8e
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8f
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8f
(XEN) stdvga.c:151:d18 leaving stdvga
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x30
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x20
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x20
(XEN) irq.c:258: Dom18 PCI link 0 changed 5 -> 0
(XEN) irq.c:258: Dom18 PCI link 1 changed 10 -> 0
(XEN) irq.c:258: Dom18 PCI link 2 changed 11 -> 0
(XEN) irq.c:258: Dom18 PCI link 3 changed 5 -> 0
(XEN) [VT-D]io.c:327: d18: unbind: m_gsi=0 g_gsi=36 device=5 intx=0
(XEN) [VT-D]io.c:386: d18 unmap: m_irq=0 device=5 intx=0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=17 g_gsi=36 device=5 intx=0
(XEN) domctl.c:920:d0 pt_irq_create_bind failed!
(XEN) irq.c:1590: dom18: forcing unbind of pirq 0
(XEN) [VT-D]io.c:327: d18: unbind: m_gsi=0 g_gsi=32 device=4 intx=0
(XEN) irq.c:1856: dom18: pirq 0 not mapped
(XEN) [VT-D]io.c:386: d18 unmap: m_irq=0 device=4 intx=0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=16 g_gsi=32 device=4 intx=0
(XEN) domctl.c:920:d0 pt_irq_create_bind failed!
(XEN) irq.c:1856: dom18: pirq 0 not mapped


[-- Attachment #3: qemu-dm-OpenSuseHVM.log --]
[-- Type: text/x-log, Size: 4425 bytes --]

domid: 18
Strip off blktap sub-type prefix to /home/VMs/openSuse11.3.img (drv 'aio')
Using file /home/VMs/openSuse11.3.img in read-write mode
Watching /local/domain/0/device-model/18/logdirty/cmd
Watching /local/domain/0/device-model/18/command
Watching /local/domain/18/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 53580b33-0ad5-8aa1-5736-93a294fd0a6c
Time offset set 0
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/18/xen_extended_power_mgmt): read error
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 00:1b.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init: No such device: setup io multiplexing failed! 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf2720004)
pt_msi_setup: pt_msi_setup requested pirq = 0
pt_msi_setup: msi mapped with pirq 0
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
vcpu-set: watch node error.
xs_read(/local/domain/18/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/18/log-throttling'
medium change watch on `/local/domain/18/log-throttling' - unknown device, ignored
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 10:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init: No such device: setup io multiplexing failed! 0x10:0x0.0x0
pt_register_regions: IO region registered (size=0x00002000 base_addr=0xf2400004)
pt_msi_setup: pt_msi_setup requested pirq = 0
pt_msi_setup: msi mapped with pirq 0
pci_intx: intx=1
register_real_device: Real physical device 10:00.0 registered successfuly!
IRQ type = MSI-INTx
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f2720000 type=0 len=16384 index=0 first_map=1
pt_iomem_map: e_phys=f3004000 maddr=f2400000 type=0 len=8192 index=0 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:04.0][Offset:30h][Length:4]
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:30h][Length:4]
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 0 gvec 49 gflags 0
pt_msi_update: Error: Binding of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 28
pt_msgctrl_reg_write: setup msi for dev 28
pt_msi_setup: msi mapped with pirq 37
pt_msi_update: Update msi with pirq 37 gvec 49 gflags 0
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 0 gvec 59 gflags 0
pt_msi_update: Error: Binding of MSI failed.
pt_msi_update: Error: Unmapping of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 20
pt_msgctrl_reg_write: setup msi for dev 20
pt_msi_setup: msi mapped with pirq 36
pt_msi_update: Update msi with pirq 36 gvec 59 gflags 0

[-- Attachment #4: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users