* Issue with MSI in a HVM domU with several passed through PCI devices
@ 2012-06-18 20:57 Rolu
2012-06-19 9:45 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Rolu @ 2012-06-18 20:57 UTC (permalink / raw)
To: xen-devel
Hi,
I have a HVM domU running Ubuntu 12.04. It has several PCI devices
passed through to it (USB controllers, onboard sound, ATI video card
with gfx_passthru 0). It has some issues, the most apparent one with
the sound, which doesn't play normally but repeats the same short
fragment over and over for a while until moving on to the next
fragment. The Xen dmesg prints lots of lines like:
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
The number behind 108:d varies occasionally.
Since vmsi.c seems to deal with passing on MSI interrupts, I tried
booting the domU with pci=nomsi. This made things work and sound now
plays normally. Not using MSI at all doesn't seem like a very good
solution though.
What makes this happen, and can I fix it in a better way?
Data:
Xen: 4.2 unstable rev. 25483
Dom0 Ubuntu 12.04 3.4.0-030400-generic #201205210521
----- Xen dmesg when starting the domU without pci=nomsi:
(XEN) [2012-06-18 20:13:35] memory.c:131:d0 Could not allocate
order=18 extent: id=3 memflags=0 (0 of 1)
(XEN) [2012-06-18 20:13:37] physdev.c:174: dom3: 16:-1 already mapped to 16
(XEN) [2012-06-18 20:13:37] physdev.c:174: dom3: 16:-1 already mapped to 16
(XEN) [2012-06-18 20:13:37] HVM3: HVM Loader
(XEN) [2012-06-18 20:13:37] HVM3: Detected Xen v4.2-unstable
(XEN) [2012-06-18 20:13:37] HVM3: Xenbus rings @0xfeffc000, event channel 3
(XEN) [2012-06-18 20:13:37] HVM3: System requested ROMBIOS
(XEN) [2012-06-18 20:13:37] HVM3: CPU speed is 3300 MHz
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 0 changed 0 -> 5
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 0 routed to IRQ5
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 1 changed 0 -> 10
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 1 routed to IRQ10
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 2 changed 0 -> 11
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 2 routed to IRQ11
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 3 changed 0 -> 5
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 3 routed to IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:2 INTD->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:3 INTA->IRQ10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 05:0 INTA->IRQ10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 06:0 INTA->IRQ11
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 07:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 08:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 10 size 10000000: e000000c
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 18 size 00020000: f3000004
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 30 size 00020000: f3020000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 05:0 bar 10 size 00010000: f3040004
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 08:0 bar 10 size 00004000: f3050004
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 02:0 bar 14 size 00001000: f3054000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 06:0 bar 10 size 00001000: f3055000
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 07:0 bar 10 size 00001000: f3056000
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 bar 10 size 00000100: 0000c101
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 bar 14 size 00000100: f3057000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 20 size 00000100: 0000c201
(XEN) [2012-06-18 20:13:37] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:2 bar 20 size 00000020: 0000c301
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:1 bar 20 size 00000010: 0000c321
(XEN) [2012-06-18 20:13:37] HVM3: Multiprocessor initialisation:
(XEN) [2012-06-18 20:13:37] HVM3: - CPU0 ... 36-bit phys ... fixed
MTRRs ... var MTRRs [3/8] ... done.
(XEN) [2012-06-18 20:13:37] HVM3: Testing HVM environment:
(XEN) [2012-06-18 20:13:37] HVM3: - REP INSB across page boundaries ... passed
(XEN) [2012-06-18 20:13:37] HVM3: - GS base MSRs and SWAPGS ... passed
(XEN) [2012-06-18 20:13:37] HVM3: Passed 2 of 2 tests
(XEN) [2012-06-18 20:13:37] HVM3: Writing SMBIOS tables ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading ROMBIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: 12636 bytes of ROMBIOS high-memory extensions:
(XEN) [2012-06-18 20:13:37] HVM3: Relocating to 0xfc001000-0xfc00415c ... done
(XEN) [2012-06-18 20:13:37] HVM3: Creating MP tables ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading Cirrus VGABIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading PCI Option ROM ...
(XEN) [2012-06-18 20:13:37] HVM3: - Manufacturer: http://ipxe.org
(XEN) [2012-06-18 20:13:37] HVM3: - Product name: iPXE
(XEN) [2012-06-18 20:13:37] HVM3: Option ROMs:
(XEN) [2012-06-18 20:13:37] HVM3: c0000-c8fff: VGA BIOS
(XEN) [2012-06-18 20:13:37] HVM3: c9000-d8fff: Etherboot ROM
(XEN) [2012-06-18 20:13:37] HVM3: Loading ACPI ...
(XEN) [2012-06-18 20:13:37] HVM3: vm86 TSS at fc010280
(XEN) [2012-06-18 20:13:37] HVM3: BIOS map:
(XEN) [2012-06-18 20:13:37] HVM3: f0000-fffff: Main BIOS
(XEN) [2012-06-18 20:13:37] HVM3: E820 table:
(XEN) [2012-06-18 20:13:37] HVM3: [00]: 00000000:00000000 -
00000000:0009e000: RAM
(XEN) [2012-06-18 20:13:37] HVM3: [01]: 00000000:0009e000 -
00000000:000a0000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3: HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) [2012-06-18 20:13:37] HVM3: [02]: 00000000:000e0000 -
00000000:00100000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3: [03]: 00000000:00100000 -
00000000:e0000000: RAM
(XEN) [2012-06-18 20:13:37] HVM3: HOLE: 00000000:e0000000 - 00000000:fc000000
(XEN) [2012-06-18 20:13:37] HVM3: [04]: 00000000:fc000000 -
00000001:00000000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3: [05]: 00000001:00000000 -
00000002:9f800000: RAM
(XEN) [2012-06-18 20:13:37] HVM3: Invoking ROMBIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: $Revision: 1.221 $ $Date: 2008/12/07
17:32:29 $
(XEN) [2012-06-18 20:13:38] stdvga.c:147:d3 entering stdvga and caching modes
(XEN) [2012-06-18 20:13:38] HVM3: VGABios $Id: vgabios.c,v 1.67
2008/01/27 09:44:12 vruppert Exp $
(XEN) [2012-06-18 20:13:38] HVM3: Bochs BIOS - build: 06/23/99
(XEN) [2012-06-18 20:13:38] HVM3: $Revision: 1.221 $ $Date: 2008/12/07
17:32:29 $
(XEN) [2012-06-18 20:13:38] HVM3: Options: apmbios pcibios eltorito PMM
(XEN) [2012-06-18 20:13:38] HVM3:
(XEN) [2012-06-18 20:13:38] HVM3: ata0-0: PCHS=16383/16/63
translation=lba LCHS=1024/255/63
(XEN) [2012-06-18 20:13:38] HVM3: ata0 master: QEMU HARDDISK ATA-7
Hard-Disk (30720 MBytes)
(XEN) [2012-06-18 20:13:40] HVM3: IDE time out
(XEN) [2012-06-18 20:13:40] HVM3: ata1 master: QEMU DVD-ROM ATAPI-4
CD-Rom/DVD-Rom
(XEN) [2012-06-18 20:13:41] HVM3: IDE time out
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3: Press F12 for boot menu.
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3: Booting from Hard Disk...
(XEN) [2012-06-18 20:13:41] HVM3: Booting from 0000:7c00
(XEN) [2012-06-18 20:13:43] stdvga.c:151:d3 leaving stdvga
(XEN) [2012-06-18 20:13:55] irq.c:350: Dom3 callback via changed to
Direct Vector 0xf3
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 0 changed 5 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 1 changed 10 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 2 changed 11 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 3 changed 5 -> 0
(XEN) [2012-06-18 20:13:56] irq.c:2234: dom3: pirq 49 or emuirq 23
already mapped
(XEN) [2012-06-18 20:13:56] vmsi.c:108:d3 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:02] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:21] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:51] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:24:36] vmsi.c:108:d32767 Unsupported delivery mode 3
----- Xen dmesg when starting dom0:
__ __ _ _ ____ _ _ _
\ \/ /___ _ __ | || | |___ \ _ _ _ __ ___| |_ __ _| |__ | | ___
\ // _ \ '_ \ | || |_ __) |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
/ \ __/ | | | |__ _| / __/|__| |_| | | | \__ \ || (_| | |_) | | __/
/_/\_\___|_| |_| |_|(_)_____| \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
(XEN) Xen version 4.2-unstable (root@roce.org) (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5) ) Fri Jun 15 05:32:29 CEST 2012
(XEN) Latest ChangeSet: Thu Jun 14 16:05:42 2012 +0100 25483:9dffb1864f2c
(XEN) Bootloader: GRUB 1.99-21ubuntu3.1
(XEN) Command line: placeholder xsave=0 loglvl=all guest_loglvl=all
console_timestamps=1 noreboot com1=115200,8n1,0x3f8,4 console=com1,vga
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 3 MBR signatures
(XEN) Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009d800 (usable)
(XEN) 000000000009d800 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 0000000020000000 (usable)
(XEN) 0000000020000000 - 0000000020200000 (reserved)
(XEN) 0000000020200000 - 0000000040004000 (usable)
(XEN) 0000000040004000 - 0000000040005000 (reserved)
(XEN) 0000000040005000 - 00000000bd8a7000 (usable)
(XEN) 00000000bd8a7000 - 00000000bde1f000 (reserved)
(XEN) 00000000bde1f000 - 00000000be09f000 (ACPI NVS)
(XEN) 00000000be09f000 - 00000000be0a4000 (ACPI data)
(XEN) 00000000be0a4000 - 00000000be0e7000 (ACPI NVS)
(XEN) 00000000be0e7000 - 00000000beb81000 (usable)
(XEN) 00000000beb81000 - 00000000beff2000 (reserved)
(XEN) 00000000beff2000 - 00000000bf000000 (usable)
(XEN) 00000000bf800000 - 00000000cfa00000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec01000 (reserved)
(XEN) 00000000fed00000 - 00000000fed04000 (reserved)
(XEN) 00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 000000042f600000 (usable)
(XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT BE084080, 0084 (r1 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: FACP BE08E218, 00F4 (r4 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: DSDT BE0841A0, A077 (r2 ALASKA A M I 14 INTL 20051117)
(XEN) ACPI: FACS BE09DF80, 0040
(XEN) ACPI: APIC BE08E310, 0072 (r3 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: ASF! BE08E388, 00A5 (r32 INTEL HCG 1 TFSM F4240)
(XEN) ACPI: MCFG BE08E430, 003C (r1 ALASKA A M I 1072009 MSFT 97)
(XEN) ACPI: SSDT BE08E470, 04A6 (r1 Intel_ AoacTabl 1000 INTL 20091112)
(XEN) ACPI: AAFT BE08E918, 00C2 (r1 ALASKA OEMAAFT 1072009 MSFT 97)
(XEN) ACPI: HPET BE08E9E0, 0038 (r1 ALASKA A M I 1072009 AMI. 5)
(XEN) ACPI: SSDT BE08EA18, 036D (r1 SataRe SataTabl 1000 INTL 20091112)
(XEN) ACPI: SSDT BE08ED88, 09AA (r1 PmRef Cpu0Ist 3000 INTL 20051117)
(XEN) ACPI: SSDT BE08F738, 0A92 (r1 PmRef CpuPm 3000 INTL 20051117)
(XEN) ACPI: DMAR BE0901D0, 00B8 (r1 INTEL SNB 1 INTL 1)
(XEN) ACPI: BGRT BE090288, 003C (r0 ALASKA A M I 1072009 AMI 10013)
(XEN) System RAM: 16086MB (16473004kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000042f600000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fceb0
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
be09df80/0000000000000000, using 32
(XEN) ACPI: wakeup_vec[be09df8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 7:10 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) Table is not found!
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3300.119 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank
0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) [2012-06-18 20:50:08] Platform timer is 14.318MHz HPET
(XEN) [2012-06-18 20:50:08] Allocated console ring of 32 KiB.
(XEN) [2012-06-18 20:50:08] VMX: Supported advanced features:
(XEN) [2012-06-18 20:50:08] - APIC MMIO access virtualisation
(XEN) [2012-06-18 20:50:08] - APIC TPR shadow
(XEN) [2012-06-18 20:50:08] - Extended Page Tables (EPT)
(XEN) [2012-06-18 20:50:08] - Virtual-Processor Identifiers (VPID)
(XEN) [2012-06-18 20:50:08] - Virtual NMI
(XEN) [2012-06-18 20:50:08] - MSR direct-access bitmap
(XEN) [2012-06-18 20:50:08] - Unrestricted Guest
(XEN) [2012-06-18 20:50:08] HVM: ASIDs enabled.
(XEN) [2012-06-18 20:50:08] HVM: VMX enabled
(XEN) [2012-06-18 20:50:08] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2012-06-18 20:50:08] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2012-06-18 20:50:08] Brought up 4 CPUs
(XEN) [2012-06-18 20:50:08] ACPI sleep modes: S3
(XEN) [2012-06-18 20:50:08] mcheck_poll: Machine check polling timer started.
(XEN) [2012-06-18 20:50:08] *** LOADING DOMAIN 0 ***
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1000000
memsz=0xad7000
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1c00000
memsz=0xd90e8
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1cda000
memsz=0x14480
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1cef000
memsz=0x656000
(XEN) [2012-06-18 20:50:08] elf_parse_binary: memory: 0x1000000 -> 0x2345000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: ENTRY = 0xffffffff81cef200
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: HYPERCALL_PAGE =
0xffffffff81001000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: LOADER = "generic"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: HV_START_LOW =
0xffff800000000000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [2012-06-18 20:50:08] elf_xen_addr_calc_check: addresses:
(XEN) [2012-06-18 20:50:08] virt_base = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08] elf_paddr_offset = 0x0
(XEN) [2012-06-18 20:50:08] virt_offset = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08] virt_kstart = 0xffffffff81000000
(XEN) [2012-06-18 20:50:08] virt_kend = 0xffffffff82345000
(XEN) [2012-06-18 20:50:08] virt_entry = 0xffffffff81cef200
(XEN) [2012-06-18 20:50:08] p2m_base = 0xffffffffffffffff
(XEN) [2012-06-18 20:50:08] Xen kernel: 64-bit, lsb, compat32
(XEN) [2012-06-18 20:50:08] Dom0 kernel: 64-bit, PAE, lsb, paddr
0x1000000 -> 0x2345000
(XEN) [2012-06-18 20:50:08] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [2012-06-18 20:50:08] Dom0 alloc.:
0000000418000000->0000000420000000 (3987939 pages to be allocated)
(XEN) [2012-06-18 20:50:08] Init. ramdisk: 000000042cff7000->000000042f5ffa00
(XEN) [2012-06-18 20:50:08] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [2012-06-18 20:50:08] Loaded kernel: ffffffff81000000->ffffffff82345000
(XEN) [2012-06-18 20:50:08] Init. ramdisk: ffffffff82345000->ffffffff8494da00
(XEN) [2012-06-18 20:50:08] Phys-Mach map: ffffffff8494e000->ffffffff8680df60
(XEN) [2012-06-18 20:50:08] Start info: ffffffff8680e000->ffffffff8680e4b4
(XEN) [2012-06-18 20:50:08] Page tables: ffffffff8680f000->ffffffff86848000
(XEN) [2012-06-18 20:50:08] Boot stack: ffffffff86848000->ffffffff86849000
(XEN) [2012-06-18 20:50:08] TOTAL: ffffffff80000000->ffffffff86c00000
(XEN) [2012-06-18 20:50:08] ENTRY ADDRESS: ffffffff81cef200
(XEN) [2012-06-18 20:50:08] Dom0 has maximum 4 VCPUs
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 0 at
0xffffffff81000000 -> 0xffffffff81ad7000
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 1 at
0xffffffff81c00000 -> 0xffffffff81cd90e8
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 2 at
0xffffffff81cda000 -> 0xffffffff81cee480
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 3 at
0xffffffff81cef000 -> 0xffffffff81dc6000
(XEN) [2012-06-18 20:50:11] Scrubbing Free RAM: .done.
(XEN) [2012-06-18 20:50:11] Initial low memory virq threshold set at
0x4000 pages.
(XEN) [2012-06-18 20:50:11] Std. Loglevel: All
(XEN) [2012-06-18 20:50:11] Guest Loglevel: All
(XEN) [2012-06-18 20:50:11] Xen is relinquishing VGA console.
(XEN) [2012-06-18 20:50:11] *** Serial input -> DOM0 (type 'CTRL-a'
three times to switch input to Xen)
(XEN) [2012-06-18 20:50:11] Freed 244kB init memory.
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:01.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:02.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:14.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:16.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1a.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1b.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.4
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.5
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.6
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.7
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1d.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.2
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.3
(XEN) [2012-06-18 20:50:12] PCI add device 0000:01:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:01:00.1
(XEN) [2012-06-18 20:50:12] PCI add device 0000:03:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:04:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:05:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:06:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:01.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:02.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:03.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:0a:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:0b:02.0
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 5
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 6
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 7
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 8
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-18 20:57 Issue with MSI in a HVM domU with several passed through PCI devices Rolu
@ 2012-06-19 9:45 ` Jan Beulich
2012-06-20 13:58 ` Rolu
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2012-06-19 9:45 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel
>>> On 18.06.12 at 22:57, Rolu <rolu@roce.org> wrote:
> I have a HVM domU running Ubuntu 12.04. It has several PCI devices
> passed through to it (USB controllers, onboard sound, ATI video card
> with gfx_passthru 0). It has some issues, the most apparent one with
> the sound, which doesn't play normally but repeats the same short
> fragment over and over for a while until moving on to the next
> fragment. The Xen dmesg prints lots of lines like:
>
> (XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
Which indicates something wrong in the guest kernel or in the
translation layers (qemu, less likely the hypervisor itself). Valid
MSIs can't have a delivery mode of 3 (a reserved encoding).
> The number behind 108:d varies occasionally.
Yes, that number is bogus (has no guaranteed relation to the
target domain).
> Since vmsi.c seems to deal with passing on MSI interrupts, I tried
> booting the domU with pci=nomsi. This made things work and sound now
> plays normally. Not using MSI at all doesn't seem like a very good
> solution though.
>
> What makes this happen, and can I fix it in a better way?
Unless this is in the guest kernel, we'll likely need a code fix here,
but for determining what and where, we'd need you to provide
the qemu log for the domain as well (there ought to be entries
starting with "Update msi with pirq", and the gflags value is of
particular interest). Depending on what we see, we may then
need you to do some further debugging.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-19 9:45 ` Jan Beulich
@ 2012-06-20 13:58 ` Rolu
2012-06-20 16:03 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Rolu @ 2012-06-20 13:58 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 18.06.12 at 22:57, Rolu <rolu@roce.org> wrote:
>> I have a HVM domU running Ubuntu 12.04. It has several PCI devices
>> passed through to it (USB controllers, onboard sound, ATI video card
>> with gfx_passthru 0). It has some issues, the most apparent one with
>> the sound, which doesn't play normally but repeats the same short
>> fragment over and over for a while until moving on to the next
>> fragment. The Xen dmesg prints lots of lines like:
>>
>> (XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
>
> Which indicates something wrong in the guest kernel or in the
> translation layers (qemu, less likely the hypervisor itself). Valid
> MSIs can't have a delivery mode of 3 (a reserved encoding).
>
>> The number behind 108:d varies occasionally.
>
> Yes, that number is bogus (has no guaranteed relation to the
> target domain).
>
>> Since vmsi.c seems to deal with passing on MSI interrupts, I tried
>> booting the domU with pci=nomsi. This made things work and sound now
>> plays normally. Not using MSI at all doesn't seem like a very good
>> solution though.
>>
>> What makes this happen, and can I fix it in a better way?
>
> Unless this is in the guest kernel, we'll likely need a code fix here,
> but for determining what and where, we'd need you to provide
> the qemu log for the domain as well (there ought to be entries
> starting with "Update msi with pirq", and the gflags value is of
> particular interest). Depending on what we see, we may then
> need you to do some further debugging.
>
There are, these: (I've copied the full logs below)
pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
To produce the logs I started the domU and let it run until it
displayed the graphical login screen, which plays a short sound.
The qemu logs give several errors and warnings, such as (there are
multiple of each of these):
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
Are these related, and/or cause for worry? I've looked around some but
apart from the fact that they have to do with PCI it doesn't tell me
much. They occur no matter whether I use pci=nomsi or not.
The qemu log, without pci=nomsi set, sound stutters:
domid: 3
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
Strip off blktap sub-type prefix to /panda/ubuntuhv2.img (drv 'aio')
Using file /panda/ubuntuhv2.img in read-write mode
Strip off blktap sub-type prefix to
/panda/ubuntu-12.04-desktop-amd64.iso (drv 'aio')
Using file /panda/ubuntu-12.04-desktop-amd64.iso in read-only mode
Watching /local/domain/0/device-model/3/logdirty/cmd
Watching /local/domain/0/device-model/3/command
Watching /local/domain/3/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = bfd25022-34fc-43e9-82ed-70fb4fdbd7bd
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/3/xen_extended_power_mgmt): read error
medium change watch on `hdc' (index: 1):
aio:/panda/ubuntu-12.04-desktop-amd64.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
vcpu-set: watch node error.
xs_read(/local/domain/3/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/3/log-throttling'
medium change watch on `/local/domain/3/log-throttling' - unknown
device, ignored
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:14.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x14.0x0
pt_register_regions: IO region registered (size=0x00010000 base_addr=0xf7d00004)
pt_msi_setup: msi mapped with pirq 37
pci_intx: intx=1
register_real_device: Real physical device 00:14.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1a.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1a.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d18000)
pci_intx: intx=1
register_real_device: Real physical device 00:1a.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1d.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1d.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d17000)
pci_intx: intx=1
register_real_device: Real physical device 00:1d.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1b.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf7d10004)
pt_msi_setup: msi mapped with pirq 36
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00020000 base_addr=0xf7c20004)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000e001)
pt_register_regions: Expansion ROM registered (size=0x00020000
base_addr=0xf7c00000)
pt_msi_setup: msi mapped with pirq 35
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = MSI-INTx
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=1
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=1
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=1
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=1
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=1
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
Unknown PV product 3 loaded in guest
PV driver build 1
region type 1 at [c100,c200).
region type 0 at [f3057000,f3057100).
squash iomem [f3057000, f3057100).
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:06:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:07:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_pci_write_config: [00:08:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
The qemu log, with pci=nomsi set, sound is okay:
domid: 2
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
Strip off blktap sub-type prefix to /panda/ubuntuhv2.img (drv 'aio')
Using file /panda/ubuntuhv2.img in read-write mode
Strip off blktap sub-type prefix to
/panda/ubuntu-12.04-desktop-amd64.iso (drv 'aio')
Using file /panda/ubuntu-12.04-desktop-amd64.iso in read-only mode
Watching /local/domain/0/device-model/2/logdirty/cmd
Watching /local/domain/0/device-model/2/command
Watching /local/domain/2/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 376e7089-c20e-40d9-8e93-c7e4bec2e72d
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/2/xen_extended_power_mgmt): read error
medium change watch on `hdc' (index: 1):
aio:/panda/ubuntu-12.04-desktop-amd64.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
vcpu-set: watch node error.
xs_read(/local/domain/2/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/2/log-throttling'
medium change watch on `/local/domain/2/log-throttling' - unknown
device, ignored
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:14.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x14.0x0
pt_register_regions: IO region registered (size=0x00010000 base_addr=0xf7d00004)
pt_msi_setup: msi mapped with pirq 37
pci_intx: intx=1
register_real_device: Real physical device 00:14.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1a.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1a.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d18000)
pci_intx: intx=1
register_real_device: Real physical device 00:1a.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1d.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1d.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d17000)
pci_intx: intx=1
register_real_device: Real physical device 00:1d.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1b.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf7d10004)
pt_msi_setup: msi mapped with pirq 36
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
pt_iomul_init: Error: pt_iomul_init can't open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00020000 base_addr=0xf7c20004)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000e001)
pt_register_regions: Expansion ROM registered (size=0x00020000
base_addr=0xf7c00000)
pt_msi_setup: msi mapped with pirq 35
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = MSI-INTx
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=1
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=1
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=1
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=1
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=1
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
Unknown PV product 3 loaded in guest
PV driver build 1
region type 1 at [c100,c200).
region type 0 at [f3057000,f3057100).
squash iomem [f3057000, f3057100).
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:06:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:07:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_pci_write_config: [00:08:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-20 13:58 ` Rolu
@ 2012-06-20 16:03 ` Jan Beulich
2012-06-24 2:21 ` Rolu
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2012-06-20 16:03 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel
>>> On 20.06.12 at 15:58, Rolu <rolu@roce.org> wrote:
> On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> Unless this is in the guest kernel, we'll likely need a code fix here,
>> but for determining what and where, we'd need you to provide
>> the qemu log for the domain as well (there ought to be entries
>> starting with "Update msi with pirq", and the gflags value is of
>> particular interest). Depending on what we see, we may then
>> need you to do some further debugging.
>>
>
> There are, these: (I've copied the full logs below)
>
> pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
> pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
> pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
Okay, so at that point the bad value is already there. I'd
suggest taking it up the usage chain, so adding some logging
in pt_msgdata_reg_write() (where the original value -
ptdev->msi->data - is being computed) would likely be a
good first step.
At the same time, adding logging to the guest kernel would
be nice, to see what value it actually writes (in a current
kernel this would be in __write_msi_msg()).
> The qemu logs give several errors and warnings, such as (there are
> multiple of each of these):
>
> pt_iomul_init: Error: pt_iomul_init can't open file
> /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
> pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
> to unused Base Address Register. [Offset:30h][Length:4]
> pt_pci_read_config: [00:10:0] Error: Failed to read register with
> offset exceeding FFh. [Offset:ffh][Length:1]
>
> Are these related, and/or cause for worry? I've looked around some but
> apart from the fact that they have to do with PCI it doesn't tell me
> much. They occur no matter whether I use pci=nomsi or not.
I don't know.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-20 16:03 ` Jan Beulich
@ 2012-06-24 2:21 ` Rolu
2012-06-25 7:49 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Rolu @ 2012-06-24 2:21 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 20.06.12 at 15:58, Rolu <rolu@roce.org> wrote:
>> On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>> Unless this is in the guest kernel, we'll likely need a code fix here,
>>> but for determining what and where, we'd need you to provide
>>> the qemu log for the domain as well (there ought to be entries
>>> starting with "Update msi with pirq", and the gflags value is of
>>> particular interest). Depending on what we see, we may then
>>> need you to do some further debugging.
>>>
>>
>> There are, these: (I've copied the full logs below)
>>
>> pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
>> pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
>> pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
>
> Okay, so at that point the bad value is already there. I'd
> suggest taking it up the usage chain, so adding some logging
> in pt_msgdata_reg_write() (where the original value -
> ptdev->msi->data - is being computed) would likely be a
> good first step.
>
This took a while as I'm not very familiar with it all yet.
The write function gets passed a *value of 0x4300 and that is also
what gets assigned to ptdev->msi->data. The delivery mode is bit 8 to
10 of that value, which is indeed 3. lspci -vvv on the domU also shows
that the MSI data field is 4300, but I suppose it just reads that back
from qemu. Anyway, this seems in order, at least as far as I can see.
> At the same time, adding logging to the guest kernel would
> be nice, to see what value it actually writes (in a current
> kernel this would be in __write_msi_msg()).
>
Turns out that msg->data here is also 0x4300, so it seems the guest
kernel is producing these values. I caused it to make a stack trace
and this pointed back to xen_hvm_setup_msi_irqs. This function uses
the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
current data field and if it isn't equal to the macro it uses
xen_msi_compose_msg to make a new message, but that function just sets
the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
then gets passed to __write_msi_msg and that's that. There are no
other writes through __write_msi_msg (except for the same thing for
other devices).
The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
decoded as the delivery mode, so it seems the kernel is intentionally
setting it to 3.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-24 2:21 ` Rolu
@ 2012-06-25 7:49 ` Jan Beulich
2012-06-25 11:38 ` Stefano Stabellini
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2012-06-25 7:49 UTC (permalink / raw)
To: Stefano Stabellini, Rolu; +Cc: xen-devel
>>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> At the same time, adding logging to the guest kernel would
>> be nice, to see what value it actually writes (in a current
>> kernel this would be in __write_msi_msg()).
>>
>
> Turns out that msg->data here is also 0x4300, so it seems the guest
> kernel is producing these values. I caused it to make a stack trace
> and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> current data field and if it isn't equal to the macro it uses
> xen_msi_compose_msg to make a new message, but that function just sets
> the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> then gets passed to __write_msi_msg and that's that. There are no
> other writes through __write_msi_msg (except for the same thing for
> other devices).
>
> The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
> decoded as the delivery mode, so it seems the kernel is intentionally
> setting it to 3.
So that can never have worked properly afaict. Stefano, the
code as it is currently - using literal (3 << 8) - is clearly bogus.
Your original commit at least had a comment saying that the
reserved delivery mode encoding is intentional here, but that
comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
In any case - the cooperation with qemu apparently doesn't
work, as the reserved encoding should never make it through
to the hypervisor. Could you explain what the intention here
was?
And regardless of anything, can the literal numbers please be
replaced by proper manifest constants - the "8" here already
has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
proper symbolic would permit locating where this is being (or
really, as it doesn't appear to work supposed to be) consumed
in qemu, provided it uses the same definition (i.e. that one
should go into one of the public headers).
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-25 7:49 ` Jan Beulich
@ 2012-06-25 11:38 ` Stefano Stabellini
2012-06-26 2:31 ` Rolu
0 siblings, 1 reply; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-25 11:38 UTC (permalink / raw)
To: Jan Beulich; +Cc: Rolu, xen-devel, Stefano Stabellini
On Mon, 25 Jun 2012, Jan Beulich wrote:
> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >> At the same time, adding logging to the guest kernel would
> >> be nice, to see what value it actually writes (in a current
> >> kernel this would be in __write_msi_msg()).
> >>
> >
> > Turns out that msg->data here is also 0x4300, so it seems the guest
> > kernel is producing these values. I caused it to make a stack trace
> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> > current data field and if it isn't equal to the macro it uses
> > xen_msi_compose_msg to make a new message, but that function just sets
> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> > then gets passed to __write_msi_msg and that's that. There are no
> > other writes through __write_msi_msg (except for the same thing for
> > other devices).
> >
> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
> > decoded as the delivery mode, so it seems the kernel is intentionally
> > setting it to 3.
>
> So that can never have worked properly afaict. Stefano, the
> code as it is currently - using literal (3 << 8) - is clearly bogus.
> Your original commit at least had a comment saying that the
> reserved delivery mode encoding is intentional here, but that
> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
> In any case - the cooperation with qemu apparently doesn't
> work, as the reserved encoding should never make it through
> to the hypervisor. Could you explain what the intention here
> was?
>
> And regardless of anything, can the literal numbers please be
> replaced by proper manifest constants - the "8" here already
> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> proper symbolic would permit locating where this is being (or
> really, as it doesn't appear to work supposed to be) consumed
> in qemu, provided it uses the same definition (i.e. that one
> should go into one of the public headers).
The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
because notifications are not supposed to be delivered as MSI anymore.
This is what should happen:
1) Linux configures the device with a 0 vector number and the pirq number
in the address field;
2) QEMU notices a vector number of 0 and reads the pirq number from the
address field, passing it to xc_domain_update_msi_irq;
3) Xen assignes the given pirq to the physical MSI;
4) The guest issues a EVTCHNOP_bind_pirq hypercall;
5) Xen sets the pirq as "IRQ_PT";
6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
returns true so Xen calls send_guest_pirq instead.
Obviously 6) is not happening. hvm_domain_use_pirq is:
is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
above).
So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
(__startup_pirq doesn't get called), or Xen is erroring out in
map_domain_emuirq_pirq.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-25 11:38 ` Stefano Stabellini
@ 2012-06-26 2:31 ` Rolu
2012-06-26 7:42 ` Jan Beulich
2012-06-26 12:59 ` Stefano Stabellini
0 siblings, 2 replies; 20+ messages in thread
From: Rolu @ 2012-06-26 2:31 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Jan Beulich, xen-devel
On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
>> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >> At the same time, adding logging to the guest kernel would
>> >> be nice, to see what value it actually writes (in a current
>> >> kernel this would be in __write_msi_msg()).
>> >>
>> >
>> > Turns out that msg->data here is also 0x4300, so it seems the guest
>> > kernel is producing these values. I caused it to make a stack trace
>> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
>> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
>> > current data field and if it isn't equal to the macro it uses
>> > xen_msi_compose_msg to make a new message, but that function just sets
>> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
>> > then gets passed to __write_msi_msg and that's that. There are no
>> > other writes through __write_msi_msg (except for the same thing for
>> > other devices).
>> >
>> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
>> > decoded as the delivery mode, so it seems the kernel is intentionally
>> > setting it to 3.
>>
>> So that can never have worked properly afaict. Stefano, the
>> code as it is currently - using literal (3 << 8) - is clearly bogus.
>> Your original commit at least had a comment saying that the
>> reserved delivery mode encoding is intentional here, but that
>> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
>> In any case - the cooperation with qemu apparently doesn't
>> work, as the reserved encoding should never make it through
>> to the hypervisor. Could you explain what the intention here
>> was?
>>
>> And regardless of anything, can the literal numbers please be
>> replaced by proper manifest constants - the "8" here already
>> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> proper symbolic would permit locating where this is being (or
>> really, as it doesn't appear to work supposed to be) consumed
>> in qemu, provided it uses the same definition (i.e. that one
>> should go into one of the public headers).
>
> The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
> because notifications are not supposed to be delivered as MSI anymore.
>
> This is what should happen:
>
> 1) Linux configures the device with a 0 vector number and the pirq number
> in the address field;
>
> 2) QEMU notices a vector number of 0 and reads the pirq number from the
> address field, passing it to xc_domain_update_msi_irq;
>
> 3) Xen assignes the given pirq to the physical MSI;
>
> 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>
> 5) Xen sets the pirq as "IRQ_PT";
>
> 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
> returns true so Xen calls send_guest_pirq instead.
>
>
> Obviously 6) is not happening. hvm_domain_use_pirq is:
>
> is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
>
> My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
> above).
This appears to be true. I added logging to hvm_pci_msi_assert in
xen/drivers/passthrough/io.c and it indicates that
pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
before an unsupported delivery mode message.
I also log pirq->pirq but I found that most of the time I can't find
this value anywhere else (I'm not sure how to interpret the value,
though). For example, in my last try:
* I get an unsupported delivery mode error for pirq->pirq 55, 54 and
53. The vast majority are for 54.
* I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
but complains it's already mapped.
* I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
22, 52, 48, 47. Also never 54 or 53.
* map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
* The qemu log mentions pirq 35, 36 and 37
It seems pirq values don't always mean the same? Is it a coincidence
that 55 occurs almost everywhere, or is something going wrong with the
other two values (53 and 54 versus 16 and 17)?
I have three MSI capable devices passed through to the domU, and I do
see groups of three distinct pirqs in the data above - just not the
same ones in every place I look.
> So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
> (__startup_pirq doesn't get called), or Xen is erroring out in
> map_domain_emuirq_pirq.
evtchn_bind_pirq gets called, though I'm not sure if it is with the right data.
map_domain_emuirq_pirq always gets past the checks in the top half
(i.e. up to the line /* do not store emuirq mappings for pt devices
*/), except for one time with pirq=49,emuirq=23 where it finds they
are already mapped.
It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
This implies their info->arch.hvm.emuirq is also set to -2 (haven't
directly logged that but it's the only assignment there).
Interestingly, I get an unsupported delivery mode error for pirq 55
where my logging says pirq->arch.hvm.emuirq is -1, *after*
map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-26 2:31 ` Rolu
@ 2012-06-26 7:42 ` Jan Beulich
2012-06-26 12:59 ` Stefano Stabellini
1 sibling, 0 replies; 20+ messages in thread
From: Jan Beulich @ 2012-06-26 7:42 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel, Stefano Stabellini
>>> On 26.06.12 at 04:31, Rolu <rolu@roce.org> wrote:
> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>> This is what should happen:
>>
>> 1) Linux configures the device with a 0 vector number and the pirq number
>> in the address field;
>>
>> 2) QEMU notices a vector number of 0 and reads the pirq number from the
>> address field, passing it to xc_domain_update_msi_irq;
>>
>> 3) Xen assignes the given pirq to the physical MSI;
>>
>> 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>>
>> 5) Xen sets the pirq as "IRQ_PT";
>>
>> 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
>> returns true so Xen calls send_guest_pirq instead.
>>
>>
>> Obviously 6) is not happening. hvm_domain_use_pirq is:
>>
>> is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
>>
>> My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
>> above).
>
> This appears to be true. I added logging to hvm_pci_msi_assert in
> xen/drivers/passthrough/io.c and it indicates that
> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
> before an unsupported delivery mode message.
>
> I also log pirq->pirq but I found that most of the time I can't find
> this value anywhere else (I'm not sure how to interpret the value,
> though). For example, in my last try:
>
> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
> 53. The vast majority are for 54.
> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
> but complains it's already mapped.
> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
> 22, 52, 48, 47. Also never 54 or 53.
> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17,
> 55.
> * The qemu log mentions pirq 35, 36 and 37
>
> It seems pirq values don't always mean the same? Is it a coincidence
> that 55 occurs almost everywhere, or is something going wrong with the
> other two values (53 and 54 versus 16 and 17)?
>
> I have three MSI capable devices passed through to the domU, and I do
> see groups of three distinct pirqs in the data above - just not the
> same ones in every place I look.
>
>> So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
>> (__startup_pirq doesn't get called), or Xen is erroring out in
>> map_domain_emuirq_pirq.
>
> evtchn_bind_pirq gets called, though I'm not sure if it is with the right
> data.
>
> map_domain_emuirq_pirq always gets past the checks in the top half
> (i.e. up to the line /* do not store emuirq mappings for pt devices
> */), except for one time with pirq=49,emuirq=23 where it finds they
> are already mapped.
> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
> This implies their info->arch.hvm.emuirq is also set to -2 (haven't
> directly logged that but it's the only assignment there).
>
> Interestingly, I get an unsupported delivery mode error for pirq 55
> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
Adding some logging at the kernel side would seem much more
promising to me based on Stefano's analysis yesterday.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-26 2:31 ` Rolu
2012-06-26 7:42 ` Jan Beulich
@ 2012-06-26 12:59 ` Stefano Stabellini
2012-06-26 21:01 ` Rolu
1 sibling, 1 reply; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-26 12:59 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel, Jan Beulich, Stefano Stabellini
On Tue, 26 Jun 2012, Rolu wrote:
> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 25 Jun 2012, Jan Beulich wrote:
> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >> >> At the same time, adding logging to the guest kernel would
> >> >> be nice, to see what value it actually writes (in a current
> >> >> kernel this would be in __write_msi_msg()).
> >> >>
> >> >
> >> > Turns out that msg->data here is also 0x4300, so it seems the guest
> >> > kernel is producing these values. I caused it to make a stack trace
> >> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> >> > current data field and if it isn't equal to the macro it uses
> >> > xen_msi_compose_msg to make a new message, but that function just sets
> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> >> > then gets passed to __write_msi_msg and that's that. There are no
> >> > other writes through __write_msi_msg (except for the same thing for
> >> > other devices).
> >> >
> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
> >> > decoded as the delivery mode, so it seems the kernel is intentionally
> >> > setting it to 3.
> >>
> >> So that can never have worked properly afaict. Stefano, the
> >> code as it is currently - using literal (3 << 8) - is clearly bogus.
> >> Your original commit at least had a comment saying that the
> >> reserved delivery mode encoding is intentional here, but that
> >> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
> >> In any case - the cooperation with qemu apparently doesn't
> >> work, as the reserved encoding should never make it through
> >> to the hypervisor. Could you explain what the intention here
> >> was?
> >>
> >> And regardless of anything, can the literal numbers please be
> >> replaced by proper manifest constants - the "8" here already
> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> >> proper symbolic would permit locating where this is being (or
> >> really, as it doesn't appear to work supposed to be) consumed
> >> in qemu, provided it uses the same definition (i.e. that one
> >> should go into one of the public headers).
> >
> > The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
> > because notifications are not supposed to be delivered as MSI anymore.
> >
> > This is what should happen:
> >
> > 1) Linux configures the device with a 0 vector number and the pirq number
> > in the address field;
> >
> > 2) QEMU notices a vector number of 0 and reads the pirq number from the
> > address field, passing it to xc_domain_update_msi_irq;
> >
> > 3) Xen assignes the given pirq to the physical MSI;
> >
> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
> >
> > 5) Xen sets the pirq as "IRQ_PT";
> >
> > 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
> > returns true so Xen calls send_guest_pirq instead.
> >
> >
> > Obviously 6) is not happening. hvm_domain_use_pirq is:
> >
> > is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
> >
> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
> > above).
>
> This appears to be true. I added logging to hvm_pci_msi_assert in
> xen/drivers/passthrough/io.c and it indicates that
> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
> before an unsupported delivery mode message.
>
> I also log pirq->pirq but I found that most of the time I can't find
> this value anywhere else (I'm not sure how to interpret the value,
> though). For example, in my last try:
>
> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
> 53. The vast majority are for 54.
> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
> but complains it's already mapped.
> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
> 22, 52, 48, 47. Also never 54 or 53.
> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
> * The qemu log mentions pirq 35, 36 and 37
>
> It seems pirq values don't always mean the same? Is it a coincidence
> that 55 occurs almost everywhere, or is something going wrong with the
> other two values (53 and 54 versus 16 and 17)?
>
> I have three MSI capable devices passed through to the domU, and I do
> see groups of three distinct pirqs in the data above - just not the
> same ones in every place I look.
>
> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
> > (__startup_pirq doesn't get called), or Xen is erroring out in
> > map_domain_emuirq_pirq.
>
> evtchn_bind_pirq gets called, though I'm not sure if it is with the right data.
>
> map_domain_emuirq_pirq always gets past the checks in the top half
> (i.e. up to the line /* do not store emuirq mappings for pt devices
> */), except for one time with pirq=49,emuirq=23 where it finds they
> are already mapped.
> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
> This implies their info->arch.hvm.emuirq is also set to -2 (haven't
> directly logged that but it's the only assignment there).
>
> Interestingly, I get an unsupported delivery mode error for pirq 55
> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
Looking back at your QEMU logs, it seems that pt_msi_setup is not
called (or it is not called at the right time), otherwise you should
get:
pt_msi_setup requested pirq = %d
in your logs.
Could you try disabling msitranslate? You can do that adding
pci_msitranslate=0
to your VM config file.
If that works, probably this (untested) QEMU patch could fix your problem:
diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..09b1391 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -59,6 +59,26 @@ static void msix_set_enable(struct pt_dev *dev, int en)
/* MSI virtuailization functions */
+int pt_msi_pirq(struct pt_dev *d, int *pirq)
+{
+ int gvec = d->msi->data & 0xFF;
+ if (gvec != 0) {
+ return -1;
+ }
+
+ /* if gvec is 0, the guest is asking for a particular pirq that
+ * is passed as dest_id */
+ *pirq = (dev->msi->addr_hi & 0xffffff00) |
+ ((dev->msi->addr_lo >> MSI_TARGET_CPU_SHIFT) & 0xff);
+ if (!pirq) {
+ /* this probably identifies an misconfiguration of the guest,
+ * try the emulated path */
+ *pirq = -1;
+ return -1;
+ } else {
+ PT_LOG("%s requested pirq = %d\n", __func__, *pirq);
+ }
+}
/*
* setup physical msi, but didn't enable it
*/
@@ -74,18 +94,7 @@ int pt_msi_setup(struct pt_dev *dev)
}
gvec = dev->msi->data & 0xFF;
- if (!gvec) {
- /* if gvec is 0, the guest is asking for a particular pirq that
- * is passed as dest_id */
- pirq = (dev->msi->addr_hi & 0xffffff00) |
- ((dev->msi->addr_lo >> MSI_TARGET_CPU_SHIFT) & 0xff);
- if (!pirq)
- /* this probably identifies an misconfiguration of the guest,
- * try the emulated path */
- pirq = -1;
- else
- PT_LOG("pt_msi_setup requested pirq = %d\n", pirq);
- }
+ pt_msi_pirq(d, &pirq);
if ( xc_physdev_map_pirq_msi(xc_handle, domid, AUTO_ASSIGN, &pirq,
PCI_DEVFN(dev->pci_dev->dev,
@@ -138,6 +147,8 @@ int pt_msi_update(struct pt_dev *d)
addr = (uint64_t)d->msi->addr_hi << 32 | d->msi->addr_lo;
gflags = __get_msi_gflags(d->msi->data, addr);
+ pt_msi_pirq(d, &d->msi->pirq);
+
PT_LOG("Update msi with pirq %x gvec %x gflags %x\n",
d->msi->pirq, gvec, gflags);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-26 12:59 ` Stefano Stabellini
@ 2012-06-26 21:01 ` Rolu
2012-06-27 13:36 ` Stefano Stabellini
0 siblings, 1 reply; 20+ messages in thread
From: Rolu @ 2012-06-26 21:01 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Jan Beulich, xen-devel
On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 26 Jun 2012, Rolu wrote:
>> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
>> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >> >> At the same time, adding logging to the guest kernel would
>> >> >> be nice, to see what value it actually writes (in a current
>> >> >> kernel this would be in __write_msi_msg()).
>> >> >>
>> >> >
>> >> > Turns out that msg->data here is also 0x4300, so it seems the guest
>> >> > kernel is producing these values. I caused it to make a stack trace
>> >> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
>> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
>> >> > current data field and if it isn't equal to the macro it uses
>> >> > xen_msi_compose_msg to make a new message, but that function just sets
>> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
>> >> > then gets passed to __write_msi_msg and that's that. There are no
>> >> > other writes through __write_msi_msg (except for the same thing for
>> >> > other devices).
>> >> >
>> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
>> >> > decoded as the delivery mode, so it seems the kernel is intentionally
>> >> > setting it to 3.
>> >>
>> >> So that can never have worked properly afaict. Stefano, the
>> >> code as it is currently - using literal (3 << 8) - is clearly bogus.
>> >> Your original commit at least had a comment saying that the
>> >> reserved delivery mode encoding is intentional here, but that
>> >> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
>> >> In any case - the cooperation with qemu apparently doesn't
>> >> work, as the reserved encoding should never make it through
>> >> to the hypervisor. Could you explain what the intention here
>> >> was?
>> >>
>> >> And regardless of anything, can the literal numbers please be
>> >> replaced by proper manifest constants - the "8" here already
>> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> >> proper symbolic would permit locating where this is being (or
>> >> really, as it doesn't appear to work supposed to be) consumed
>> >> in qemu, provided it uses the same definition (i.e. that one
>> >> should go into one of the public headers).
>> >
>> > The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
>> > because notifications are not supposed to be delivered as MSI anymore.
>> >
>> > This is what should happen:
>> >
>> > 1) Linux configures the device with a 0 vector number and the pirq number
>> > in the address field;
>> >
>> > 2) QEMU notices a vector number of 0 and reads the pirq number from the
>> > address field, passing it to xc_domain_update_msi_irq;
>> >
>> > 3) Xen assignes the given pirq to the physical MSI;
>> >
>> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>> >
>> > 5) Xen sets the pirq as "IRQ_PT";
>> >
>> > 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
>> > returns true so Xen calls send_guest_pirq instead.
>> >
>> >
>> > Obviously 6) is not happening. hvm_domain_use_pirq is:
>> >
>> > is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
>> >
>> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
>> > above).
>>
>> This appears to be true. I added logging to hvm_pci_msi_assert in
>> xen/drivers/passthrough/io.c and it indicates that
>> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
>> before an unsupported delivery mode message.
>>
>> I also log pirq->pirq but I found that most of the time I can't find
>> this value anywhere else (I'm not sure how to interpret the value,
>> though). For example, in my last try:
>>
>> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
>> 53. The vast majority are for 54.
>> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
>> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
>> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
>> but complains it's already mapped.
>> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
>> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
>> 22, 52, 48, 47. Also never 54 or 53.
>> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
>> * The qemu log mentions pirq 35, 36 and 37
>>
>> It seems pirq values don't always mean the same? Is it a coincidence
>> that 55 occurs almost everywhere, or is something going wrong with the
>> other two values (53 and 54 versus 16 and 17)?
>>
>> I have three MSI capable devices passed through to the domU, and I do
>> see groups of three distinct pirqs in the data above - just not the
>> same ones in every place I look.
>>
>> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
>> > (__startup_pirq doesn't get called), or Xen is erroring out in
>> > map_domain_emuirq_pirq.
>>
>> evtchn_bind_pirq gets called, though I'm not sure if it is with the right data.
>>
>> map_domain_emuirq_pirq always gets past the checks in the top half
>> (i.e. up to the line /* do not store emuirq mappings for pt devices
>> */), except for one time with pirq=49,emuirq=23 where it finds they
>> are already mapped.
>> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
>> This implies their info->arch.hvm.emuirq is also set to -2 (haven't
>> directly logged that but it's the only assignment there).
>>
>> Interestingly, I get an unsupported delivery mode error for pirq 55
>> where my logging says pirq->arch.hvm.emuirq is -1, *after*
>> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
>
> Looking back at your QEMU logs, it seems that pt_msi_setup is not
> called (or it is not called at the right time), otherwise you should
> get:
>
> pt_msi_setup requested pirq = %d
>
> in your logs.
> Could you try disabling msitranslate? You can do that adding
>
> pci_msitranslate=0
>
> to your VM config file.
I tried that, but it didn't work.
> If that works, probably this (untested) QEMU patch could fix your problem:
>
I appreciate the help.
I applied the patch anyway just to see what would happen (had to edit
a few dev versus d variable names) but it didn't help. It also breaks
pt_msi_update, as I get in the qemu log:
pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
pt_msi_update: Error: Binding of MSI failed.
pt_msi_update: Error: Unmapping of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80
I added some logging to pt_msi_setup (without the patch). It does get
called, and it does so rather early in the boot process, each time
right before lines as these:
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
At this point dev->msi->data, addr_hi and addr_lo are all 0, which
doesn't seem right. Is it being called prematurely?
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-26 21:01 ` Rolu
@ 2012-06-27 13:36 ` Stefano Stabellini
2012-06-27 18:42 ` Rolu
0 siblings, 1 reply; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-27 13:36 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel, Jan Beulich, Stefano Stabellini
On Tue, 26 Jun 2012, Rolu wrote:
> On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 26 Jun 2012, Rolu wrote:
> >> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
> >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > On Mon, 25 Jun 2012, Jan Beulich wrote:
> >> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> >> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >> >> >> At the same time, adding logging to the guest kernel would
> >> >> >> be nice, to see what value it actually writes (in a current
> >> >> >> kernel this would be in __write_msi_msg()).
> >> >> >>
> >> >> >
> >> >> > Turns out that msg->data here is also 0x4300, so it seems the guest
> >> >> > kernel is producing these values. I caused it to make a stack trace
> >> >> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> >> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> >> >> > current data field and if it isn't equal to the macro it uses
> >> >> > xen_msi_compose_msg to make a new message, but that function just sets
> >> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> >> >> > then gets passed to __write_msi_msg and that's that. There are no
> >> >> > other writes through __write_msi_msg (except for the same thing for
> >> >> > other devices).
> >> >> >
> >> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
> >> >> > decoded as the delivery mode, so it seems the kernel is intentionally
> >> >> > setting it to 3.
> >> >>
> >> >> So that can never have worked properly afaict. Stefano, the
> >> >> code as it is currently - using literal (3 << 8) - is clearly bogus.
> >> >> Your original commit at least had a comment saying that the
> >> >> reserved delivery mode encoding is intentional here, but that
> >> >> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
> >> >> In any case - the cooperation with qemu apparently doesn't
> >> >> work, as the reserved encoding should never make it through
> >> >> to the hypervisor. Could you explain what the intention here
> >> >> was?
> >> >>
> >> >> And regardless of anything, can the literal numbers please be
> >> >> replaced by proper manifest constants - the "8" here already
> >> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> >> >> proper symbolic would permit locating where this is being (or
> >> >> really, as it doesn't appear to work supposed to be) consumed
> >> >> in qemu, provided it uses the same definition (i.e. that one
> >> >> should go into one of the public headers).
> >> >
> >> > The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
> >> > because notifications are not supposed to be delivered as MSI anymore.
> >> >
> >> > This is what should happen:
> >> >
> >> > 1) Linux configures the device with a 0 vector number and the pirq number
> >> > in the address field;
> >> >
> >> > 2) QEMU notices a vector number of 0 and reads the pirq number from the
> >> > address field, passing it to xc_domain_update_msi_irq;
> >> >
> >> > 3) Xen assignes the given pirq to the physical MSI;
> >> >
> >> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
> >> >
> >> > 5) Xen sets the pirq as "IRQ_PT";
> >> >
> >> > 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
> >> > returns true so Xen calls send_guest_pirq instead.
> >> >
> >> >
> >> > Obviously 6) is not happening. hvm_domain_use_pirq is:
> >> >
> >> > is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
> >> >
> >> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
> >> > above).
> >>
> >> This appears to be true. I added logging to hvm_pci_msi_assert in
> >> xen/drivers/passthrough/io.c and it indicates that
> >> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
> >> before an unsupported delivery mode message.
> >>
> >> I also log pirq->pirq but I found that most of the time I can't find
> >> this value anywhere else (I'm not sure how to interpret the value,
> >> though). For example, in my last try:
> >>
> >> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
> >> 53. The vast majority are for 54.
> >> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
> >> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> >> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
> >> but complains it's already mapped.
> >> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
> >> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
> >> 22, 52, 48, 47. Also never 54 or 53.
> >> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
> >> * The qemu log mentions pirq 35, 36 and 37
> >>
> >> It seems pirq values don't always mean the same? Is it a coincidence
> >> that 55 occurs almost everywhere, or is something going wrong with the
> >> other two values (53 and 54 versus 16 and 17)?
> >>
> >> I have three MSI capable devices passed through to the domU, and I do
> >> see groups of three distinct pirqs in the data above - just not the
> >> same ones in every place I look.
> >>
> >> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
> >> > (__startup_pirq doesn't get called), or Xen is erroring out in
> >> > map_domain_emuirq_pirq.
> >>
> >> evtchn_bind_pirq gets called, though I'm not sure if it is with the right data.
> >>
> >> map_domain_emuirq_pirq always gets past the checks in the top half
> >> (i.e. up to the line /* do not store emuirq mappings for pt devices
> >> */), except for one time with pirq=49,emuirq=23 where it finds they
> >> are already mapped.
> >> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
> >> This implies their info->arch.hvm.emuirq is also set to -2 (haven't
> >> directly logged that but it's the only assignment there).
> >>
> >> Interestingly, I get an unsupported delivery mode error for pirq 55
> >> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> >> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
> >
> > Looking back at your QEMU logs, it seems that pt_msi_setup is not
> > called (or it is not called at the right time), otherwise you should
> > get:
> >
> > pt_msi_setup requested pirq = %d
> >
> > in your logs.
> > Could you try disabling msitranslate? You can do that adding
> >
> > pci_msitranslate=0
> >
> > to your VM config file.
>
> I tried that, but it didn't work.
>
> > If that works, probably this (untested) QEMU patch could fix your problem:
> >
>
> I appreciate the help.
>
> I applied the patch anyway just to see what would happen (had to edit
> a few dev versus d variable names) but it didn't help. It also breaks
> pt_msi_update, as I get in the qemu log:
>
> pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
> pt_msi_update: Error: Binding of MSI failed.
> pt_msi_update: Error: Unmapping of MSI failed.
> pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80
>
> I added some logging to pt_msi_setup (without the patch). It does get
> called, and it does so rather early in the boot process, each time
> right before lines as these:
>
> pci_intx: intx=1
> register_real_device: Real physical device 00:1b.0 registered successfuly!
> IRQ type = MSI-INTx
>
> At this point dev->msi->data, addr_hi and addr_lo are all 0, which
> doesn't seem right. Is it being called prematurely?
That's because msitranslate is still enabled somehow, that is a
toolstack bug.
While we fix that bug, could you try this QEMU patch to forcefully disable
msitranslate?
diff --git a/xenstore.c b/xenstore.c
index ac90366..8af280e 100644
--- a/xenstore.c
+++ b/xenstore.c
@@ -427,7 +427,7 @@ uint32_t xenstore_read_target(void)
return target_domid;
}
-#define PT_PCI_MSITRANSLATE_DEFAULT 1
+#define PT_PCI_MSITRANSLATE_DEFAULT 0
#define PT_PCI_POWER_MANAGEMENT_DEFAULT 0
int direct_pci_msitranslate;
int direct_pci_power_mgmt;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-27 13:36 ` Stefano Stabellini
@ 2012-06-27 18:42 ` Rolu
2012-06-28 10:49 ` Stefano Stabellini
2012-06-29 11:21 ` [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery Stefano Stabellini
0 siblings, 2 replies; 20+ messages in thread
From: Rolu @ 2012-06-27 18:42 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Jan Beulich, xen-devel
On Wed, Jun 27, 2012 at 3:36 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 26 Jun 2012, Rolu wrote:
>> On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Tue, 26 Jun 2012, Rolu wrote:
>> >> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
>> >> <stefano.stabellini@eu.citrix.com> wrote:
>> >> > On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
>> >> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >> >> >> At the same time, adding logging to the guest kernel would
>> >> >> >> be nice, to see what value it actually writes (in a current
>> >> >> >> kernel this would be in __write_msi_msg()).
>> >> >> >>
>> >> >> >
>> >> >> > Turns out that msg->data here is also 0x4300, so it seems the guest
>> >> >> > kernel is producing these values. I caused it to make a stack trace
>> >> >> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
>> >> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
>> >> >> > current data field and if it isn't equal to the macro it uses
>> >> >> > xen_msi_compose_msg to make a new message, but that function just sets
>> >> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
>> >> >> > then gets passed to __write_msi_msg and that's that. There are no
>> >> >> > other writes through __write_msi_msg (except for the same thing for
>> >> >> > other devices).
>> >> >> >
>> >> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
>> >> >> > decoded as the delivery mode, so it seems the kernel is intentionally
>> >> >> > setting it to 3.
>> >> >>
>> >> >> So that can never have worked properly afaict. Stefano, the
>> >> >> code as it is currently - using literal (3 << 8) - is clearly bogus.
>> >> >> Your original commit at least had a comment saying that the
>> >> >> reserved delivery mode encoding is intentional here, but that
>> >> >> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
>> >> >> In any case - the cooperation with qemu apparently doesn't
>> >> >> work, as the reserved encoding should never make it through
>> >> >> to the hypervisor. Could you explain what the intention here
>> >> >> was?
>> >> >>
>> >> >> And regardless of anything, can the literal numbers please be
>> >> >> replaced by proper manifest constants - the "8" here already
>> >> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> >> >> proper symbolic would permit locating where this is being (or
>> >> >> really, as it doesn't appear to work supposed to be) consumed
>> >> >> in qemu, provided it uses the same definition (i.e. that one
>> >> >> should go into one of the public headers).
>> >> >
>> >> > The (3 << 8) is unimportant. The delivery mode chosen is "reserved"
>> >> > because notifications are not supposed to be delivered as MSI anymore.
>> >> >
>> >> > This is what should happen:
>> >> >
>> >> > 1) Linux configures the device with a 0 vector number and the pirq number
>> >> > in the address field;
>> >> >
>> >> > 2) QEMU notices a vector number of 0 and reads the pirq number from the
>> >> > address field, passing it to xc_domain_update_msi_irq;
>> >> >
>> >> > 3) Xen assignes the given pirq to the physical MSI;
>> >> >
>> >> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>> >> >
>> >> > 5) Xen sets the pirq as "IRQ_PT";
>> >> >
>> >> > 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
>> >> > returns true so Xen calls send_guest_pirq instead.
>> >> >
>> >> >
>> >> > Obviously 6) is not happening. hvm_domain_use_pirq is:
>> >> >
>> >> > is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq != IRQ_UNBOUND
>> >> >
>> >> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
>> >> > above).
>> >>
>> >> This appears to be true. I added logging to hvm_pci_msi_assert in
>> >> xen/drivers/passthrough/io.c and it indicates that
>> >> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
>> >> before an unsupported delivery mode message.
>> >>
>> >> I also log pirq->pirq but I found that most of the time I can't find
>> >> this value anywhere else (I'm not sure how to interpret the value,
>> >> though). For example, in my last try:
>> >>
>> >> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
>> >> 53. The vast majority are for 54.
>> >> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
>> >> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
>> >> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
>> >> but complains it's already mapped.
>> >> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
>> >> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
>> >> 22, 52, 48, 47. Also never 54 or 53.
>> >> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
>> >> * The qemu log mentions pirq 35, 36 and 37
>> >>
>> >> It seems pirq values don't always mean the same? Is it a coincidence
>> >> that 55 occurs almost everywhere, or is something going wrong with the
>> >> other two values (53 and 54 versus 16 and 17)?
>> >>
>> >> I have three MSI capable devices passed through to the domU, and I do
>> >> see groups of three distinct pirqs in the data above - just not the
>> >> same ones in every place I look.
>> >>
>> >> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
>> >> > (__startup_pirq doesn't get called), or Xen is erroring out in
>> >> > map_domain_emuirq_pirq.
>> >>
>> >> evtchn_bind_pirq gets called, though I'm not sure if it is with the right data.
>> >>
>> >> map_domain_emuirq_pirq always gets past the checks in the top half
>> >> (i.e. up to the line /* do not store emuirq mappings for pt devices
>> >> */), except for one time with pirq=49,emuirq=23 where it finds they
>> >> are already mapped.
>> >> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
>> >> This implies their info->arch.hvm.emuirq is also set to -2 (haven't
>> >> directly logged that but it's the only assignment there).
>> >>
>> >> Interestingly, I get an unsupported delivery mode error for pirq 55
>> >> where my logging says pirq->arch.hvm.emuirq is -1, *after*
>> >> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
>> >
>> > Looking back at your QEMU logs, it seems that pt_msi_setup is not
>> > called (or it is not called at the right time), otherwise you should
>> > get:
>> >
>> > pt_msi_setup requested pirq = %d
>> >
>> > in your logs.
>> > Could you try disabling msitranslate? You can do that adding
>> >
>> > pci_msitranslate=0
>> >
>> > to your VM config file.
>>
>> I tried that, but it didn't work.
>>
>> > If that works, probably this (untested) QEMU patch could fix your problem:
>> >
>>
>> I appreciate the help.
>>
>> I applied the patch anyway just to see what would happen (had to edit
>> a few dev versus d variable names) but it didn't help. It also breaks
>> pt_msi_update, as I get in the qemu log:
>>
>> pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
>> pt_msi_update: Error: Binding of MSI failed.
>> pt_msi_update: Error: Unmapping of MSI failed.
>> pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80
>>
>> I added some logging to pt_msi_setup (without the patch). It does get
>> called, and it does so rather early in the boot process, each time
>> right before lines as these:
>>
>> pci_intx: intx=1
>> register_real_device: Real physical device 00:1b.0 registered successfuly!
>> IRQ type = MSI-INTx
>>
>> At this point dev->msi->data, addr_hi and addr_lo are all 0, which
>> doesn't seem right. Is it being called prematurely?
>
> That's because msitranslate is still enabled somehow, that is a
> toolstack bug.
> While we fix that bug, could you try this QEMU patch to forcefully disable
> msitranslate?
>
This worked!
The "unsupported delivery mode" message is gone. Sound works, although
there is still occasionally a very short stutter, but I expect that's
a different issue. I've been testing with a KDE desktop with 3D
effects (cube, expo, that sort of stuff) and performance there has
gone up noticeably, from around 30-40 fps in most cases to near 60.
> diff --git a/xenstore.c b/xenstore.c
> index ac90366..8af280e 100644
> --- a/xenstore.c
> +++ b/xenstore.c
> @@ -427,7 +427,7 @@ uint32_t xenstore_read_target(void)
> return target_domid;
> }
>
> -#define PT_PCI_MSITRANSLATE_DEFAULT 1
> +#define PT_PCI_MSITRANSLATE_DEFAULT 0
> #define PT_PCI_POWER_MANAGEMENT_DEFAULT 0
> int direct_pci_msitranslate;
> int direct_pci_power_mgmt;
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-27 18:42 ` Rolu
@ 2012-06-28 10:49 ` Stefano Stabellini
2012-06-29 1:42 ` Rolu
2012-06-29 11:21 ` [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery Stefano Stabellini
1 sibling, 1 reply; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-28 10:49 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel, Jan Beulich, Stefano Stabellini
On Wed, 27 Jun 2012, Rolu wrote:
> > That's because msitranslate is still enabled somehow, that is a
> > toolstack bug.
> > While we fix that bug, could you try this QEMU patch to forcefully disable
> > msitranslate?
> >
>
> This worked!
>
> The "unsupported delivery mode" message is gone. Sound works, although
> there is still occasionally a very short stutter, but I expect that's
> a different issue. I've been testing with a KDE desktop with 3D
> effects (cube, expo, that sort of stuff) and performance there has
> gone up noticeably, from around 30-40 fps in most cases to near 60.
Great, good to hear!
Now that we have narrowed down the issue to msitranslate (that should be
disabled by default anyway), I would like to fix it entirely if possible.
Could you please try the appended patch, without any other patches
(therefore msitranslate would still be enabled)?
Thanks for testing!!
---
diff --git a/hw/pass-through.c b/hw/pass-through.c
index 8581253..fc8c49f 100644
--- a/hw/pass-through.c
+++ b/hw/pass-through.c
@@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev *ptdev,
PT_LOG("guest enabling MSI, disable MSI-INTx translation\n");
pt_disable_msi_translate(ptdev);
}
- else
+ /* Init physical one */
+ PT_LOG("setup msi for dev %x\n", pd->devfn);
+ if (pt_msi_setup(ptdev))
{
- /* Init physical one */
- PT_LOG("setup msi for dev %x\n", pd->devfn);
- if (pt_msi_setup(ptdev))
- {
- /* We do not broadcast the error to the framework code, so
- * that MSI errors are contained in MSI emulation code and
- * QEMU can go on running.
- * Guest MSI would be actually not working.
- */
- *value &= ~PCI_MSI_FLAGS_ENABLE;
- PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
- return 0;
- }
+ /* We do not broadcast the error to the framework code, so
+ * that MSI errors are contained in MSI emulation code and
+ * QEMU can go on running.
+ * Guest MSI would be actually not working.
+ */
+ *value &= ~PCI_MSI_FLAGS_ENABLE;
+ PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
+ return 0;
}
if (pt_msi_update(ptdev))
{
diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..73f737d 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
uint8_t e_device = 0;
uint8_t e_intx = 0;
- /* MSI_ENABLE bit should be disabed until the new handler is set */
- msi_set_enable(dev, 0);
-
- e_device = PCI_SLOT(dev->dev.devfn);
- e_intx = pci_intx(dev);
-
- if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
- PT_IRQ_TYPE_MSI_TRANSLATE, 0,
- e_device, e_intx, 0))
- PT_LOG("Error: Unbinding pt irq for MSI-INTx failed!\n");
+ pt_msi_disable(dev);
+ dev->msi->flags |= MSI_FLAG_UNINIT;
if (dev->machine_irq)
{
@@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
0, e_device, e_intx))
PT_LOG("Error: Rebinding of interrupt failed!\n");
}
-
- dev->msi_trans_en = 0;
}
static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-28 10:49 ` Stefano Stabellini
@ 2012-06-29 1:42 ` Rolu
2012-06-29 11:11 ` Stefano Stabellini
0 siblings, 1 reply; 20+ messages in thread
From: Rolu @ 2012-06-29 1:42 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Jan Beulich, xen-devel
On Thu, Jun 28, 2012 at 12:49 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 27 Jun 2012, Rolu wrote:
>> > That's because msitranslate is still enabled somehow, that is a
>> > toolstack bug.
>> > While we fix that bug, could you try this QEMU patch to forcefully disable
>> > msitranslate?
>> >
>>
>> This worked!
>>
>> The "unsupported delivery mode" message is gone. Sound works, although
>> there is still occasionally a very short stutter, but I expect that's
>> a different issue. I've been testing with a KDE desktop with 3D
>> effects (cube, expo, that sort of stuff) and performance there has
>> gone up noticeably, from around 30-40 fps in most cases to near 60.
>
> Great, good to hear!
> Now that we have narrowed down the issue to msitranslate (that should be
> disabled by default anyway), I would like to fix it entirely if possible.
> Could you please try the appended patch, without any other patches
> (therefore msitranslate would still be enabled)?
>
> Thanks for testing!!
>
I've reverted the previous patch and applied this one, and I'm pleased
to report it's still working. Thanks for your help!
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Issue with MSI in a HVM domU with several passed through PCI devices
2012-06-29 1:42 ` Rolu
@ 2012-06-29 11:11 ` Stefano Stabellini
0 siblings, 0 replies; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-29 11:11 UTC (permalink / raw)
To: Rolu; +Cc: xen-devel, Jan Beulich, Stefano Stabellini
On Fri, 29 Jun 2012, Rolu wrote:
> On Thu, Jun 28, 2012 at 12:49 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Wed, 27 Jun 2012, Rolu wrote:
> >> > That's because msitranslate is still enabled somehow, that is a
> >> > toolstack bug.
> >> > While we fix that bug, could you try this QEMU patch to forcefully disable
> >> > msitranslate?
> >> >
> >>
> >> This worked!
> >>
> >> The "unsupported delivery mode" message is gone. Sound works, although
> >> there is still occasionally a very short stutter, but I expect that's
> >> a different issue. I've been testing with a KDE desktop with 3D
> >> effects (cube, expo, that sort of stuff) and performance there has
> >> gone up noticeably, from around 30-40 fps in most cases to near 60.
> >
> > Great, good to hear!
> > Now that we have narrowed down the issue to msitranslate (that should be
> > disabled by default anyway), I would like to fix it entirely if possible.
> > Could you please try the appended patch, without any other patches
> > (therefore msitranslate would still be enabled)?
> >
> > Thanks for testing!!
> >
>
> I've reverted the previous patch and applied this one, and I'm pleased
> to report it's still working. Thanks for your help!
>
Thanks for testing! I am going to send the patch and add your tested-by.
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery
2012-06-27 18:42 ` Rolu
2012-06-28 10:49 ` Stefano Stabellini
@ 2012-06-29 11:21 ` Stefano Stabellini
2012-08-22 11:10 ` Stefano Stabellini
2012-08-30 15:12 ` Ian Jackson
1 sibling, 2 replies; 20+ messages in thread
From: Stefano Stabellini @ 2012-06-29 11:21 UTC (permalink / raw)
To: xen-devel; +Cc: rolu, Ian Jackson, Jan Beulich, Stefano Stabellini
When switching from msitranslate to straight msi we need to make sure
that we respect PV event delivery for the msi if the guest asked for it:
- completely disable MSI on the device in pt_disable_msi_translate;
- then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Tested-by: Rolu <rolu@roce.org>
---
diff --git a/hw/pass-through.c b/hw/pass-through.c
index 8581253..fc8c49f 100644
--- a/hw/pass-through.c
+++ b/hw/pass-through.c
@@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev *ptdev,
PT_LOG("guest enabling MSI, disable MSI-INTx translation\n");
pt_disable_msi_translate(ptdev);
}
- else
+ /* Init physical one */
+ PT_LOG("setup msi for dev %x\n", pd->devfn);
+ if (pt_msi_setup(ptdev))
{
- /* Init physical one */
- PT_LOG("setup msi for dev %x\n", pd->devfn);
- if (pt_msi_setup(ptdev))
- {
- /* We do not broadcast the error to the framework code, so
- * that MSI errors are contained in MSI emulation code and
- * QEMU can go on running.
- * Guest MSI would be actually not working.
- */
- *value &= ~PCI_MSI_FLAGS_ENABLE;
- PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
- return 0;
- }
+ /* We do not broadcast the error to the framework code, so
+ * that MSI errors are contained in MSI emulation code and
+ * QEMU can go on running.
+ * Guest MSI would be actually not working.
+ */
+ *value &= ~PCI_MSI_FLAGS_ENABLE;
+ PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
+ return 0;
}
if (pt_msi_update(ptdev))
{
diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..73f737d 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
uint8_t e_device = 0;
uint8_t e_intx = 0;
- /* MSI_ENABLE bit should be disabed until the new handler is set */
- msi_set_enable(dev, 0);
-
- e_device = PCI_SLOT(dev->dev.devfn);
- e_intx = pci_intx(dev);
-
- if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
- PT_IRQ_TYPE_MSI_TRANSLATE, 0,
- e_device, e_intx, 0))
- PT_LOG("Error: Unbinding pt irq for MSI-INTx failed!\n");
+ pt_msi_disable(dev);
+ dev->msi->flags |= MSI_FLAG_UNINIT;
if (dev->machine_irq)
{
@@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
0, e_device, e_intx))
PT_LOG("Error: Rebinding of interrupt failed!\n");
}
-
- dev->msi_trans_en = 0;
}
static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery
2012-06-29 11:21 ` [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery Stefano Stabellini
@ 2012-08-22 11:10 ` Stefano Stabellini
2012-08-30 15:12 ` Ian Jackson
1 sibling, 0 replies; 20+ messages in thread
From: Stefano Stabellini @ 2012-08-22 11:10 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: rolu, xen-devel, Ian Jackson, Jan Beulich
On Fri, 29 Jun 2012, Stefano Stabellini wrote:
> When switching from msitranslate to straight msi we need to make sure
> that we respect PV event delivery for the msi if the guest asked for it:
>
> - completely disable MSI on the device in pt_disable_msi_translate;
> - then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> Tested-by: Rolu <rolu@roce.org>
ping
> ---
>
> diff --git a/hw/pass-through.c b/hw/pass-through.c
> index 8581253..fc8c49f 100644
> --- a/hw/pass-through.c
> +++ b/hw/pass-through.c
> @@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev *ptdev,
> PT_LOG("guest enabling MSI, disable MSI-INTx translation\n");
> pt_disable_msi_translate(ptdev);
> }
> - else
> + /* Init physical one */
> + PT_LOG("setup msi for dev %x\n", pd->devfn);
> + if (pt_msi_setup(ptdev))
> {
> - /* Init physical one */
> - PT_LOG("setup msi for dev %x\n", pd->devfn);
> - if (pt_msi_setup(ptdev))
> - {
> - /* We do not broadcast the error to the framework code, so
> - * that MSI errors are contained in MSI emulation code and
> - * QEMU can go on running.
> - * Guest MSI would be actually not working.
> - */
> - *value &= ~PCI_MSI_FLAGS_ENABLE;
> - PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
> - return 0;
> - }
> + /* We do not broadcast the error to the framework code, so
> + * that MSI errors are contained in MSI emulation code and
> + * QEMU can go on running.
> + * Guest MSI would be actually not working.
> + */
> + *value &= ~PCI_MSI_FLAGS_ENABLE;
> + PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
> + return 0;
> }
> if (pt_msi_update(ptdev))
> {
> diff --git a/hw/pt-msi.c b/hw/pt-msi.c
> index 70c4023..73f737d 100644
> --- a/hw/pt-msi.c
> +++ b/hw/pt-msi.c
> @@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
> uint8_t e_device = 0;
> uint8_t e_intx = 0;
>
> - /* MSI_ENABLE bit should be disabed until the new handler is set */
> - msi_set_enable(dev, 0);
> -
> - e_device = PCI_SLOT(dev->dev.devfn);
> - e_intx = pci_intx(dev);
> -
> - if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
> - PT_IRQ_TYPE_MSI_TRANSLATE, 0,
> - e_device, e_intx, 0))
> - PT_LOG("Error: Unbinding pt irq for MSI-INTx failed!\n");
> + pt_msi_disable(dev);
> + dev->msi->flags |= MSI_FLAG_UNINIT;
>
> if (dev->machine_irq)
> {
> @@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
> 0, e_device, e_intx))
> PT_LOG("Error: Rebinding of interrupt failed!\n");
> }
> -
> - dev->msi_trans_en = 0;
> }
>
> static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery
2012-06-29 11:21 ` [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery Stefano Stabellini
2012-08-22 11:10 ` Stefano Stabellini
@ 2012-08-30 15:12 ` Ian Jackson
2012-09-03 9:44 ` Ian Campbell
1 sibling, 1 reply; 20+ messages in thread
From: Ian Jackson @ 2012-08-30 15:12 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: rolu, xen-devel, Jan Beulich
Stefano Stabellini writes ("[Xen-devel] [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery"):
> When switching from msitranslate to straight msi we need to make sure
> that we respect PV event delivery for the msi if the guest asked for it:
>
> - completely disable MSI on the device in pt_disable_msi_translate;
> - then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
I have applied this to 4.2.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery
2012-08-30 15:12 ` Ian Jackson
@ 2012-09-03 9:44 ` Ian Campbell
0 siblings, 0 replies; 20+ messages in thread
From: Ian Campbell @ 2012-09-03 9:44 UTC (permalink / raw)
To: Ian Jackson; +Cc: rolu, xen-devel, Jan Beulich, Stefano Stabellini
On Thu, 2012-08-30 at 16:12 +0100, Ian Jackson wrote:
> Stefano Stabellini writes ("[Xen-devel] [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery"):
> > When switching from msitranslate to straight msi we need to make sure
> > that we respect PV event delivery for the msi if the guest asked for it:
> >
> > - completely disable MSI on the device in pt_disable_msi_translate;
> > - then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
>
> I have applied this to 4.2.
You've not updated QEMU_TAG.
Ian.
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2012-09-03 9:44 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-18 20:57 Issue with MSI in a HVM domU with several passed through PCI devices Rolu
2012-06-19 9:45 ` Jan Beulich
2012-06-20 13:58 ` Rolu
2012-06-20 16:03 ` Jan Beulich
2012-06-24 2:21 ` Rolu
2012-06-25 7:49 ` Jan Beulich
2012-06-25 11:38 ` Stefano Stabellini
2012-06-26 2:31 ` Rolu
2012-06-26 7:42 ` Jan Beulich
2012-06-26 12:59 ` Stefano Stabellini
2012-06-26 21:01 ` Rolu
2012-06-27 13:36 ` Stefano Stabellini
2012-06-27 18:42 ` Rolu
2012-06-28 10:49 ` Stefano Stabellini
2012-06-29 1:42 ` Rolu
2012-06-29 11:11 ` Stefano Stabellini
2012-06-29 11:21 ` [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery Stefano Stabellini
2012-08-22 11:10 ` Stefano Stabellini
2012-08-30 15:12 ` Ian Jackson
2012-09-03 9:44 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.