Hi Alex,

VGA passthrough is working great here, with the exception of device reset.

In short, everything works the first time the guest runs. But the second
time I start the guest, before anything comes on the screen, the host
grinds to a halt and freezes (gradually, until after a few moments, magic
sysrq doesn't even work). No 'reduced performance' here, just a completely
frozen system.

Suspending and waking the host in between guest runs, while inconvenient,
completely avoids the problem. No more freezes and full graphics
performance in the guest. So my guess is that the PCI reset from software
just isn't happening for some reason.

The passthru devices are being assigned to pci-stub and vfio-pci as they
should be. Secondary passthrough (-vga cirrus or -vga std) doesn't change
much: things still work, and reset still doesn't. One small difference is
that the freeze on second boot is delayed until Windows initializes the
secondary graphics adapter, since the device isn't touched by the BIOS
prior to that.

Overriding the video BIOS doesn't seem to change anything. Passing through
just the VGA device (excluding the HDMI audio device) doesn't seem to make
much of a difference either.

- hardware
ASUS M5A99X EVO (AMD 990X/SB950; the IVRS is broken and overriden)
AMD Radeon HD 5750 (for the host)
AMD Radeon HD 7870 (for passthru)

- software: host
linux (Joerg Roedel's iommu tree) with linux-vfio merged in
qemu (latest git with vfio)
seabios (latest git)

- software: guest
windows 8
amd catalyst 13.5b2
virtio drivers

- lspci (abbreviated)
00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory
Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (PCI express gpp port B)
00:03.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (PCI express gpp port C)
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller
(rev 42)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
Juniper [Radeon HD 5700 Series]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Juniper HDMI
Audio [Radeon HD 5700 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
Pitcairn [Radeon HD 7800]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape
Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

- lspci -n (abbreviated)
01:00.0 0300: 1002:68be
01:00.1 0403: 1002:aa58
02:00.0 0300: 1002:6818
02:00.1 0403: 1002:aab0

- lspci -t (abbreviated)
-[0000:00]-+-00.0
           +-00.2
           +-02.0-[01]--+-00.0
           |            \-00.1
           +-03.0-[02]--+-00.0
                        \-00.1

- kernel cmdline
/vmlinuz-linux-iommu initrd=/initramfs-linux-iommu.img
root=/dev/CorsairVG/ArchLinux rootflags=subvol=root
systemd.unit=graphical.target debug nomodeset vga=804 acpi.debug_level=0x2
acpi.debug_layer=0xFFFFFFFF amd_iommu_dump
vfio_iommu_type1.allow_unsafe_interrupts=1 ivrs_ioapic[9]=00:14.0
ivrs_ioapic[10]=00:00.1

- qemu options
qemu-system-x86_64 -enable-kvm -name Windows8 \
 -M q35 -nodefconfig -readconfig /pool/KVM/Windows8/q35-chipset.cfg \
 -m 4096 -balloon none \
 -rtc base=localtime \
 -cpu host -smp 8,sockets=1,cores=4,threads=2 \
 -bios /usr/share/qemu/bios.bin \
 -vga none \
 -drive
if=virtio,format=raw,discard=on,cache=none,file=/dev/CorsairVG/Windows \
 -drive if=virtio,format=raw,file=/pool/KVM/Windows8/WinData.ntfs.img \
 -drive id=cdrom,media=cdrom,format=raw,file=/dev/null \
 -device ide-cd,bus=ide.0,drive=cdrom \
 -boot order=dc,menu=on \
 -net nic,model=virtio,macaddr=00:55:aa:00:00:01 -net bridge,br=vm_br \
 -soundhw hda \
 -usbdevice tablet \
 -device
vfio-pci,host=02:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on
\
 -device vfio-pci,host=02:00.1,bus=ich9-pcie-port-1,addr=0.1

- dmesg | egrep -i '(iommu|ioapic|pci-stub|vfio)' | grep -vi 'command line'
[    0.000000] ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI
0-23
[    0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI
24-55
[    0.223213] AMD-Vi:   DEV_SPECIAL(IOAPIC[0])         devid: 00:14.0
[    0.223219] AMD-Vi:   DEV_SPECIAL(IOAPIC[255])               devid:
00:00.1
[    0.600863] ACPI: Using IOAPIC for interrupt routing
[    2.033551] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.191528] VFIO - User Level meta-driver version: 0.3
[    3.601002] pci-stub: add 1002:6818 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    3.607895] pci-stub 0000:02:00.0: claimed by stub
[    3.614688] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    3.621640] pci-stub 0000:02:00.1: claimed by stub
[ 5137.969990] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[ 5137.995842] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[ 5137.995849] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
[ 5166.727114] vfio-pci 0000:02:00.0: irq 93 for MSI/MSI-X
(this is from before the second boot attempt)

- last two lines from dmesg before the freeze (netcat'd to another box)
Clocksource tsc unstable (delta = -416526709 ns)
Switching to clocksource hpet

- output with DEBUG_VFIO: lots and lots, see attachment

If you need any more information, I'll be glad to provide it.

Justin