VFIO VGA test branches

VFIO VGA test branches

[Alex Williamson]

Hi folks,

A number of people have been trying VFIO's VGA support, a few have even
been successful.  Resetting devices has been a problem and makes it
very, very difficult to really use VGA assignment effectively.  The code
in the branches below attempts to address this.  Discrete graphics
devices are typically on their own bus, which we can reset so we
theoretically get something pretty close to a power-on state for the GPU
on each run (or after each guest reset).  With this I'm able to get
multiple runs on my HD7850 with no need to reset the host.  Hopefully
this will also cleanup after any host uses of the device so we can
unload driver rather than blacklisting them.

If you've been playing with VFIO and VGA, please give the branches below
a shot and report successes and failures.  Note that this new reset is
only enable with the x-vga=on option, so should not do gratuitous bus
resets for other devices.  Thanks,

Alex

git://github.com/awilliam/linux-vfio.git vfio-vga-reset
git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

PS - The above linux branch is v3.9 based which has a known kvm emulator
bug.  If you're on Intel and nothing happens, try:

sudo modprobe -r kvm_intel
sudo modprobe kvm_intel emulate_invalid_guest_state=0

This is required to execute the VGA BIOS on my HD7850.

If things still don't work, apply the following patch:

--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -40,7 +40,7 @@
 #include "sysemu/kvm.h"
 #include "sysemu/sysemu.h"
 
-/* #define DEBUG_VFIO */
+#define DEBUG_VFIO
 #ifdef DEBUG_VFIO
 #define DPRINTF(fmt, ...) \
     do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)

And log the output (there will be lots).

Also, AMD/ATI and Nvidia are the only devices expected to have a
reasonable shot at working.  I'm seeing reports of success on AMD/ATI HD
5xxx, 6xxx, and 7xxx, as well as Nvidia Geforce 7-series, 8-series, 4xx
series, and 6xx series.  Older cards from those vendors probably aren't
very interesting to support (honestly I wouldn't care much about 7/8
series Nvidia or HD5xxx AMD, except I happen to have some for testing -
use emulated VGA if you don't care about performance).

Intel IGD graphics has numerous issues since it's partially incorporated
into the chipset.  Please don't bother to report IGD is broken unless
you're interested in fixing it.

[Qemu-devel] VFIO VGA test branches

[Alex Williamson]

Hi folks,

A number of people have been trying VFIO's VGA support, a few have even
been successful.  Resetting devices has been a problem and makes it
very, very difficult to really use VGA assignment effectively.  The code
in the branches below attempts to address this.  Discrete graphics
devices are typically on their own bus, which we can reset so we
theoretically get something pretty close to a power-on state for the GPU
on each run (or after each guest reset).  With this I'm able to get
multiple runs on my HD7850 with no need to reset the host.  Hopefully
this will also cleanup after any host uses of the device so we can
unload driver rather than blacklisting them.

If you've been playing with VFIO and VGA, please give the branches below
a shot and report successes and failures.  Note that this new reset is
only enable with the x-vga=on option, so should not do gratuitous bus
resets for other devices.  Thanks,

Alex

git://github.com/awilliam/linux-vfio.git vfio-vga-reset
git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

PS - The above linux branch is v3.9 based which has a known kvm emulator
bug.  If you're on Intel and nothing happens, try:

sudo modprobe -r kvm_intel
sudo modprobe kvm_intel emulate_invalid_guest_state=0

This is required to execute the VGA BIOS on my HD7850.

If things still don't work, apply the following patch:

--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -40,7 +40,7 @@
 #include "sysemu/kvm.h"
 #include "sysemu/sysemu.h"
 
-/* #define DEBUG_VFIO */
+#define DEBUG_VFIO
 #ifdef DEBUG_VFIO
 #define DPRINTF(fmt, ...) \
     do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)

And log the output (there will be lots).

Also, AMD/ATI and Nvidia are the only devices expected to have a
reasonable shot at working.  I'm seeing reports of success on AMD/ATI HD
5xxx, 6xxx, and 7xxx, as well as Nvidia Geforce 7-series, 8-series, 4xx
series, and 6xx series.  Older cards from those vendors probably aren't
very interesting to support (honestly I wouldn't care much about 7/8
series Nvidia or HD5xxx AMD, except I happen to have some for testing -
use emulated VGA if you don't care about performance).

Intel IGD graphics has numerous issues since it's partially incorporated
into the chipset.  Please don't bother to report IGD is broken unless
you're interested in fixing it.

Re: VFIO VGA test branches

[Alex Williamson]

A few notes for anyone trying this...

      * I recommend the q35 machine type and using the default config
        file found in the docs directory.  This means your command line
        should include:

         -M q35 -nodefconfig -readconfig /path/to/qemu.git/docs/q35-chipset.cfg

      * You're likely passing through a graphics card that is attached
        to the host system below a root port, so make it appear that way
        to the guest too.  If your graphics card has a graphics function
        and audio function, assign them as:

        -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
        -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1
        
        The bus name comes from the q35-chipset.cfg above.  If your
        graphics doesn't include a separate audio device, drop the
        second line and the multifunction option of the first (addr is
        also optional at that point, 0.0 will be the default).
        
      * If you follow both of the above, your VGA device is now below a
        root port, but the version of seabios in qemu doesn't support
        initializing VGA routing to that device.  To fix, use upstream
        seabios: git://git.seabios.org/seabios.git  The default config
        should work.  Then add the following to your qemu commandline:
        
        -L /path/to/seabios.git/out/ -L /path/to/qemu/bios/files/
        
        (the latter is likely /usr/local/share/qemu/)
        
      * You can use -nographic to prevent QEMU from trying to start SDL
        or need a vnc parameter.  You can also specify a -vnc option and
        use the window for mouse input.

      * Use -vga none.  At this point I'm not really interested in
        dual-headed VMs unless you're interested in working on it.
        Having an emulated VGA means we're not really testing VGA
        support through VFIO.

      * Do no use the vfio-pci romfile option unless you need it (ie.
        try w/o first).  Option ROMs check an internal signature against
        the hardware.  If they don't match, it isn't run.  If you
        download a ROM from the internet, you may get nowhere.  If you
        do need a ROM, it's best to scrape it off the device you're
        using.  You can do this through the "rom" file in sysfs for the
        device.  "echo 1 > rom" to enable it, the read it as "cat rom
        > /tmp/rom".  To do this, it should be a secondary graphics
        device and be untouched by host drivers.  You may have better
        luck booting from an install CD to get an environment where the
        device is untouched for this.

      * USB passthrough is handy for input and easier than figuring out
        which ports are connected to which USB controllers for vfio-pci
        assignment.  Use lsusb to find the devices, note the bus and
        device numbers, the use:

        -device usb-host,hostbus=8,hostaddr=2

I think that's it.  Feel free to reply with other best practices.
Thanks,

Alex

On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> Hi folks,
> 
> A number of people have been trying VFIO's VGA support, a few have even
> been successful.  Resetting devices has been a problem and makes it
> very, very difficult to really use VGA assignment effectively.  The code
> in the branches below attempts to address this.  Discrete graphics
> devices are typically on their own bus, which we can reset so we
> theoretically get something pretty close to a power-on state for the GPU
> on each run (or after each guest reset).  With this I'm able to get
> multiple runs on my HD7850 with no need to reset the host.  Hopefully
> this will also cleanup after any host uses of the device so we can
> unload driver rather than blacklisting them.
> 
> If you've been playing with VFIO and VGA, please give the branches below
> a shot and report successes and failures.  Note that this new reset is
> only enable with the x-vga=on option, so should not do gratuitous bus
> resets for other devices.  Thanks,
> 
> Alex
> 
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> PS - The above linux branch is v3.9 based which has a known kvm emulator
> bug.  If you're on Intel and nothing happens, try:
> 
> sudo modprobe -r kvm_intel
> sudo modprobe kvm_intel emulate_invalid_guest_state=0
> 
> This is required to execute the VGA BIOS on my HD7850.
> 
> If things still don't work, apply the following patch:
> 
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -40,7 +40,7 @@
>  #include "sysemu/kvm.h"
>  #include "sysemu/sysemu.h"
>  
> -/* #define DEBUG_VFIO */
> +#define DEBUG_VFIO
>  #ifdef DEBUG_VFIO
>  #define DPRINTF(fmt, ...) \
>      do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)
> 
> And log the output (there will be lots).
> 
> Also, AMD/ATI and Nvidia are the only devices expected to have a
> reasonable shot at working.  I'm seeing reports of success on AMD/ATI HD
> 5xxx, 6xxx, and 7xxx, as well as Nvidia Geforce 7-series, 8-series, 4xx
> series, and 6xx series.  Older cards from those vendors probably aren't
> very interesting to support (honestly I wouldn't care much about 7/8
> series Nvidia or HD5xxx AMD, except I happen to have some for testing -
> use emulated VGA if you don't care about performance).
> 
> Intel IGD graphics has numerous issues since it's partially incorporated
> into the chipset.  Please don't bother to report IGD is broken unless
> you're interested in fixing it.

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

A few notes for anyone trying this...

      * I recommend the q35 machine type and using the default config
        file found in the docs directory.  This means your command line
        should include:

         -M q35 -nodefconfig -readconfig /path/to/qemu.git/docs/q35-chipset.cfg

      * You're likely passing through a graphics card that is attached
        to the host system below a root port, so make it appear that way
        to the guest too.  If your graphics card has a graphics function
        and audio function, assign them as:

        -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
        -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1
        
        The bus name comes from the q35-chipset.cfg above.  If your
        graphics doesn't include a separate audio device, drop the
        second line and the multifunction option of the first (addr is
        also optional at that point, 0.0 will be the default).
        
      * If you follow both of the above, your VGA device is now below a
        root port, but the version of seabios in qemu doesn't support
        initializing VGA routing to that device.  To fix, use upstream
        seabios: git://git.seabios.org/seabios.git  The default config
        should work.  Then add the following to your qemu commandline:
        
        -L /path/to/seabios.git/out/ -L /path/to/qemu/bios/files/
        
        (the latter is likely /usr/local/share/qemu/)
        
      * You can use -nographic to prevent QEMU from trying to start SDL
        or need a vnc parameter.  You can also specify a -vnc option and
        use the window for mouse input.

      * Use -vga none.  At this point I'm not really interested in
        dual-headed VMs unless you're interested in working on it.
        Having an emulated VGA means we're not really testing VGA
        support through VFIO.

      * Do no use the vfio-pci romfile option unless you need it (ie.
        try w/o first).  Option ROMs check an internal signature against
        the hardware.  If they don't match, it isn't run.  If you
        download a ROM from the internet, you may get nowhere.  If you
        do need a ROM, it's best to scrape it off the device you're
        using.  You can do this through the "rom" file in sysfs for the
        device.  "echo 1 > rom" to enable it, the read it as "cat rom
        > /tmp/rom".  To do this, it should be a secondary graphics
        device and be untouched by host drivers.  You may have better
        luck booting from an install CD to get an environment where the
        device is untouched for this.

      * USB passthrough is handy for input and easier than figuring out
        which ports are connected to which USB controllers for vfio-pci
        assignment.  Use lsusb to find the devices, note the bus and
        device numbers, the use:

        -device usb-host,hostbus=8,hostaddr=2

I think that's it.  Feel free to reply with other best practices.
Thanks,

Alex

On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> Hi folks,
> 
> A number of people have been trying VFIO's VGA support, a few have even
> been successful.  Resetting devices has been a problem and makes it
> very, very difficult to really use VGA assignment effectively.  The code
> in the branches below attempts to address this.  Discrete graphics
> devices are typically on their own bus, which we can reset so we
> theoretically get something pretty close to a power-on state for the GPU
> on each run (or after each guest reset).  With this I'm able to get
> multiple runs on my HD7850 with no need to reset the host.  Hopefully
> this will also cleanup after any host uses of the device so we can
> unload driver rather than blacklisting them.
> 
> If you've been playing with VFIO and VGA, please give the branches below
> a shot and report successes and failures.  Note that this new reset is
> only enable with the x-vga=on option, so should not do gratuitous bus
> resets for other devices.  Thanks,
> 
> Alex
> 
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> PS - The above linux branch is v3.9 based which has a known kvm emulator
> bug.  If you're on Intel and nothing happens, try:
> 
> sudo modprobe -r kvm_intel
> sudo modprobe kvm_intel emulate_invalid_guest_state=0
> 
> This is required to execute the VGA BIOS on my HD7850.
> 
> If things still don't work, apply the following patch:
> 
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -40,7 +40,7 @@
>  #include "sysemu/kvm.h"
>  #include "sysemu/sysemu.h"
>  
> -/* #define DEBUG_VFIO */
> +#define DEBUG_VFIO
>  #ifdef DEBUG_VFIO
>  #define DPRINTF(fmt, ...) \
>      do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)
> 
> And log the output (there will be lots).
> 
> Also, AMD/ATI and Nvidia are the only devices expected to have a
> reasonable shot at working.  I'm seeing reports of success on AMD/ATI HD
> 5xxx, 6xxx, and 7xxx, as well as Nvidia Geforce 7-series, 8-series, 4xx
> series, and 6xx series.  Older cards from those vendors probably aren't
> very interesting to support (honestly I wouldn't care much about 7/8
> series Nvidia or HD5xxx AMD, except I happen to have some for testing -
> use emulated VGA if you don't care about performance).
> 
> Intel IGD graphics has numerous issues since it's partially incorporated
> into the chipset.  Please don't bother to report IGD is broken unless
> you're interested in fixing it.

Re: VFIO VGA test branches

[Knut Omang]

Hi all,

Perfect timing from my perspective, thanks Alex!

I spent the better part of the weekend testing your branches on a new system 
I just put together for this purpose, results below..

On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
...
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

System setup: 

- Fedora 18 on
- Gigabyte Z77X-UD5H motherboard
- Intel Core i7 3770 (Ivy bridge w/integrated graphics)
- 2 discrete graphics cards:

lspci | egrep 'VGA|Audio'
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

Short summary:

- Once I got past a few time consuming obstacles explained below
   - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
     on both discrete graphics cards 
     (though so far only one at at time and with some minor issues, see below)
   - no success with the hdmi audio yet (ideas for further investigation appreciated!)

- Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
  with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 

Details:

- I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
   (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
   - this caused the system to freeze (with color artifacts on the console)

- In my naivity and because of the "i" in ifgx I tried both with 
  'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
  and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
  that iommu support was turned off for **all** graphics cards with igfx_off

- The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':

- iommu groups:

ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
total 0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

- eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:

# lspci -n
...
01:00.0 0300: 1002:683f
01:00.1 0403: 1002:aab0
02:00.0 0300: 1002:6779
02:00.1 0403: 1002:aa98
...

modprobe vfio_pci
echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id

# lsusb 
...
Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
...

- I also applied your suggested patch to the quirk function in VFIO (see below)

- Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
  notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
  until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)


qemu-kvm -M q35 \
  -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
  -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
  -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
  -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
  -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
  -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
  -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
  -device ide-drive,drive=ivm03,bus=ide.0 \
  -device ide-cd,drive=cd,bus=ide.1 \
  -net nic,vlan=0,model=virtio -net tap,vlan=0 \
  -enable-kvm \
  -device usb-host,hostbus=1,hostaddr=8 \
  -device usb-host,hostbus=1,hostaddr=4

- Both the graphics card seems to have a rom but only the HD6450 let itself to "scraping". 
Anyway, supplying it to vfio did not seem to make any difference.

find /sys -name rom
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
/sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
...

Some observations and remaining unresolved issues:

- VFIO patch:
  Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
  reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
  vfio_ati_3c3_quirk_read and repeated logs: 
    vfio: vfio_vga_read(0x3c3, 1) = 0x0
  I patched up accordingly with 


diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index da0e5f9..a361d06 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
     uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
                                   addr + quirk->data.base_offset, size);
 
-    if (data == quirk->data.address_match) {
+    if (1 || data == quirk->data.address_match) {
         data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
         DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
     }


  This of course did not help much until I actually got the iommu 
  enabled for the radeons (similar "repeated patters" as deniv reported)
  but what I have observed after I got it working is that if 
  I disable the patch above, things are not that well: the Fedora VM 
  comes up with VGA and the Fedora boot screen, then goes blank when 
  switching to X.

- The fact that the iommu group now extends across all my available graphics 
  devices now makes it difficult to  get the radeon (or catalyst) driver use to 
  the other card since the vfio_pci driver needs to hold it.
  Not a complete showstopper since the vesa driver comes up with 1024x768..
  Might it be a good idea to have an override option (exception list or similar?) 
  to allow the vfio_pci to be less restrictive about owning the whole group 
   - allow functionality over security in such case? This of course is further complicated
  by the need for graphics drivers to be disabled/enabled already at the kernel prompt..

- There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
  The BIOS offers (undocumented) a full range of selections of which PCIe 
  (or PCIe 1x) graphics card to use as primary, but any other selection 
  than the first PCIe 16x slot has no effect and the motherboard reverts 
  to the first slot, so to be able to test both cards, I had to put the card under test
  into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
  fixes for this in newer BIOSes.

- The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
  on the new system, using the discrete graphics with HDMI/Displayport audio. 
  With the HD7700 moved to the second PCIe slot I tested both Windows and 
  Linux guests to try to get some sound through the HDMI audio device. 
  Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
  the system settings -> multimedia dialogue never opens up which seems to indicate that 
  PulseAudio has problems communicating with the passed through device (?), 
  any hints/pointers here appreciated. From the vfio log it seems at least
  config space is accessed ok.

- There also seems to be issues with radeon and intel_iommu=on - if I try 
  to enable modesetting and normal X support for the radeon cards, X fails to start.

- It would be nice if the integrated graphics could be used as the host primary display - 
  I would be happy if someone has any hints as to if/how the ifgx_off option 
  could be extended/modified to only affect iommu operation on selected device(s),
  if at all possible..

Thanks,

Knut Omang

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

Hi all,

Perfect timing from my perspective, thanks Alex!

I spent the better part of the weekend testing your branches on a new system 
I just put together for this purpose, results below..

On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
...
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

System setup: 

- Fedora 18 on
- Gigabyte Z77X-UD5H motherboard
- Intel Core i7 3770 (Ivy bridge w/integrated graphics)
- 2 discrete graphics cards:

lspci | egrep 'VGA|Audio'
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

Short summary:

- Once I got past a few time consuming obstacles explained below
   - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
     on both discrete graphics cards 
     (though so far only one at at time and with some minor issues, see below)
   - no success with the hdmi audio yet (ideas for further investigation appreciated!)

- Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
  with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 

Details:

- I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
   (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
   - this caused the system to freeze (with color artifacts on the console)

- In my naivity and because of the "i" in ifgx I tried both with 
  'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
  and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
  that iommu support was turned off for **all** graphics cards with igfx_off

- The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':

- iommu groups:

ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
total 0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

- eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:

# lspci -n
...
01:00.0 0300: 1002:683f
01:00.1 0403: 1002:aab0
02:00.0 0300: 1002:6779
02:00.1 0403: 1002:aa98
...

modprobe vfio_pci
echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id

# lsusb 
...
Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
...

- I also applied your suggested patch to the quirk function in VFIO (see below)

- Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
  notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
  until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)


qemu-kvm -M q35 \
  -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
  -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
  -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
  -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
  -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
  -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
  -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
  -device ide-drive,drive=ivm03,bus=ide.0 \
  -device ide-cd,drive=cd,bus=ide.1 \
  -net nic,vlan=0,model=virtio -net tap,vlan=0 \
  -enable-kvm \
  -device usb-host,hostbus=1,hostaddr=8 \
  -device usb-host,hostbus=1,hostaddr=4

- Both the graphics card seems to have a rom but only the HD6450 let itself to "scraping". 
Anyway, supplying it to vfio did not seem to make any difference.

find /sys -name rom
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
/sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
...

Some observations and remaining unresolved issues:

- VFIO patch:
  Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
  reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
  vfio_ati_3c3_quirk_read and repeated logs: 
    vfio: vfio_vga_read(0x3c3, 1) = 0x0
  I patched up accordingly with 


diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index da0e5f9..a361d06 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
     uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
                                   addr + quirk->data.base_offset, size);
 
-    if (data == quirk->data.address_match) {
+    if (1 || data == quirk->data.address_match) {
         data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
         DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
     }


  This of course did not help much until I actually got the iommu 
  enabled for the radeons (similar "repeated patters" as deniv reported)
  but what I have observed after I got it working is that if 
  I disable the patch above, things are not that well: the Fedora VM 
  comes up with VGA and the Fedora boot screen, then goes blank when 
  switching to X.

- The fact that the iommu group now extends across all my available graphics 
  devices now makes it difficult to  get the radeon (or catalyst) driver use to 
  the other card since the vfio_pci driver needs to hold it.
  Not a complete showstopper since the vesa driver comes up with 1024x768..
  Might it be a good idea to have an override option (exception list or similar?) 
  to allow the vfio_pci to be less restrictive about owning the whole group 
   - allow functionality over security in such case? This of course is further complicated
  by the need for graphics drivers to be disabled/enabled already at the kernel prompt..

- There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
  The BIOS offers (undocumented) a full range of selections of which PCIe 
  (or PCIe 1x) graphics card to use as primary, but any other selection 
  than the first PCIe 16x slot has no effect and the motherboard reverts 
  to the first slot, so to be able to test both cards, I had to put the card under test
  into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
  fixes for this in newer BIOSes.

- The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
  on the new system, using the discrete graphics with HDMI/Displayport audio. 
  With the HD7700 moved to the second PCIe slot I tested both Windows and 
  Linux guests to try to get some sound through the HDMI audio device. 
  Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
  the system settings -> multimedia dialogue never opens up which seems to indicate that 
  PulseAudio has problems communicating with the passed through device (?), 
  any hints/pointers here appreciated. From the vfio log it seems at least
  config space is accessed ok.

- There also seems to be issues with radeon and intel_iommu=on - if I try 
  to enable modesetting and normal X support for the radeon cards, X fails to start.

- It would be nice if the integrated graphics could be used as the host primary display - 
  I would be happy if someone has any hints as to if/how the ifgx_off option 
  could be extended/modified to only affect iommu operation on selected device(s),
  if at all possible..

Thanks,

Knut Omang

Re: VFIO VGA test branches

[Alex Williamson]

On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> Hi all,
> 
> Perfect timing from my perspective, thanks Alex!
> 
> I spent the better part of the weekend testing your branches on a new system 
> I just put together for this purpose, results below..
> 
> On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> ...
> > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> System setup: 
> 
> - Fedora 18 on
> - Gigabyte Z77X-UD5H motherboard
> - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> - 2 discrete graphics cards:
> 
> lspci | egrep 'VGA|Audio'
> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> Short summary:
> 
> - Once I got past a few time consuming obstacles explained below
>    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
>      on both discrete graphics cards 
>      (though so far only one at at time and with some minor issues, see below)
>    - no success with the hdmi audio yet (ideas for further investigation appreciated!)

I've had hdmi audio working with an HD7850, but only in Windows (7) and
it was using legacy interrupts for some reason instead of MSI.  I wonder
if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
what's wrong with MSI, but it seems to be new with the PCI bus reset
support.

> - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
>   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> 
> Details:
> 
> - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
>    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
>    - this caused the system to freeze (with color artifacts on the console)
> 
> - In my naivity and because of the "i" in ifgx I tried both with 
>   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
>   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
>   that iommu support was turned off for **all** graphics cards with igfx_off

I'm not sure why this is, looks like the code only tries to turn it off
when only graphics is under the remapping device.  We'd probably need to
see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).

> - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> 
> - iommu groups:
> 
> ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> total 0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> 
> - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:

Ick.  Intel has been pretty good about advertising ACS support on their
root ports.  I wonder if this is an oversight or if they are actually
not isolated from each other.

> # lspci -n
> ...
> 01:00.0 0300: 1002:683f
> 01:00.1 0403: 1002:aab0
> 02:00.0 0300: 1002:6779
> 02:00.1 0403: 1002:aa98
> ...
> 
> modprobe vfio_pci
> echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> # lsusb 
> ...
> Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> ...
> 
> - I also applied your suggested patch to the quirk function in VFIO (see below)
> 
> - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
>   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
>   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> 
> 
> qemu-kvm -M q35 \
>   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
>   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
>   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
>   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
>   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
>   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
>   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
>   -device ide-drive,drive=ivm03,bus=ide.0 \
>   -device ide-cd,drive=cd,bus=ide.1 \
>   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
>   -enable-kvm \
>   -device usb-host,hostbus=1,hostaddr=8 \
>   -device usb-host,hostbus=1,hostaddr=4
> 
> - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 

Did you try scraping the HD6450 while the HD7700 was the boot VGA and
vica versa?  The boot VGA ROM is handled in a special way and what you
really get is the shadow copy, which isn't what we want.

> Anyway, supplying it to vfio did not seem to make any difference.
> 
> find /sys -name rom
> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> ...
> 
> Some observations and remaining unresolved issues:
> 
> - VFIO patch:
>   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
>   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
>   vfio_ati_3c3_quirk_read and repeated logs: 
>     vfio: vfio_vga_read(0x3c3, 1) = 0x0
>   I patched up accordingly with 
> 
> 
> diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> index da0e5f9..a361d06 100644
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
>      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
>                                    addr + quirk->data.base_offset, size);
>  
> -    if (data == quirk->data.address_match) {
> +    if (1 || data == quirk->data.address_match) {
>          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
>          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
>      }
> 
> 
>   This of course did not help much until I actually got the iommu 
>   enabled for the radeons (similar "repeated patters" as deniv reported)
>   but what I have observed after I got it working is that if 
>   I disable the patch above, things are not that well: the Fedora VM 
>   comes up with VGA and the Fedora boot screen, then goes blank when 
>   switching to X.

Hmm, I think we'd probably have better luck making that unconditional
until we have reason to do otherwise.

> - The fact that the iommu group now extends across all my available graphics 
>   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
>   the other card since the vfio_pci driver needs to hold it.
>   Not a complete showstopper since the vesa driver comes up with 1024x768..
>   Might it be a good idea to have an override option (exception list or similar?) 
>   to allow the vfio_pci to be less restrictive about owning the whole group 
>    - allow functionality over security in such case? This of course is further complicated
>   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..

We have a quirk in the kernel that enables us to witelist devices, but
yes, there is no flexibility in this w/o modifying the code and
rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
the example above w/ pci_dev_dma_source - function can just return 1)

> - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
>   The BIOS offers (undocumented) a full range of selections of which PCIe 
>   (or PCIe 1x) graphics card to use as primary, but any other selection 
>   than the first PCIe 16x slot has no effect and the motherboard reverts 
>   to the first slot, so to be able to test both cards, I had to put the card under test
>   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
>   fixes for this in newer BIOSes.
> 
> - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
>   on the new system, using the discrete graphics with HDMI/Displayport audio. 
>   With the HD7700 moved to the second PCIe slot I tested both Windows and 
>   Linux guests to try to get some sound through the HDMI audio device. 
>   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
>   the system settings -> multimedia dialogue never opens up which seems to indicate that 
>   PulseAudio has problems communicating with the passed through device (?), 
>   any hints/pointers here appreciated. From the vfio log it seems at least
>   config space is accessed ok.
> 
> - There also seems to be issues with radeon and intel_iommu=on - if I try 
>   to enable modesetting and normal X support for the radeon cards, X fails to start.
> 
> - It would be nice if the integrated graphics could be used as the host primary display - 
>   I would be happy if someone has any hints as to if/how the ifgx_off option 
>   could be extended/modified to only affect iommu operation on selected device(s),
>   if at all possible..

Let's see what we can discover from your DMAR.  Also send along sudo
lspci -vvv.  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> Hi all,
> 
> Perfect timing from my perspective, thanks Alex!
> 
> I spent the better part of the weekend testing your branches on a new system 
> I just put together for this purpose, results below..
> 
> On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> ...
> > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> System setup: 
> 
> - Fedora 18 on
> - Gigabyte Z77X-UD5H motherboard
> - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> - 2 discrete graphics cards:
> 
> lspci | egrep 'VGA|Audio'
> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> Short summary:
> 
> - Once I got past a few time consuming obstacles explained below
>    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
>      on both discrete graphics cards 
>      (though so far only one at at time and with some minor issues, see below)
>    - no success with the hdmi audio yet (ideas for further investigation appreciated!)

I've had hdmi audio working with an HD7850, but only in Windows (7) and
it was using legacy interrupts for some reason instead of MSI.  I wonder
if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
what's wrong with MSI, but it seems to be new with the PCI bus reset
support.

> - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
>   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> 
> Details:
> 
> - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
>    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
>    - this caused the system to freeze (with color artifacts on the console)
> 
> - In my naivity and because of the "i" in ifgx I tried both with 
>   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
>   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
>   that iommu support was turned off for **all** graphics cards with igfx_off

I'm not sure why this is, looks like the code only tries to turn it off
when only graphics is under the remapping device.  We'd probably need to
see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).

> - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> 
> - iommu groups:
> 
> ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> total 0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> 
> - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:

Ick.  Intel has been pretty good about advertising ACS support on their
root ports.  I wonder if this is an oversight or if they are actually
not isolated from each other.

> # lspci -n
> ...
> 01:00.0 0300: 1002:683f
> 01:00.1 0403: 1002:aab0
> 02:00.0 0300: 1002:6779
> 02:00.1 0403: 1002:aa98
> ...
> 
> modprobe vfio_pci
> echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> # lsusb 
> ...
> Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> ...
> 
> - I also applied your suggested patch to the quirk function in VFIO (see below)
> 
> - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
>   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
>   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> 
> 
> qemu-kvm -M q35 \
>   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
>   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
>   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
>   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
>   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
>   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
>   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
>   -device ide-drive,drive=ivm03,bus=ide.0 \
>   -device ide-cd,drive=cd,bus=ide.1 \
>   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
>   -enable-kvm \
>   -device usb-host,hostbus=1,hostaddr=8 \
>   -device usb-host,hostbus=1,hostaddr=4
> 
> - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 

Did you try scraping the HD6450 while the HD7700 was the boot VGA and
vica versa?  The boot VGA ROM is handled in a special way and what you
really get is the shadow copy, which isn't what we want.

> Anyway, supplying it to vfio did not seem to make any difference.
> 
> find /sys -name rom
> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> ...
> 
> Some observations and remaining unresolved issues:
> 
> - VFIO patch:
>   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
>   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
>   vfio_ati_3c3_quirk_read and repeated logs: 
>     vfio: vfio_vga_read(0x3c3, 1) = 0x0
>   I patched up accordingly with 
> 
> 
> diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> index da0e5f9..a361d06 100644
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
>      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
>                                    addr + quirk->data.base_offset, size);
>  
> -    if (data == quirk->data.address_match) {
> +    if (1 || data == quirk->data.address_match) {
>          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
>          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
>      }
> 
> 
>   This of course did not help much until I actually got the iommu 
>   enabled for the radeons (similar "repeated patters" as deniv reported)
>   but what I have observed after I got it working is that if 
>   I disable the patch above, things are not that well: the Fedora VM 
>   comes up with VGA and the Fedora boot screen, then goes blank when 
>   switching to X.

Hmm, I think we'd probably have better luck making that unconditional
until we have reason to do otherwise.

> - The fact that the iommu group now extends across all my available graphics 
>   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
>   the other card since the vfio_pci driver needs to hold it.
>   Not a complete showstopper since the vesa driver comes up with 1024x768..
>   Might it be a good idea to have an override option (exception list or similar?) 
>   to allow the vfio_pci to be less restrictive about owning the whole group 
>    - allow functionality over security in such case? This of course is further complicated
>   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..

We have a quirk in the kernel that enables us to witelist devices, but
yes, there is no flexibility in this w/o modifying the code and
rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
the example above w/ pci_dev_dma_source - function can just return 1)

> - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
>   The BIOS offers (undocumented) a full range of selections of which PCIe 
>   (or PCIe 1x) graphics card to use as primary, but any other selection 
>   than the first PCIe 16x slot has no effect and the motherboard reverts 
>   to the first slot, so to be able to test both cards, I had to put the card under test
>   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
>   fixes for this in newer BIOSes.
> 
> - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
>   on the new system, using the discrete graphics with HDMI/Displayport audio. 
>   With the HD7700 moved to the second PCIe slot I tested both Windows and 
>   Linux guests to try to get some sound through the HDMI audio device. 
>   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
>   the system settings -> multimedia dialogue never opens up which seems to indicate that 
>   PulseAudio has problems communicating with the passed through device (?), 
>   any hints/pointers here appreciated. From the vfio log it seems at least
>   config space is accessed ok.
> 
> - There also seems to be issues with radeon and intel_iommu=on - if I try 
>   to enable modesetting and normal X support for the radeon cards, X fails to start.
> 
> - It would be nice if the integrated graphics could be used as the host primary display - 
>   I would be happy if someone has any hints as to if/how the ifgx_off option 
>   could be extended/modified to only affect iommu operation on selected device(s),
>   if at all possible..

Let's see what we can discover from your DMAR.  Also send along sudo
lspci -vvv.  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

[-- Attachment #1: Type: text/plain, Size: 16529 bytes --]

On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > Hi all,
> > 
> > Perfect timing from my perspective, thanks Alex!
> > 
> > I spent the better part of the weekend testing your branches on a new system 
> > I just put together for this purpose, results below..
> > 
> > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > ...
> > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > 
> > System setup: 
> > 
> > - Fedora 18 on
> > - Gigabyte Z77X-UD5H motherboard
> > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > - 2 discrete graphics cards:
> > 
> > lspci | egrep 'VGA|Audio'
> > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > 
> > Short summary:
> > 
> > - Once I got past a few time consuming obstacles explained below
> >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> >      on both discrete graphics cards 
> >      (though so far only one at at time and with some minor issues, see below)
> >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> 
> I've had hdmi audio working with an HD7850, but only in Windows (7) and
> it was using legacy interrupts for some reason instead of MSI.  I wonder
> if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> what's wrong with MSI, but it seems to be new with the PCI bus reset
> support.

I tried 

modprobe -r snd_hda_intel
modprobe snd_hda_intel enable_msi=0

- did not seem to have any effect on Linux.
Here is the guest's lspci -vvv entry for the audio after the above:

01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
        Subsystem: PC Partner Limited Device aab0
        Physical Slot: 0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 17
        Region 0: Memory at fea60000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 4072
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: snd_hda_intel

> > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > 
> > Details:
> > 
> > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> >    - this caused the system to freeze (with color artifacts on the console)
> > 
> > - In my naivity and because of the "i" in ifgx I tried both with 
> >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> >   that iommu support was turned off for **all** graphics cards with igfx_off
> 
> I'm not sure why this is, looks like the code only tries to turn it off
> when only graphics is under the remapping device.  We'd probably need to
> see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).

Attaching both a decoded one and the raw dump.

> > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > 
> > - iommu groups:
> > 
> > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > total 0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > 
> > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> 
> Ick.  Intel has been pretty good about advertising ACS support on their
> root ports.  I wonder if this is an oversight or if they are actually
> not isolated from each other.

Sad state I'm afraid - one of the reasons I went for Intel this time - I
have usually chosen AMD in the past but had a bad experience with an FM1
board with no IOMMU support.. 

No ACS on any of the root ports (or anything else..) - see attachment.. 

I wish there were lspci -vvv's out there for all hardware - quite a
gamble to buy motherboards if one wish to utilize it for something more
than a plain Windows install..

> > # lspci -n
> > ...
> > 01:00.0 0300: 1002:683f
> > 01:00.1 0403: 1002:aab0
> > 02:00.0 0300: 1002:6779
> > 02:00.1 0403: 1002:aa98
> > ...
> > 
> > modprobe vfio_pci
> > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> > # lsusb 
> > ...
> > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > ...
> > 
> > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > 
> > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > 
> > 
> > qemu-kvm -M q35 \
> >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> >   -device ide-drive,drive=ivm03,bus=ide.0 \
> >   -device ide-cd,drive=cd,bus=ide.1 \
> >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> >   -enable-kvm \
> >   -device usb-host,hostbus=1,hostaddr=8 \
> >   -device usb-host,hostbus=1,hostaddr=4
> > 
> > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> 
> Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> vica versa?  The boot VGA ROM is handled in a special way and what you
> really get is the shadow copy, which isn't what we want.

I did all the scraping work with the Radeons with my initial setup while
the integrated graphics was the primary display. I tried once more now
to scrape the HD7700 while the HD6450 is the primary VGA and still get
the same result:

# echo 1 > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
# cat /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom > HD7700.rom 
cat: /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom: Input/output error

system log reports:
May 14 07:15:54 asu kernel: [   82.344189] pci 0000:02:00.0: Invalid ROM contents

> 
> > Anyway, supplying it to vfio did not seem to make any difference.
> > 
> > find /sys -name rom
> > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > ...
> > 
> > Some observations and remaining unresolved issues:
> > 
> > - VFIO patch:
> >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> >   vfio_ati_3c3_quirk_read and repeated logs: 
> >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> >   I patched up accordingly with 
> > 
> > 
> > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > index da0e5f9..a361d06 100644
> > --- a/hw/misc/vfio.c
> > +++ b/hw/misc/vfio.c
> > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> >                                    addr + quirk->data.base_offset, size);
> >  
> > -    if (data == quirk->data.address_match) {
> > +    if (1 || data == quirk->data.address_match) {
> >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> >      }
> > 
> > 
> >   This of course did not help much until I actually got the iommu 
> >   enabled for the radeons (similar "repeated patters" as deniv reported)
> >   but what I have observed after I got it working is that if 
> >   I disable the patch above, things are not that well: the Fedora VM 
> >   comes up with VGA and the Fedora boot screen, then goes blank when 
> >   switching to X.
> 
> Hmm, I think we'd probably have better luck making that unconditional
> until we have reason to do otherwise.

I'm starting to wonder whether there's some timing issue or maybe
something with the initial state of the hardware affecting this.
It might be that the blank screen situation is more likely to occur if
debug is enabled - this morning I saw the same behavior even with the
patch enabled - then tried once more and got success. This was right
after a reboot so I tried a "warm" reboot and the same happened again:
First attempt got through the initial VGA phase then blanked, I ^C'ed
the VM then restarted and got all the way to the GUI again..

> > - The fact that the iommu group now extends across all my available graphics 
> >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> >   the other card since the vfio_pci driver needs to hold it.
> >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> >   Might it be a good idea to have an override option (exception list or similar?) 
> >   to allow the vfio_pci to be less restrictive about owning the whole group 
> >    - allow functionality over security in such case? This of course is further complicated
> >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> 
> We have a quirk in the kernel that enables us to witelist devices, but
> yes, there is no flexibility in this w/o modifying the code and
> rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> the example above w/ pci_dev_dma_source - function can just return 1)

Thanks, I'll have a look at that,

> > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> >   to the first slot, so to be able to test both cards, I had to put the card under test
> >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> >   fixes for this in newer BIOSes.
> > 
> > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> >   Linux guests to try to get some sound through the HDMI audio device. 
> >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> >   PulseAudio has problems communicating with the passed through device (?), 
> >   any hints/pointers here appreciated. From the vfio log it seems at least
> >   config space is accessed ok.
> > 
> > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > 
> > - It would be nice if the integrated graphics could be used as the host primary display - 
> >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> >   could be extended/modified to only affect iommu operation on selected device(s),
> >   if at all possible..
> 
> Let's see what we can discover from your DMAR.  Also send along sudo
> lspci -vvv.  Thanks,

Attached.

One "interesting feature" I have never seen before on a motherboard is 
that I get "pcilib: sysfs_read_vpd: read failed: Connection timed out"
while doing the lspci -vvv but this appears to come from trying to read
the Vital Product Data capability of the secondary onboard ethernet
[07:00.0 Ethernet controller: Atheros Communications Inc. AR8151 v2.0
Gigabit Ethernet (rev c0)] which should not have any significance here..

Thanks,

Knut


[-- Attachment #2: lspci_asu.txt --]
[-- Type: text/plain, Size: 51173 bytes --]

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
	Subsystem: Giga-byte Technology Device 5000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: f7e00000-f7efffff
	Prefetchable memory behind bridge: 00000000e0000000-00000000efffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [88] Subsystem: Giga-byte Technology Device 5000
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee002d8  Data: 0000
	Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 8GT/s, Width x8, ASPM unknown, Latency L0 <256ns, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [140 v1] Root Complex Link
		Desc:	PortNumber=02 ComponentID=01 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed19000
	Capabilities: [d94 v1] #19
	Kernel driver in use: pcieport

00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f7d00000-f7dfffff
	Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [88] Subsystem: Giga-byte Technology Device 5000
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee002f8  Data: 0000
	Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 8GT/s, Width x8, ASPM unknown, Latency L0 <256ns, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #2, PowerLimit 75.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [140 v1] Root Complex Link
		Desc:	PortNumber=03 ComponentID=01 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed19000
	Capabilities: [d94 v1] #19
	Kernel driver in use: pcieport

00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Giga-byte Technology Device 5007
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 46
	Region 0: Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
		Address: 00000000fee00378  Data: 0000
	Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
	Subsystem: Giga-byte Technology Device 1c3a
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 47
	Region 0: Memory at f7f3b000 (64-bit, non-prefetchable) [size=16]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00398  Data: 0000
	Kernel driver in use: mei

00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 04)
	Subsystem: Giga-byte Technology Device e000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 51
	Region 0: Memory at f7f00000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at f7f39000 (32-bit, non-prefetchable) [size=4K]
	Region 2: I/O ports at f040 [size=32]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00458  Data: 0000
	Capabilities: [e0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: e1000e

00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Giga-byte Technology Device 5006
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f7f38000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
	Subsystem: Giga-byte Technology Device a072
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 48
	Region 0: Memory at f7f30000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee003d8  Data: 0000
	Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE- FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=1 ArbSelect=Fixed TC/VC=22
			Status:	NegoPending- InProgress-
	Capabilities: [130 v1] Root Complex Link
		Desc:	PortNumber=0f ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #1, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <1us, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train+ SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Giga-byte Technology Device 5001
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pcieport

00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: f7c00000-f7cfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #2, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Giga-byte Technology Device 5001
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=05, subordinate=06, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f7b00000-f7bfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #6, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <1us, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #5, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Giga-byte Technology Device 5001
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: f7a00000-f7afffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #7, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #6, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Giga-byte Technology Device 5001
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pcieport

00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: 00009000-00009fff
	Memory behind bridge: f7900000-f79fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #8, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #7, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Giga-byte Technology Device 5001
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Giga-byte Technology Device 5006
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 0: Memory at f7f37000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
	Subsystem: Giga-byte Technology Device 5001
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: lpc_ich

00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
	Subsystem: Giga-byte Technology Device b005
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 43
	Region 0: I/O ports at f090 [size=8]
	Region 1: I/O ports at f080 [size=4]
	Region 2: I/O ports at f070 [size=8]
	Region 3: I/O ports at f060 [size=4]
	Region 4: I/O ports at f020 [size=32]
	Region 5: Memory at f7f36000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00318  Data: 0000
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Capabilities: [b0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
	Subsystem: Giga-byte Technology Device 5001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at f7f35000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at f000 [size=32]

01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450] (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device 0461
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at f7e20000 (64-bit, non-prefetchable) [size=128K]
	Region 4: I/O ports at e000 [size=256]
	Expansion ROM at f7e00000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: vfio-pci

01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
	Subsystem: ASUSTeK Computer Inc. Device aa98
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 17
	Region 0: Memory at f7e40000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 00000000fee003f8  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: vfio-pci

02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited Device e213
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M]
	Region 2: Memory at f7d00000 (64-bit, non-prefetchable) [disabled] [size=256K]
	Region 4: I/O ports at d000 [disabled] [size=256]
	Expansion ROM at f7d40000 [disabled by cmd] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [270 v1] #19
	Kernel driver in use: vfio-pci

02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
	Subsystem: PC Partner Limited Device aab0
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 18
	Region 0: Memory at f7d60000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 00000000fee00418  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: vfio-pci

04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
	Subsystem: Giga-byte Technology Device b000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 44
	Region 0: I/O ports at c040 [size=8]
	Region 1: I/O ports at c030 [size=4]
	Region 2: I/O ports at c020 [size=8]
	Region 3: I/O ports at c010 [size=4]
	Region 4: I/O ports at c000 [size=16]
	Region 5: Memory at f7c10000 (32-bit, non-prefetchable) [size=512]
	Expansion ROM at f7c00000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00338  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: ahci

05:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=05, secondary=06, subordinate=06, sec-latency=32
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f7b00000-f7bfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
	Capabilities: [90] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=55mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a0] Subsystem: Giga-byte Technology Device 8892

06:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology GA-7VT600-1394 Motherboard
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (8000ns max), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at f7b00000 (32-bit, non-prefetchable) [size=2K]
	Region 1: I/O ports at b000 [size=128]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: firewire_ohci

07:00.0 Ethernet controller: Atheros Communications Inc. AR8151 v2.0 Gigabit Ethernet (rev c0)
	Subsystem: Giga-byte Technology Device e000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 52
	Region 0: Memory at f7a00000 (64-bit, non-prefetchable) [size=256K]
	Region 2: I/O ports at a000 [size=128]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00478  Data: 0000
	Capabilities: [58] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 4096 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn+ AttnInd+ PwrInd+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 unlimited, L1 unlimited
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [6c] Vital Product Data
		Not readable
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [180 v1] Device Serial Number ff-53-8a-79-90-2b-34-ff
	Kernel driver in use: atl1c

08:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
	Subsystem: Giga-byte Technology Device b000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 45
	Region 0: I/O ports at 9040 [size=8]
	Region 1: I/O ports at 9030 [size=4]
	Region 2: I/O ports at 9020 [size=8]
	Region 3: I/O ports at 9010 [size=4]
	Region 4: I/O ports at 9000 [size=16]
	Region 5: Memory at f7910000 (32-bit, non-prefetchable) [size=512]
	Expansion ROM at f7900000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00358  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: ahci


[-- Attachment #3: DMAR_asu.dsl --]
[-- Type: text/x-dsl, Size: 3673 bytes --]

/*
 * Intel ACPI Component Architecture
 * AML Disassembler version 20100528
 *
 * Disassembly of DMAR_asu.raw, Tue May 14 08:29:49 2013
 *
 * ACPI Data Table [DMAR]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 0000  4]                    Signature : "DMAR"    /* DMA Remapping table */
[004h 0004  4]                 Table Length : 00000080
[008h 0008  1]                     Revision : 01
[009h 0009  1]                     Checksum : E2
[00Ah 0010  6]                       Oem ID : "INTEL "
[010h 0016  8]                 Oem Table ID : "SNB "
[018h 0024  4]                 Oem Revision : 00000001
[01Ch 0028  4]              Asl Compiler ID : "INTL"
[020h 0032  4]        Asl Compiler Revision : 00000001

[024h 0036  1]           Host Address Width : 23
[025h 0037  1]                        Flags : 01

[030h 0048  2]                Subtable Type : 0000 <Hardware Unit Definition>
[032h 0050  2]                       Length : 0020
[034h 0052  1]                        Flags : 01
[035h 0053  1]                     Reserved : 00
[036h 0054  2]           PCI Segment Number : 0000
[038h 0056  8]        Register Base Address : 00000000FED90000

[040h 0064  1]      Device Scope Entry Type : 03
[041h 0065  1]                 Entry Length : 08
[042h 0066  2]                     Reserved : 0000
[044h 0068  1]               Enumeration ID : 02
[045h 0069  1]               PCI Bus Number : F0
[046h 0070  2]                     PCI Path : [1F, 00]

[048h 0072  1]      Device Scope Entry Type : 04
[049h 0073  1]                 Entry Length : 08
[04Ah 0074  2]                     Reserved : 0000
[04Ch 0076  1]               Enumeration ID : 00
[04Dh 0077  1]               PCI Bus Number : F0
[04Eh 0078  2]                     PCI Path : [0F, 00]

[050h 0080  2]                Subtable Type : 0001 <Reserved Memory Region>
[052h 0082  2]                       Length : 0030
[054h 0084  2]                     Reserved : 0000
[056h 0086  2]           PCI Segment Number : 0000
[058h 0088  8]                 Base Address : 00000000CE7DC000
[060h 0096  8]          End Address (limit) : 00000000CE806FFF

[068h 0104  1]      Device Scope Entry Type : 01
[069h 0105  1]                 Entry Length : 08
[06Ah 0106  2]                     Reserved : 0000
[06Ch 0108  1]               Enumeration ID : 00
[06Dh 0109  1]               PCI Bus Number : 00
[06Eh 0110  2]                     PCI Path : [1D, 00]

[070h 0112  1]      Device Scope Entry Type : 01
[071h 0113  1]                 Entry Length : 08
[072h 0114  2]                     Reserved : 0000
[074h 0116  1]               Enumeration ID : 00
[075h 0117  1]               PCI Bus Number : 00
[076h 0118  2]                     PCI Path : [1A, 00]

[078h 0120  1]      Device Scope Entry Type : 01
[079h 0121  1]                 Entry Length : 08
[07Ah 0122  2]                     Reserved : 0000
[07Ch 0124  1]               Enumeration ID : 00
[07Dh 0125  1]               PCI Bus Number : 00
[07Eh 0126  2]                     PCI Path : [14, 00]

Raw Table Data

  0000: 44 4D 41 52 80 00 00 00 01 E2 49 4E 54 45 4C 20  DMAR......INTEL 
  0010: 53 4E 42 20 00 00 00 00 01 00 00 00 49 4E 54 4C  SNB ........INTL
  0020: 01 00 00 00 23 01 00 00 00 00 00 00 00 00 00 00  ....#...........
  0030: 00 00 20 00 01 00 00 00 00 00 D9 FE 00 00 00 00  .. .............
  0040: 03 08 00 00 02 F0 1F 00 04 08 00 00 00 F0 0F 00  ................
  0050: 01 00 30 00 00 00 00 00 00 C0 7D CE 00 00 00 00  ..0.......}.....
  0060: FF 6F 80 CE 00 00 00 00 01 08 00 00 00 00 1D 00  .o..............
  0070: 01 08 00 00 00 00 1A 00 01 08 00 00 00 00 14 00  ................

[-- Attachment #4: DMAR_asu.raw --]
[-- Type: image/x-panasonic-raw, Size: 128 bytes --]

Re: VFIO VGA test branches

[Knut Omang]

[-- Attachment #1: Type: text/plain, Size: 12356 bytes --]


On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > Hi all,
> > 
> > Perfect timing from my perspective, thanks Alex!
> > 
> > I spent the better part of the weekend testing your branches on a new system 
> > I just put together for this purpose, results below..
> > 
> > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > ...
> > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > 
> > System setup: 
> > 
> > - Fedora 18 on
> > - Gigabyte Z77X-UD5H motherboard
> > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > - 2 discrete graphics cards:
> > 
> > lspci | egrep 'VGA|Audio'
> > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > 
> > Short summary:
> > 
> > - Once I got past a few time consuming obstacles explained below
> >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> >      on both discrete graphics cards 
> >      (though so far only one at at time and with some minor issues, see below)
> >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> 
> I've had hdmi audio working with an HD7850, but only in Windows (7) and
> it was using legacy interrupts for some reason instead of MSI.  I wonder
> if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> what's wrong with MSI, but it seems to be new with the PCI bus reset
> support.

In my first tries, Windows were just using a generic
VGA driver, which still seems to work perfect with reboots and everything 
and in full screen resolution (1920x1200).
However after installing the Catalyst AMD driver stack, upon boot
Windows 7 now consequently get a BSOD from the graphics driver
with the message:

"Attempt to reset the display driver and recover from timeout failed"
- a picture of the BSOD screen attached.

I attach the corresponding vfio log where I added some timing code to
make it easier to see when the BSOD happens (with 2 seconds of silence
in the log before the VM reboots, I believe this is at 09:28:32-34 in
the log.

Similar behaviour both just after reboot/power cycle of the host and
subsequent VM boot attempts.

This is still with the HD7700 as passed through device, but after a
motherboard firmware upgrade (to F14) which did not seem to affect the
observed behaviour on Windows prior to Catalyst install or with Linux
guest, neither did it fix the bug in selecting primary devices as I 
was hoping for.

Let me know if you have ideas for further debugging this,

Thanks,

Knut

> > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > 
> > Details:
> > 
> > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> >    - this caused the system to freeze (with color artifacts on the console)
> > 
> > - In my naivity and because of the "i" in ifgx I tried both with 
> >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> >   that iommu support was turned off for **all** graphics cards with igfx_off
> 
> I'm not sure why this is, looks like the code only tries to turn it off
> when only graphics is under the remapping device.  We'd probably need to
> see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> 
> > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > 
> > - iommu groups:
> > 
> > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > total 0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > 
> > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> 
> Ick.  Intel has been pretty good about advertising ACS support on their
> root ports.  I wonder if this is an oversight or if they are actually
> not isolated from each other.
> 
> > # lspci -n
> > ...
> > 01:00.0 0300: 1002:683f
> > 01:00.1 0403: 1002:aab0
> > 02:00.0 0300: 1002:6779
> > 02:00.1 0403: 1002:aa98
> > ...
> > 
> > modprobe vfio_pci
> > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> > # lsusb 
> > ...
> > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > ...
> > 
> > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > 
> > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > 
> > 
> > qemu-kvm -M q35 \
> >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> >   -device ide-drive,drive=ivm03,bus=ide.0 \
> >   -device ide-cd,drive=cd,bus=ide.1 \
> >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> >   -enable-kvm \
> >   -device usb-host,hostbus=1,hostaddr=8 \
> >   -device usb-host,hostbus=1,hostaddr=4
> > 
> > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> 
> Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> vica versa?  The boot VGA ROM is handled in a special way and what you
> really get is the shadow copy, which isn't what we want.
> 
> > Anyway, supplying it to vfio did not seem to make any difference.
> > 
> > find /sys -name rom
> > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > ...
> > 
> > Some observations and remaining unresolved issues:
> > 
> > - VFIO patch:
> >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> >   vfio_ati_3c3_quirk_read and repeated logs: 
> >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> >   I patched up accordingly with 
> > 
> > 
> > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > index da0e5f9..a361d06 100644
> > --- a/hw/misc/vfio.c
> > +++ b/hw/misc/vfio.c
> > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> >                                    addr + quirk->data.base_offset, size);
> >  
> > -    if (data == quirk->data.address_match) {
> > +    if (1 || data == quirk->data.address_match) {
> >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> >      }
> > 
> > 
> >   This of course did not help much until I actually got the iommu 
> >   enabled for the radeons (similar "repeated patters" as deniv reported)
> >   but what I have observed after I got it working is that if 
> >   I disable the patch above, things are not that well: the Fedora VM 
> >   comes up with VGA and the Fedora boot screen, then goes blank when 
> >   switching to X.
> 
> Hmm, I think we'd probably have better luck making that unconditional
> until we have reason to do otherwise.
> 
> > - The fact that the iommu group now extends across all my available graphics 
> >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> >   the other card since the vfio_pci driver needs to hold it.
> >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> >   Might it be a good idea to have an override option (exception list or similar?) 
> >   to allow the vfio_pci to be less restrictive about owning the whole group 
> >    - allow functionality over security in such case? This of course is further complicated
> >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> 
> We have a quirk in the kernel that enables us to witelist devices, but
> yes, there is no flexibility in this w/o modifying the code and
> rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> the example above w/ pci_dev_dma_source - function can just return 1)
> 
> > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> >   to the first slot, so to be able to test both cards, I had to put the card under test
> >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> >   fixes for this in newer BIOSes.
> > 
> > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> >   Linux guests to try to get some sound through the HDMI audio device. 
> >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> >   PulseAudio has problems communicating with the passed through device (?), 
> >   any hints/pointers here appreciated. From the vfio log it seems at least
> >   config space is accessed ok.
> > 
> > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > 
> > - It would be nice if the integrated graphics could be used as the host primary display - 
> >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> >   could be extended/modified to only affect iommu operation on selected device(s),
> >   if at all possible..
> 
> Let's see what we can discover from your DMAR.  Also send along sudo
> lspci -vvv.  Thanks,
> 
> Alex
> 



[-- Attachment #2: bsod_ivm03.jpg --]
[-- Type: image/jpeg, Size: 107784 bytes --]

[-- Attachment #3: ivm03_bsod.log.bz2 --]
[-- Type: application/x-bzip, Size: 136474 bytes --]

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

[-- Attachment #1: Type: text/plain, Size: 12356 bytes --]


On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > Hi all,
> > 
> > Perfect timing from my perspective, thanks Alex!
> > 
> > I spent the better part of the weekend testing your branches on a new system 
> > I just put together for this purpose, results below..
> > 
> > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > ...
> > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > 
> > System setup: 
> > 
> > - Fedora 18 on
> > - Gigabyte Z77X-UD5H motherboard
> > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > - 2 discrete graphics cards:
> > 
> > lspci | egrep 'VGA|Audio'
> > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > 
> > Short summary:
> > 
> > - Once I got past a few time consuming obstacles explained below
> >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> >      on both discrete graphics cards 
> >      (though so far only one at at time and with some minor issues, see below)
> >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> 
> I've had hdmi audio working with an HD7850, but only in Windows (7) and
> it was using legacy interrupts for some reason instead of MSI.  I wonder
> if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> what's wrong with MSI, but it seems to be new with the PCI bus reset
> support.

In my first tries, Windows were just using a generic
VGA driver, which still seems to work perfect with reboots and everything 
and in full screen resolution (1920x1200).
However after installing the Catalyst AMD driver stack, upon boot
Windows 7 now consequently get a BSOD from the graphics driver
with the message:

"Attempt to reset the display driver and recover from timeout failed"
- a picture of the BSOD screen attached.

I attach the corresponding vfio log where I added some timing code to
make it easier to see when the BSOD happens (with 2 seconds of silence
in the log before the VM reboots, I believe this is at 09:28:32-34 in
the log.

Similar behaviour both just after reboot/power cycle of the host and
subsequent VM boot attempts.

This is still with the HD7700 as passed through device, but after a
motherboard firmware upgrade (to F14) which did not seem to affect the
observed behaviour on Windows prior to Catalyst install or with Linux
guest, neither did it fix the bug in selecting primary devices as I 
was hoping for.

Let me know if you have ideas for further debugging this,

Thanks,

Knut

> > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > 
> > Details:
> > 
> > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> >    - this caused the system to freeze (with color artifacts on the console)
> > 
> > - In my naivity and because of the "i" in ifgx I tried both with 
> >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> >   that iommu support was turned off for **all** graphics cards with igfx_off
> 
> I'm not sure why this is, looks like the code only tries to turn it off
> when only graphics is under the remapping device.  We'd probably need to
> see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> 
> > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > 
> > - iommu groups:
> > 
> > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > total 0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > 
> > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> 
> Ick.  Intel has been pretty good about advertising ACS support on their
> root ports.  I wonder if this is an oversight or if they are actually
> not isolated from each other.
> 
> > # lspci -n
> > ...
> > 01:00.0 0300: 1002:683f
> > 01:00.1 0403: 1002:aab0
> > 02:00.0 0300: 1002:6779
> > 02:00.1 0403: 1002:aa98
> > ...
> > 
> > modprobe vfio_pci
> > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> > # lsusb 
> > ...
> > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > ...
> > 
> > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > 
> > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > 
> > 
> > qemu-kvm -M q35 \
> >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> >   -device ide-drive,drive=ivm03,bus=ide.0 \
> >   -device ide-cd,drive=cd,bus=ide.1 \
> >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> >   -enable-kvm \
> >   -device usb-host,hostbus=1,hostaddr=8 \
> >   -device usb-host,hostbus=1,hostaddr=4
> > 
> > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> 
> Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> vica versa?  The boot VGA ROM is handled in a special way and what you
> really get is the shadow copy, which isn't what we want.
> 
> > Anyway, supplying it to vfio did not seem to make any difference.
> > 
> > find /sys -name rom
> > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > ...
> > 
> > Some observations and remaining unresolved issues:
> > 
> > - VFIO patch:
> >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> >   vfio_ati_3c3_quirk_read and repeated logs: 
> >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> >   I patched up accordingly with 
> > 
> > 
> > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > index da0e5f9..a361d06 100644
> > --- a/hw/misc/vfio.c
> > +++ b/hw/misc/vfio.c
> > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> >                                    addr + quirk->data.base_offset, size);
> >  
> > -    if (data == quirk->data.address_match) {
> > +    if (1 || data == quirk->data.address_match) {
> >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> >      }
> > 
> > 
> >   This of course did not help much until I actually got the iommu 
> >   enabled for the radeons (similar "repeated patters" as deniv reported)
> >   but what I have observed after I got it working is that if 
> >   I disable the patch above, things are not that well: the Fedora VM 
> >   comes up with VGA and the Fedora boot screen, then goes blank when 
> >   switching to X.
> 
> Hmm, I think we'd probably have better luck making that unconditional
> until we have reason to do otherwise.
> 
> > - The fact that the iommu group now extends across all my available graphics 
> >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> >   the other card since the vfio_pci driver needs to hold it.
> >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> >   Might it be a good idea to have an override option (exception list or similar?) 
> >   to allow the vfio_pci to be less restrictive about owning the whole group 
> >    - allow functionality over security in such case? This of course is further complicated
> >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> 
> We have a quirk in the kernel that enables us to witelist devices, but
> yes, there is no flexibility in this w/o modifying the code and
> rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> the example above w/ pci_dev_dma_source - function can just return 1)
> 
> > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> >   to the first slot, so to be able to test both cards, I had to put the card under test
> >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> >   fixes for this in newer BIOSes.
> > 
> > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> >   Linux guests to try to get some sound through the HDMI audio device. 
> >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> >   PulseAudio has problems communicating with the passed through device (?), 
> >   any hints/pointers here appreciated. From the vfio log it seems at least
> >   config space is accessed ok.
> > 
> > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > 
> > - It would be nice if the integrated graphics could be used as the host primary display - 
> >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> >   could be extended/modified to only affect iommu operation on selected device(s),
> >   if at all possible..
> 
> Let's see what we can discover from your DMAR.  Also send along sudo
> lspci -vvv.  Thanks,
> 
> Alex
> 



[-- Attachment #2: bsod_ivm03.jpg --]
[-- Type: image/jpeg, Size: 107784 bytes --]

[-- Attachment #3: ivm03_bsod.log.bz2 --]
[-- Type: application/x-bzip, Size: 136474 bytes --]

Re: VFIO VGA test branches

[Maik Broemme]

Hi Knut,

Knut Omang <knuto@ifi.uio.no> wrote:
> 
> On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > Hi all,
> > > 
> > > Perfect timing from my perspective, thanks Alex!
> > > 
> > > I spent the better part of the weekend testing your branches on a new system 
> > > I just put together for this purpose, results below..
> > > 
> > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > ...
> > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > 
> > > System setup: 
> > > 
> > > - Fedora 18 on
> > > - Gigabyte Z77X-UD5H motherboard
> > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > - 2 discrete graphics cards:
> > > 
> > > lspci | egrep 'VGA|Audio'
> > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > 
> > > Short summary:
> > > 
> > > - Once I got past a few time consuming obstacles explained below
> > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > >      on both discrete graphics cards 
> > >      (though so far only one at at time and with some minor issues, see below)
> > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > 
> > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > support.
> 
> In my first tries, Windows were just using a generic
> VGA driver, which still seems to work perfect with reboots and everything 
> and in full screen resolution (1920x1200).
> However after installing the Catalyst AMD driver stack, upon boot
> Windows 7 now consequently get a BSOD from the graphics driver
> with the message:
> 
> "Attempt to reset the display driver and recover from timeout failed"
> - a picture of the BSOD screen attached.
> 
> I attach the corresponding vfio log where I added some timing code to
> make it easier to see when the BSOD happens (with 2 seconds of silence
> in the log before the VM reboots, I believe this is at 09:28:32-34 in
> the log.
> 
> Similar behaviour both just after reboot/power cycle of the host and
> subsequent VM boot attempts.
> 
> This is still with the HD7700 as passed through device, but after a
> motherboard firmware upgrade (to F14) which did not seem to affect the
> observed behaviour on Windows prior to Catalyst install or with Linux
> guest, neither did it fix the bug in selecting primary devices as I 
> was hoping for.
> 
> Let me know if you have ideas for further debugging this,
> 

I had a similar problem a couple of days ago and posted it in this list.
I got similar BSOD and tested already the following configurations:

1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
   (with errors below)
3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
   100% CPU due to no VGA init (no picture)
4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
   BOSD on loading atikmpag.sys
5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine

However I've re-run the BSOD case already with the following branches
from Alex:

git://github.com/awilliam/linux-vfio.git vfio-vga-reset
git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

Also with latest seabios and it worked so far. No more BSOD and reboot
of VM was also possible without suspend / resume the host between.

> Thanks,
> 
> Knut
> 
> > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > 
> > > Details:
> > > 
> > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > >    - this caused the system to freeze (with color artifacts on the console)
> > > 
> > > - In my naivity and because of the "i" in ifgx I tried both with 
> > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > 
> > I'm not sure why this is, looks like the code only tries to turn it off
> > when only graphics is under the remapping device.  We'd probably need to
> > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > 
> > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > 
> > > - iommu groups:
> > > 
> > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > total 0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > 
> > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > 
> > Ick.  Intel has been pretty good about advertising ACS support on their
> > root ports.  I wonder if this is an oversight or if they are actually
> > not isolated from each other.
> > 
> > > # lspci -n
> > > ...
> > > 01:00.0 0300: 1002:683f
> > > 01:00.1 0403: 1002:aab0
> > > 02:00.0 0300: 1002:6779
> > > 02:00.1 0403: 1002:aa98
> > > ...
> > > 
> > > modprobe vfio_pci
> > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > 
> > > # lsusb 
> > > ...
> > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > ...
> > > 
> > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > 
> > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > 
> > > 
> > > qemu-kvm -M q35 \
> > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > >   -device ide-cd,drive=cd,bus=ide.1 \
> > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > >   -enable-kvm \
> > >   -device usb-host,hostbus=1,hostaddr=8 \
> > >   -device usb-host,hostbus=1,hostaddr=4
> > > 
> > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > 
> > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > vica versa?  The boot VGA ROM is handled in a special way and what you
> > really get is the shadow copy, which isn't what we want.
> > 
> > > Anyway, supplying it to vfio did not seem to make any difference.
> > > 
> > > find /sys -name rom
> > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > ...
> > > 
> > > Some observations and remaining unresolved issues:
> > > 
> > > - VFIO patch:
> > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > >   I patched up accordingly with 
> > > 
> > > 
> > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > index da0e5f9..a361d06 100644
> > > --- a/hw/misc/vfio.c
> > > +++ b/hw/misc/vfio.c
> > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > >                                    addr + quirk->data.base_offset, size);
> > >  
> > > -    if (data == quirk->data.address_match) {
> > > +    if (1 || data == quirk->data.address_match) {
> > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > >      }
> > > 
> > > 
> > >   This of course did not help much until I actually got the iommu 
> > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > >   but what I have observed after I got it working is that if 
> > >   I disable the patch above, things are not that well: the Fedora VM 
> > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > >   switching to X.
> > 
> > Hmm, I think we'd probably have better luck making that unconditional
> > until we have reason to do otherwise.
> > 
> > > - The fact that the iommu group now extends across all my available graphics 
> > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > >   the other card since the vfio_pci driver needs to hold it.
> > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > >   Might it be a good idea to have an override option (exception list or similar?) 
> > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > >    - allow functionality over security in such case? This of course is further complicated
> > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > 
> > We have a quirk in the kernel that enables us to witelist devices, but
> > yes, there is no flexibility in this w/o modifying the code and
> > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > the example above w/ pci_dev_dma_source - function can just return 1)
> > 
> > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > >   fixes for this in newer BIOSes.
> > > 
> > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > >   Linux guests to try to get some sound through the HDMI audio device. 
> > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > >   PulseAudio has problems communicating with the passed through device (?), 
> > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > >   config space is accessed ok.
> > > 
> > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > 
> > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > >   could be extended/modified to only affect iommu operation on selected device(s),
> > >   if at all possible..
> > 
> > Let's see what we can discover from your DMAR.  Also send along sudo
> > lspci -vvv.  Thanks,
> > 
> > Alex
> > 
> 
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi Knut,

Knut Omang <knuto@ifi.uio.no> wrote:
> 
> On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > Hi all,
> > > 
> > > Perfect timing from my perspective, thanks Alex!
> > > 
> > > I spent the better part of the weekend testing your branches on a new system 
> > > I just put together for this purpose, results below..
> > > 
> > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > ...
> > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > 
> > > System setup: 
> > > 
> > > - Fedora 18 on
> > > - Gigabyte Z77X-UD5H motherboard
> > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > - 2 discrete graphics cards:
> > > 
> > > lspci | egrep 'VGA|Audio'
> > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > 
> > > Short summary:
> > > 
> > > - Once I got past a few time consuming obstacles explained below
> > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > >      on both discrete graphics cards 
> > >      (though so far only one at at time and with some minor issues, see below)
> > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > 
> > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > support.
> 
> In my first tries, Windows were just using a generic
> VGA driver, which still seems to work perfect with reboots and everything 
> and in full screen resolution (1920x1200).
> However after installing the Catalyst AMD driver stack, upon boot
> Windows 7 now consequently get a BSOD from the graphics driver
> with the message:
> 
> "Attempt to reset the display driver and recover from timeout failed"
> - a picture of the BSOD screen attached.
> 
> I attach the corresponding vfio log where I added some timing code to
> make it easier to see when the BSOD happens (with 2 seconds of silence
> in the log before the VM reboots, I believe this is at 09:28:32-34 in
> the log.
> 
> Similar behaviour both just after reboot/power cycle of the host and
> subsequent VM boot attempts.
> 
> This is still with the HD7700 as passed through device, but after a
> motherboard firmware upgrade (to F14) which did not seem to affect the
> observed behaviour on Windows prior to Catalyst install or with Linux
> guest, neither did it fix the bug in selecting primary devices as I 
> was hoping for.
> 
> Let me know if you have ideas for further debugging this,
> 

I had a similar problem a couple of days ago and posted it in this list.
I got similar BSOD and tested already the following configurations:

1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
   (with errors below)
3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
   100% CPU due to no VGA init (no picture)
4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
   BOSD on loading atikmpag.sys
5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine

However I've re-run the BSOD case already with the following branches
from Alex:

git://github.com/awilliam/linux-vfio.git vfio-vga-reset
git://github.com/awilliam/qemu-vfio.git vfio-vga-reset

Also with latest seabios and it worked so far. No more BSOD and reboot
of VM was also possible without suspend / resume the host between.

> Thanks,
> 
> Knut
> 
> > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > 
> > > Details:
> > > 
> > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > >    - this caused the system to freeze (with color artifacts on the console)
> > > 
> > > - In my naivity and because of the "i" in ifgx I tried both with 
> > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > 
> > I'm not sure why this is, looks like the code only tries to turn it off
> > when only graphics is under the remapping device.  We'd probably need to
> > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > 
> > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > 
> > > - iommu groups:
> > > 
> > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > total 0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > 
> > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > 
> > Ick.  Intel has been pretty good about advertising ACS support on their
> > root ports.  I wonder if this is an oversight or if they are actually
> > not isolated from each other.
> > 
> > > # lspci -n
> > > ...
> > > 01:00.0 0300: 1002:683f
> > > 01:00.1 0403: 1002:aab0
> > > 02:00.0 0300: 1002:6779
> > > 02:00.1 0403: 1002:aa98
> > > ...
> > > 
> > > modprobe vfio_pci
> > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > 
> > > # lsusb 
> > > ...
> > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > ...
> > > 
> > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > 
> > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > 
> > > 
> > > qemu-kvm -M q35 \
> > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > >   -device ide-cd,drive=cd,bus=ide.1 \
> > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > >   -enable-kvm \
> > >   -device usb-host,hostbus=1,hostaddr=8 \
> > >   -device usb-host,hostbus=1,hostaddr=4
> > > 
> > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > 
> > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > vica versa?  The boot VGA ROM is handled in a special way and what you
> > really get is the shadow copy, which isn't what we want.
> > 
> > > Anyway, supplying it to vfio did not seem to make any difference.
> > > 
> > > find /sys -name rom
> > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > ...
> > > 
> > > Some observations and remaining unresolved issues:
> > > 
> > > - VFIO patch:
> > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > >   I patched up accordingly with 
> > > 
> > > 
> > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > index da0e5f9..a361d06 100644
> > > --- a/hw/misc/vfio.c
> > > +++ b/hw/misc/vfio.c
> > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > >                                    addr + quirk->data.base_offset, size);
> > >  
> > > -    if (data == quirk->data.address_match) {
> > > +    if (1 || data == quirk->data.address_match) {
> > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > >      }
> > > 
> > > 
> > >   This of course did not help much until I actually got the iommu 
> > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > >   but what I have observed after I got it working is that if 
> > >   I disable the patch above, things are not that well: the Fedora VM 
> > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > >   switching to X.
> > 
> > Hmm, I think we'd probably have better luck making that unconditional
> > until we have reason to do otherwise.
> > 
> > > - The fact that the iommu group now extends across all my available graphics 
> > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > >   the other card since the vfio_pci driver needs to hold it.
> > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > >   Might it be a good idea to have an override option (exception list or similar?) 
> > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > >    - allow functionality over security in such case? This of course is further complicated
> > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > 
> > We have a quirk in the kernel that enables us to witelist devices, but
> > yes, there is no flexibility in this w/o modifying the code and
> > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > the example above w/ pci_dev_dma_source - function can just return 1)
> > 
> > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > >   fixes for this in newer BIOSes.
> > > 
> > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > >   Linux guests to try to get some sound through the HDMI audio device. 
> > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > >   PulseAudio has problems communicating with the passed through device (?), 
> > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > >   config space is accessed ok.
> > > 
> > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > 
> > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > >   could be extended/modified to only affect iommu operation on selected device(s),
> > >   if at all possible..
> > 
> > Let's see what we can discover from your DMAR.  Also send along sudo
> > lspci -vvv.  Thanks,
> > 
> > Alex
> > 
> 
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Sun, 2013-05-19 at 23:26 +0400, Maik Broemme wrote:
> Hi Knut,
> 
> Knut Omang <knuto@ifi.uio.no> wrote:
> > 
> > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > Hi all,
> > > > 
> > > > Perfect timing from my perspective, thanks Alex!
> > > > 
> > > > I spent the better part of the weekend testing your branches on a new system 
> > > > I just put together for this purpose, results below..
> > > > 
> > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > ...
> > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > 
> > > > System setup: 
> > > > 
> > > > - Fedora 18 on
> > > > - Gigabyte Z77X-UD5H motherboard
> > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > - 2 discrete graphics cards:
> > > > 
> > > > lspci | egrep 'VGA|Audio'
> > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 
> > > > Short summary:
> > > > 
> > > > - Once I got past a few time consuming obstacles explained below
> > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > >      on both discrete graphics cards 
> > > >      (though so far only one at at time and with some minor issues, see below)
> > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > 
> > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > support.
> > 
> > In my first tries, Windows were just using a generic
> > VGA driver, which still seems to work perfect with reboots and everything 
> > and in full screen resolution (1920x1200).
> > However after installing the Catalyst AMD driver stack, upon boot
> > Windows 7 now consequently get a BSOD from the graphics driver
> > with the message:
> > 
> > "Attempt to reset the display driver and recover from timeout failed"
> > - a picture of the BSOD screen attached.
> > 
> > I attach the corresponding vfio log where I added some timing code to
> > make it easier to see when the BSOD happens (with 2 seconds of silence
> > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > the log.
> > 
> > Similar behaviour both just after reboot/power cycle of the host and
> > subsequent VM boot attempts.
> > 
> > This is still with the HD7700 as passed through device, but after a
> > motherboard firmware upgrade (to F14) which did not seem to affect the
> > observed behaviour on Windows prior to Catalyst install or with Linux
> > guest, neither did it fix the bug in selecting primary devices as I 
> > was hoping for.
> > 
> > Let me know if you have ideas for further debugging this,
> > 
> 
> I had a similar problem a couple of days ago and posted it in this list.
> I got similar BSOD and tested already the following configurations:
> 
> 1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
> 2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
>    (with errors below)
> 3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
>    100% CPU due to no VGA init (no picture)
> 4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
>    BOSD on loading atikmpag.sys
> 5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine
> 
> However I've re-run the BSOD case already with the following branches
> from Alex:
> 
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> Also with latest seabios and it worked so far. No more BSOD and reboot
> of VM was also possible without suspend / resume the host between.

Good to hear.  It looks like you have the same motherboard as my AMD
test system.  An HD7850 in that system runs quite reliably with the
branches above although I do occasionally get VGA palette corruption.

Are you still require -vga cirrus or do the -vga none, x-vga=on cases
work now too?  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Sun, 2013-05-19 at 23:26 +0400, Maik Broemme wrote:
> Hi Knut,
> 
> Knut Omang <knuto@ifi.uio.no> wrote:
> > 
> > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > Hi all,
> > > > 
> > > > Perfect timing from my perspective, thanks Alex!
> > > > 
> > > > I spent the better part of the weekend testing your branches on a new system 
> > > > I just put together for this purpose, results below..
> > > > 
> > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > ...
> > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > 
> > > > System setup: 
> > > > 
> > > > - Fedora 18 on
> > > > - Gigabyte Z77X-UD5H motherboard
> > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > - 2 discrete graphics cards:
> > > > 
> > > > lspci | egrep 'VGA|Audio'
> > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 
> > > > Short summary:
> > > > 
> > > > - Once I got past a few time consuming obstacles explained below
> > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > >      on both discrete graphics cards 
> > > >      (though so far only one at at time and with some minor issues, see below)
> > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > 
> > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > support.
> > 
> > In my first tries, Windows were just using a generic
> > VGA driver, which still seems to work perfect with reboots and everything 
> > and in full screen resolution (1920x1200).
> > However after installing the Catalyst AMD driver stack, upon boot
> > Windows 7 now consequently get a BSOD from the graphics driver
> > with the message:
> > 
> > "Attempt to reset the display driver and recover from timeout failed"
> > - a picture of the BSOD screen attached.
> > 
> > I attach the corresponding vfio log where I added some timing code to
> > make it easier to see when the BSOD happens (with 2 seconds of silence
> > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > the log.
> > 
> > Similar behaviour both just after reboot/power cycle of the host and
> > subsequent VM boot attempts.
> > 
> > This is still with the HD7700 as passed through device, but after a
> > motherboard firmware upgrade (to F14) which did not seem to affect the
> > observed behaviour on Windows prior to Catalyst install or with Linux
> > guest, neither did it fix the bug in selecting primary devices as I 
> > was hoping for.
> > 
> > Let me know if you have ideas for further debugging this,
> > 
> 
> I had a similar problem a couple of days ago and posted it in this list.
> I got similar BSOD and tested already the following configurations:
> 
> 1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
> 2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
>    (with errors below)
> 3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
>    100% CPU due to no VGA init (no picture)
> 4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
>    BOSD on loading atikmpag.sys
> 5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine
> 
> However I've re-run the BSOD case already with the following branches
> from Alex:
> 
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> Also with latest seabios and it worked so far. No more BSOD and reboot
> of VM was also possible without suspend / resume the host between.

Good to hear.  It looks like you have the same motherboard as my AMD
test system.  An HD7850 in that system runs quite reliably with the
branches above although I do occasionally get VGA palette corruption.

Are you still require -vga cirrus or do the -vga none, x-vga=on cases
work now too?  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi Alex,

Alex Williamson <alex.williamson@redhat.com> wrote:
> On Sun, 2013-05-19 at 23:26 +0400, Maik Broemme wrote:
> > Hi Knut,
> > 
> > Knut Omang <knuto@ifi.uio.no> wrote:
> > > 
> > > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > > Hi all,
> > > > > 
> > > > > Perfect timing from my perspective, thanks Alex!
> > > > > 
> > > > > I spent the better part of the weekend testing your branches on a new system 
> > > > > I just put together for this purpose, results below..
> > > > > 
> > > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > > ...
> > > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > > 
> > > > > System setup: 
> > > > > 
> > > > > - Fedora 18 on
> > > > > - Gigabyte Z77X-UD5H motherboard
> > > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > > - 2 discrete graphics cards:
> > > > > 
> > > > > lspci | egrep 'VGA|Audio'
> > > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > > 
> > > > > Short summary:
> > > > > 
> > > > > - Once I got past a few time consuming obstacles explained below
> > > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > > >      on both discrete graphics cards 
> > > > >      (though so far only one at at time and with some minor issues, see below)
> > > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > > 
> > > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > > support.
> > > 
> > > In my first tries, Windows were just using a generic
> > > VGA driver, which still seems to work perfect with reboots and everything 
> > > and in full screen resolution (1920x1200).
> > > However after installing the Catalyst AMD driver stack, upon boot
> > > Windows 7 now consequently get a BSOD from the graphics driver
> > > with the message:
> > > 
> > > "Attempt to reset the display driver and recover from timeout failed"
> > > - a picture of the BSOD screen attached.
> > > 
> > > I attach the corresponding vfio log where I added some timing code to
> > > make it easier to see when the BSOD happens (with 2 seconds of silence
> > > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > > the log.
> > > 
> > > Similar behaviour both just after reboot/power cycle of the host and
> > > subsequent VM boot attempts.
> > > 
> > > This is still with the HD7700 as passed through device, but after a
> > > motherboard firmware upgrade (to F14) which did not seem to affect the
> > > observed behaviour on Windows prior to Catalyst install or with Linux
> > > guest, neither did it fix the bug in selecting primary devices as I 
> > > was hoping for.
> > > 
> > > Let me know if you have ideas for further debugging this,
> > > 
> > 
> > I had a similar problem a couple of days ago and posted it in this list.
> > I got similar BSOD and tested already the following configurations:
> > 
> > 1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
> > 2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
> >    (with errors below)
> > 3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
> >    100% CPU due to no VGA init (no picture)
> > 4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
> >    BOSD on loading atikmpag.sys
> > 5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine
> > 
> > However I've re-run the BSOD case already with the following branches
> > from Alex:
> > 
> > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > 
> > Also with latest seabios and it worked so far. No more BSOD and reboot
> > of VM was also possible without suspend / resume the host between.
> 
> Good to hear.  It looks like you have the same motherboard as my AMD
> test system.  An HD7850 in that system runs quite reliably with the
> branches above although I do occasionally get VGA palette corruption.
> 

Good to know. I'm using a Radeon HD7870 which works fine now. I have the
same VGA palette corruption occasionally but only until Catalyst driver
is loaded. So it happens sometimes during VGA init if Windows 7 boot
logo is shown with very strange colors and went away if Catalyst driver
is loaded.

> Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> work now too?  Thanks,
> 

No longer required, -vga none with x-vga=on work on your branches fine
now. Not sure if there was something more changed because with original
Fedora 3.9.2 kernel it still doesn't work.

> Alex
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi Alex,

Alex Williamson <alex.williamson@redhat.com> wrote:
> On Sun, 2013-05-19 at 23:26 +0400, Maik Broemme wrote:
> > Hi Knut,
> > 
> > Knut Omang <knuto@ifi.uio.no> wrote:
> > > 
> > > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > > Hi all,
> > > > > 
> > > > > Perfect timing from my perspective, thanks Alex!
> > > > > 
> > > > > I spent the better part of the weekend testing your branches on a new system 
> > > > > I just put together for this purpose, results below..
> > > > > 
> > > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > > ...
> > > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > > 
> > > > > System setup: 
> > > > > 
> > > > > - Fedora 18 on
> > > > > - Gigabyte Z77X-UD5H motherboard
> > > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > > - 2 discrete graphics cards:
> > > > > 
> > > > > lspci | egrep 'VGA|Audio'
> > > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > > 
> > > > > Short summary:
> > > > > 
> > > > > - Once I got past a few time consuming obstacles explained below
> > > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > > >      on both discrete graphics cards 
> > > > >      (though so far only one at at time and with some minor issues, see below)
> > > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > > 
> > > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > > support.
> > > 
> > > In my first tries, Windows were just using a generic
> > > VGA driver, which still seems to work perfect with reboots and everything 
> > > and in full screen resolution (1920x1200).
> > > However after installing the Catalyst AMD driver stack, upon boot
> > > Windows 7 now consequently get a BSOD from the graphics driver
> > > with the message:
> > > 
> > > "Attempt to reset the display driver and recover from timeout failed"
> > > - a picture of the BSOD screen attached.
> > > 
> > > I attach the corresponding vfio log where I added some timing code to
> > > make it easier to see when the BSOD happens (with 2 seconds of silence
> > > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > > the log.
> > > 
> > > Similar behaviour both just after reboot/power cycle of the host and
> > > subsequent VM boot attempts.
> > > 
> > > This is still with the HD7700 as passed through device, but after a
> > > motherboard firmware upgrade (to F14) which did not seem to affect the
> > > observed behaviour on Windows prior to Catalyst install or with Linux
> > > guest, neither did it fix the bug in selecting primary devices as I 
> > > was hoping for.
> > > 
> > > Let me know if you have ideas for further debugging this,
> > > 
> > 
> > I had a similar problem a couple of days ago and posted it in this list.
> > I got similar BSOD and tested already the following configurations:
> > 
> > 1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
> > 2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
> >    (with errors below)
> > 3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
> >    100% CPU due to no VGA init (no picture)
> > 4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
> >    BOSD on loading atikmpag.sys
> > 5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine
> > 
> > However I've re-run the BSOD case already with the following branches
> > from Alex:
> > 
> > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > 
> > Also with latest seabios and it worked so far. No more BSOD and reboot
> > of VM was also possible without suspend / resume the host between.
> 
> Good to hear.  It looks like you have the same motherboard as my AMD
> test system.  An HD7850 in that system runs quite reliably with the
> branches above although I do occasionally get VGA palette corruption.
> 

Good to know. I'm using a Radeon HD7870 which works fine now. I have the
same VGA palette corruption occasionally but only until Catalyst driver
is loaded. So it happens sometimes during VGA init if Windows 7 boot
logo is shown with very strange colors and went away if Catalyst driver
is loaded.

> Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> work now too?  Thanks,
> 

No longer required, -vga none with x-vga=on work on your branches fine
now. Not sure if there was something more changed because with original
Fedora 3.9.2 kernel it still doesn't work.

> Alex
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi Alex,

Maik Broemme <mbroemme@parallels.com> wrote:
> Hi Alex,
> 
> Alex Williamson <alex.williamson@redhat.com> wrote:
> > 
> > Good to hear.  It looks like you have the same motherboard as my AMD
> > test system.  An HD7850 in that system runs quite reliably with the
> > branches above although I do occasionally get VGA palette corruption.
> > 
> 
> Good to know. I'm using a Radeon HD7870 which works fine now. I have the
> same VGA palette corruption occasionally but only until Catalyst driver
> is loaded. So it happens sometimes during VGA init if Windows 7 boot
> logo is shown with very strange colors and went away if Catalyst driver
> is loaded.
> 
> > Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> > work now too?  Thanks,
> > 
> 
> No longer required, -vga none with x-vga=on work on your branches fine
> now. Not sure if there was something more changed because with original
> Fedora 3.9.2 kernel it still doesn't work.
> 

Alex, I have a strange issue now with either the 'vfio-vga-reset'
branches or with the stable 3.9.4 kernel. This is my 'lspci' output:

00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Pitcairn [Radeon HD 7800]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

The '01:00.0' is my primary device used for Linux and '02:00.0' my
secondary for QEMU. Two new different problems:

1) If the 'nvidia.ko' binary driver is loaded for the first card, QEMU
immediately get stuck after startup and hangs with:

1140  futex(0x7f0ad9b21300, FUTEX_WAIT_PRIVATE, 2, NULL

I have the complete strace output if needed. After that I can only
terminate qemu with 'kill -9' and if I start it again the following
Oops occurs:

[  655.684121] ------------[ cut here ]------------
[  655.684134] WARNING: at lib/list_debug.c:29 __list_add+0x77/0xd0()
[  655.684151] Hardware name: GA-990FXA-UD3
[  655.684271] list_add corruption. next->prev should be prev (ffffffff81ca3d98), but was           (null). (next=ffff88041bc3fe08).
[  655.684477] Modules linked in: vhost_net macvtap macvlan tun arc4 md4 nls_utf8 cifs dns_resolver fscache vfio_pci vfio_iommu_type1 vfio bridge stp llc ip6table_filter ip6_tables it87 hwmon_vid snd_hda_codec_hdmi nvidia(POF) acpi_cpufreq mperf kvm_amd snd_hda_codec_realtek kvm crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec microcode edac_core snd_hwdep fam15h_power snd_seq edac_mce_amd snd_seq_device k10temp r8169 sp5100_tco snd_pcm mii i2c_piix4 snd_page_alloc snd_timer i2c_core snd soundcore mxm_wmi firewire_ohci firewire_core crc_itu_t wmi
[  655.685451] Pid: 2097, comm: qemu-system-x86 Tainted: PF          O 3.9.4-200.fc18.x86_64 #1
[  655.685642] Call Trace:
[  655.685738]  [<ffffffff8105f125>] warn_slowpath_common+0x75/0xa0
[  655.685851]  [<ffffffff8105f206>] warn_slowpath_fmt+0x46/0x50
[  655.685955]  [<ffffffff81316ef7>] __list_add+0x77/0xd0
[  655.686058]  [<ffffffff8108392c>] add_wait_queue+0x3c/0x60
[  655.686162]  [<ffffffff813f241d>] vga_get+0xdd/0x190
[  655.686266]  [<ffffffff81093e40>] ? try_to_wake_up+0x2d0/0x2d0
[  655.686373]  [<ffffffffa01ac625>] vfio_pci_vga_rw+0xb5/0x230 [vfio_pci]
[  655.686481]  [<ffffffffa01aa279>] vfio_pci_rw+0x39/0x80 [vfio_pci]
[  655.686587]  [<ffffffffa01aa30c>] vfio_pci_read+0x1c/0x20 [vfio_pci]
[  655.686701]  [<ffffffffa01a40e3>] vfio_device_fops_read+0x23/0x30 [vfio]
[  655.686814]  [<ffffffff811a01b9>] vfs_read+0xa9/0x180
[  655.686915]  [<ffffffff811a05ba>] sys_pread64+0x9a/0xb0
[  655.687018]  [<ffffffff81669f59>] system_call_fastpath+0x16/0x1b
[  655.687123] ---[ end trace a68eabc3660237b1 ]---

This is always reproducible. I know it is the binary driver and maybe
nobody cares but it is widely used. :)

2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
as I start qemu all my SATA links get a hard reset and kernel freezes.
No SysRQs are working anymore and only reboot helps. If needed I can
look if I can get some dumps from this freeze because it writes nothing
more to the disks.

But it is getting even more strange. I was putting the secondary card
in another PCI slot and then it started to work with nouveau module
loaded and passthrough ATI card to QEMU. But this worked only until I
started X server with nouveau X driver. As soon as X is running and I
started QEMU it hanged again in FUTEX_WAIT_PRIVATE.

3) Without loading 'nvidia.ko' or 'nouveau.ko' modules it works out of
the box with several start/stop cycles. However I have no X in this
case. ;)

Any ideas? :)

> > Alex
> > 
> 
> --Maik
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Tue, 2013-05-28 at 03:40 +0200, Maik Broemme wrote:
> Hi Alex,
> 
> Maik Broemme <mbroemme@parallels.com> wrote:
> > Hi Alex,
> > 
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > 
> > > Good to hear.  It looks like you have the same motherboard as my AMD
> > > test system.  An HD7850 in that system runs quite reliably with the
> > > branches above although I do occasionally get VGA palette corruption.
> > > 
> > 
> > Good to know. I'm using a Radeon HD7870 which works fine now. I have the
> > same VGA palette corruption occasionally but only until Catalyst driver
> > is loaded. So it happens sometimes during VGA init if Windows 7 boot
> > logo is shown with very strange colors and went away if Catalyst driver
> > is loaded.
> > 
> > > Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> > > work now too?  Thanks,
> > > 
> > 
> > No longer required, -vga none with x-vga=on work on your branches fine
> > now. Not sure if there was something more changed because with original
> > Fedora 3.9.2 kernel it still doesn't work.
> > 
> 
> Alex, I have a strange issue now with either the 'vfio-vga-reset'
> branches or with the stable 3.9.4 kernel. This is my 'lspci' output:
> 
> 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> 01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1)
> 01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
> 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Pitcairn [Radeon HD 7800]
> 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> The '01:00.0' is my primary device used for Linux and '02:00.0' my
> secondary for QEMU. Two new different problems:
> 
> 1) If the 'nvidia.ko' binary driver is loaded for the first card, QEMU
> immediately get stuck after startup and hangs with:
> 
> 1140  futex(0x7f0ad9b21300, FUTEX_WAIT_PRIVATE, 2, NULL
> 
> I have the complete strace output if needed. After that I can only
> terminate qemu with 'kill -9' and if I start it again the following
> Oops occurs:
> 
> [  655.684121] ------------[ cut here ]------------
> [  655.684134] WARNING: at lib/list_debug.c:29 __list_add+0x77/0xd0()
> [  655.684151] Hardware name: GA-990FXA-UD3
> [  655.684271] list_add corruption. next->prev should be prev (ffffffff81ca3d98), but was           (null). (next=ffff88041bc3fe08).
> [  655.684477] Modules linked in: vhost_net macvtap macvlan tun arc4 md4 nls_utf8 cifs dns_resolver fscache vfio_pci vfio_iommu_type1 vfio bridge stp llc ip6table_filter ip6_tables it87 hwmon_vid snd_hda_codec_hdmi nvidia(POF) acpi_cpufreq mperf kvm_amd snd_hda_codec_realtek kvm crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec microcode edac_core snd_hwdep fam15h_power snd_seq edac_mce_amd snd_seq_device k10temp r8169 sp5100_tco snd_pcm mii i2c_piix4 snd_page_alloc snd_timer i2c_core snd soundcore mxm_wmi firewire_ohci firewire_core crc_itu_t wmi
> [  655.685451] Pid: 2097, comm: qemu-system-x86 Tainted: PF          O 3.9.4-200.fc18.x86_64 #1
> [  655.685642] Call Trace:
> [  655.685738]  [<ffffffff8105f125>] warn_slowpath_common+0x75/0xa0
> [  655.685851]  [<ffffffff8105f206>] warn_slowpath_fmt+0x46/0x50
> [  655.685955]  [<ffffffff81316ef7>] __list_add+0x77/0xd0
> [  655.686058]  [<ffffffff8108392c>] add_wait_queue+0x3c/0x60
> [  655.686162]  [<ffffffff813f241d>] vga_get+0xdd/0x190
> [  655.686266]  [<ffffffff81093e40>] ? try_to_wake_up+0x2d0/0x2d0
> [  655.686373]  [<ffffffffa01ac625>] vfio_pci_vga_rw+0xb5/0x230 [vfio_pci]
> [  655.686481]  [<ffffffffa01aa279>] vfio_pci_rw+0x39/0x80 [vfio_pci]
> [  655.686587]  [<ffffffffa01aa30c>] vfio_pci_read+0x1c/0x20 [vfio_pci]
> [  655.686701]  [<ffffffffa01a40e3>] vfio_device_fops_read+0x23/0x30 [vfio]
> [  655.686814]  [<ffffffff811a01b9>] vfs_read+0xa9/0x180
> [  655.686915]  [<ffffffff811a05ba>] sys_pread64+0x9a/0xb0
> [  655.687018]  [<ffffffff81669f59>] system_call_fastpath+0x16/0x1b
> [  655.687123] ---[ end trace a68eabc3660237b1 ]---
> 
> This is always reproducible. I know it is the binary driver and maybe
> nobody cares but it is widely used. :)

Hmm, so perhaps the first attempt called into the vga arbiter to get the
VGA resources and the hang is because it was never able to get them.
VFIO only uses the interruptible vga_get call, so you were able to kill
the process, but maybe the vga arbiter didn't cleanup so well.  There's
not much we can do if nvidia.ko never releases the VGA resources.  The
VGA arbiter could do a better job with list cleanup on interruption, but
it doesn't seem like it would help you run w/ nvidia.ko in the host.

> 2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
> as I start qemu all my SATA links get a hard reset and kernel freezes.
> No SysRQs are working anymore and only reboot helps. If needed I can
> look if I can get some dumps from this freeze because it writes nothing
> more to the disks.

This was after a host reboot, I hope?  If you're using the
vfio-vga-reset kernel branch then a secondary bus reset happens when the
guest is started.  I have seen cases on my GA-990FXA-UD3 where the bus
doesn't come back from reset (possibly due to queued I/O on the bus).
After reset we attempt to restore device config space which hangs with
the PCI config space access lock held.  This generally results in soft
lockups and a mostly unusable system.  Does this sound similar to what
you're seeing?  This is the main problem preventing me from trying to
push the PCI bus reset patches upstream.

> But it is getting even more strange. I was putting the secondary card
> in another PCI slot and then it started to work with nouveau module
> loaded and passthrough ATI card to QEMU. But this worked only until I
> started X server with nouveau X driver. As soon as X is running and I
> started QEMU it hanged again in FUTEX_WAIT_PRIVATE.
>
> 3) Without loading 'nvidia.ko' or 'nouveau.ko' modules it works out of
> the box with several start/stop cycles. However I have no X in this
> case. ;)
> 
> Any ideas? :)

I'd suspect that the nouveau X driver also isn't releasing the VGA
resources through the VGA arbiter.  That's pretty disappointing.  You
can check from userspace who own VGA via: sudo head /dev/vga_arbiter.
Without VGA arbiter we have no coordination of legacy VGA resources
between various drivers, but not all drivers support it (vgacon) and
those that do apparently don't attempt to be very fair.  We'll need to
look into fixing Xorg on the host if this is actually the problem.
Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi,

Maik Broemme <mbroemme@parallels.com> wrote:
> Hi Alex,
> 
> Maik Broemme <mbroemme@parallels.com> wrote:
> > Hi Alex,
> > 
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > 
> > > Good to hear.  It looks like you have the same motherboard as my AMD
> > > test system.  An HD7850 in that system runs quite reliably with the
> > > branches above although I do occasionally get VGA palette corruption.
> > > 
> > 
> > Good to know. I'm using a Radeon HD7870 which works fine now. I have the
> > same VGA palette corruption occasionally but only until Catalyst driver
> > is loaded. So it happens sometimes during VGA init if Windows 7 boot
> > logo is shown with very strange colors and went away if Catalyst driver
> > is loaded.
> > 
> > > Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> > > work now too?  Thanks,
> > > 
> > 
> > No longer required, -vga none with x-vga=on work on your branches fine
> > now. Not sure if there was something more changed because with original
> > Fedora 3.9.2 kernel it still doesn't work.
> > 
> 
> Alex, I have a strange issue now with either the 'vfio-vga-reset'
> branches or with the stable 3.9.4 kernel. This is my 'lspci' output:
> 
> 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> 01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1)
> 01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
> 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Pitcairn [Radeon HD 7800]
> 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> The '01:00.0' is my primary device used for Linux and '02:00.0' my
> secondary for QEMU. Two new different problems:
> 
> 1) If the 'nvidia.ko' binary driver is loaded for the first card, QEMU
> immediately get stuck after startup and hangs with:
> 
> 1140  futex(0x7f0ad9b21300, FUTEX_WAIT_PRIVATE, 2, NULL
> 
> I have the complete strace output if needed. After that I can only
> terminate qemu with 'kill -9' and if I start it again the following
> Oops occurs:
> 
> [  655.684121] ------------[ cut here ]------------
> [  655.684134] WARNING: at lib/list_debug.c:29 __list_add+0x77/0xd0()
> [  655.684151] Hardware name: GA-990FXA-UD3
> [  655.684271] list_add corruption. next->prev should be prev (ffffffff81ca3d98), but was           (null). (next=ffff88041bc3fe08).
> [  655.684477] Modules linked in: vhost_net macvtap macvlan tun arc4 md4 nls_utf8 cifs dns_resolver fscache vfio_pci vfio_iommu_type1 vfio bridge stp llc ip6table_filter ip6_tables it87 hwmon_vid snd_hda_codec_hdmi nvidia(POF) acpi_cpufreq mperf kvm_amd snd_hda_codec_realtek kvm crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec microcode edac_core snd_hwdep fam15h_power snd_seq edac_mce_amd snd_seq_device k10temp r8169 sp5100_tco snd_pcm mii i2c_piix4 snd_page_alloc snd_timer i2c_core snd soundcore mxm_wmi firewire_ohci firewire_core crc_itu_t wmi
> [  655.685451] Pid: 2097, comm: qemu-system-x86 Tainted: PF          O 3.9.4-200.fc18.x86_64 #1
> [  655.685642] Call Trace:
> [  655.685738]  [<ffffffff8105f125>] warn_slowpath_common+0x75/0xa0
> [  655.685851]  [<ffffffff8105f206>] warn_slowpath_fmt+0x46/0x50
> [  655.685955]  [<ffffffff81316ef7>] __list_add+0x77/0xd0
> [  655.686058]  [<ffffffff8108392c>] add_wait_queue+0x3c/0x60
> [  655.686162]  [<ffffffff813f241d>] vga_get+0xdd/0x190
> [  655.686266]  [<ffffffff81093e40>] ? try_to_wake_up+0x2d0/0x2d0
> [  655.686373]  [<ffffffffa01ac625>] vfio_pci_vga_rw+0xb5/0x230 [vfio_pci]
> [  655.686481]  [<ffffffffa01aa279>] vfio_pci_rw+0x39/0x80 [vfio_pci]
> [  655.686587]  [<ffffffffa01aa30c>] vfio_pci_read+0x1c/0x20 [vfio_pci]
> [  655.686701]  [<ffffffffa01a40e3>] vfio_device_fops_read+0x23/0x30 [vfio]
> [  655.686814]  [<ffffffff811a01b9>] vfs_read+0xa9/0x180
> [  655.686915]  [<ffffffff811a05ba>] sys_pread64+0x9a/0xb0
> [  655.687018]  [<ffffffff81669f59>] system_call_fastpath+0x16/0x1b
> [  655.687123] ---[ end trace a68eabc3660237b1 ]---
> 
> This is always reproducible. I know it is the binary driver and maybe
> nobody cares but it is widely used. :)

Here is the DEBUG_VFIO output:

vfio: vfio_initfn(0000:04:00.0) group 14
vfio: region_add 0 - afffffff [0x7f8698000000]
vfio: SKIPPING region_add fec00000 - fec00fff
vfio: SKIPPING region_add fed00000 - fed003ff
vfio: SKIPPING region_add fee00000 - feefffff
vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
vfio: Device 0000:04:00.0 region 0:
vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
vfio: Device 0000:04:00.0 region 1:
vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
vfio: Device 0000:04:00.0 region 2:
vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
vfio: Device 0000:04:00.0 region 3:
vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
vfio: Device 0000:04:00.0 region 4:
vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
vfio: Device 0000:04:00.0 region 5:
vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
vfio: Device 0000:04:00.0 ROM:
vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
vfio: Device 0000:04:00.0 config:
vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
vfio: vfio_load_rom(0000:04:00.0)
vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
vfio: 0000:04:00.0 PCI MSI CAP @0xa0
vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
vfio: vfio_enable_intx(0000:04:00.0)
vfio: vfio_initfn(0000:04:00.1) group 14
vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
vfio: Device 0000:04:00.1 region 0:
vfio:   size: 0x4000, offset: 0x0, flags: 0x7
vfio: Device 0000:04:00.1 region 1:
vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
vfio: Device 0000:04:00.1 region 2:
vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
vfio: Device 0000:04:00.1 region 3:
vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
vfio: Device 0000:04:00.1 region 4:
vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
vfio: Device 0000:04:00.1 region 5:
vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
vfio: Device 0000:04:00.1 ROM:
vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
vfio: Device 0000:04:00.1 config:
vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
vfio: 0000:04:00.1 PCI MSI CAP @0xa0
vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
vfio: vfio_enable_intx(0000:04:00.1)
vfio: region_del 0 - afffffff
vfio: region_add 0 - bffff [0x7f8698000000]
vfio: region_add c0000 - dffff [0x7f88aa200000]
vfio: region_add e0000 - fffff [0x7f88aa400000]
vfio: region_add 100000 - afffffff [0x7f8698100000]
vfio: vfio_pci_reset(0000:04:00.0)
vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
vfio: vfio_disable_intx(0000:04:00.0)
vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
vfio: vfio_enable_intx(0000:04:00.0)
vfio: vfio_pci_reset(0000:04:00.1)
vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
vfio: vfio_disable_intx(0000:04:00.1)
vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
vfio: vfio_enable_intx(0000:04:00.1)
vfio: region_del 0 - bffff
vfio: region_del c0000 - dffff
vfio: region_add 0 - c7fff [0x7f8698000000]
vfio: region_add c8000 - dffff [0x7f88aa208000]
vfio: region_del 0 - c7fff
vfio: region_del c8000 - dffff
vfio: region_add 0 - cffff [0x7f8698000000]
vfio: region_add d0000 - dffff [0x7f88aa210000]
vfio: region_del 0 - cffff
vfio: region_del d0000 - dffff
vfio: region_add 0 - d7fff [0x7f8698000000]
vfio: region_add d8000 - dffff [0x7f88aa218000]
vfio: region_del 0 - d7fff
vfio: region_del d8000 - dffff
vfio: region_add 0 - dffff [0x7f8698000000]
vfio: region_del 0 - dffff
vfio: region_del e0000 - fffff
vfio: region_add 0 - e7fff [0x7f8698000000]
vfio: region_add e8000 - fffff [0x7f88aa408000]
vfio: region_del 0 - e7fff
vfio: region_del e8000 - fffff
vfio: region_add 0 - effff [0x7f8698000000]
vfio: region_add f0000 - fffff [0x7f88aa410000]
vfio: region_del 0 - effff
vfio: region_del f0000 - fffff
vfio: region_del 100000 - afffffff
vfio: region_add 0 - afffffff [0x7f8698000000]
vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
vfio: SKIPPING region_add b0000000 - bfffffff
vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
vfio: SKIPPING region_add feb40000 - feb4002f
vfio: SKIPPING region_add feb40800 - feb40807
vfio: SKIPPING region_add feb41000 - feb4101f
vfio: SKIPPING region_add feb41800 - feb41807
vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
vfio: SKIPPING region_add feb42000 - feb42fff
vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
vfio: region_add e0000000 - efffffff [0x7f8688000000]
vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
vfio: SKIPPING region_add fea04000 - fea04fff
vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
vfio: region_del 0 - afffffff
vfio: region_add 0 - 9ffff [0x7f8698000000]
vfio: SKIPPING region_add a0000 - bffff
vfio: region_add c0000 - afffffff [0x7f86980c0000]
vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
vfio: region_del fea40000 - fea5ffff

Here is the strace output from this failure:

1110  ioctl(14, KVM_RUN, 0)             = 0
1110  pread(20,  <unfinished ...>
1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
1109  _exit(0)                          = ?
1109  +++ exited with 0 +++

>From reading the source 'hw/misc/vfio.c' it looks like the following
in 'vfio_vga_read' never finished:

    if (pread(vga->fd, &buf, size, offset) != size) {
        error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
                     __func__, region->offset + addr, size);
        return (uint64_t)-1;
    }

> 
> 2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
> as I start qemu all my SATA links get a hard reset and kernel freezes.
> No SysRQs are working anymore and only reboot helps. If needed I can
> look if I can get some dumps from this freeze because it writes nothing
> more to the disks.
> 
> But it is getting even more strange. I was putting the secondary card
> in another PCI slot and then it started to work with nouveau module
> loaded and passthrough ATI card to QEMU. But this worked only until I
> started X server with nouveau X driver. As soon as X is running and I
> started QEMU it hanged again in FUTEX_WAIT_PRIVATE.
> 
> 3) Without loading 'nvidia.ko' or 'nouveau.ko' modules it works out of
> the box with several start/stop cycles. However I have no X in this
> case. ;)
> 
> Any ideas? :)
> 
> > > Alex
> > > 
> > 
> > --Maik
> > 
> 
> --Maik
> 

--Maik

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Tue, 2013-05-28 at 20:45 +0200, Maik Broemme wrote:
> Hi,
> 
> Maik Broemme <mbroemme@parallels.com> wrote:
> > Hi Alex,
> > 
> > Maik Broemme <mbroemme@parallels.com> wrote:
> > > Hi Alex,
> > > 
> > > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > > 
> > > > Good to hear.  It looks like you have the same motherboard as my AMD
> > > > test system.  An HD7850 in that system runs quite reliably with the
> > > > branches above although I do occasionally get VGA palette corruption.
> > > > 
> > > 
> > > Good to know. I'm using a Radeon HD7870 which works fine now. I have the
> > > same VGA palette corruption occasionally but only until Catalyst driver
> > > is loaded. So it happens sometimes during VGA init if Windows 7 boot
> > > logo is shown with very strange colors and went away if Catalyst driver
> > > is loaded.
> > > 
> > > > Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> > > > work now too?  Thanks,
> > > > 
> > > 
> > > No longer required, -vga none with x-vga=on work on your branches fine
> > > now. Not sure if there was something more changed because with original
> > > Fedora 3.9.2 kernel it still doesn't work.
> > > 
> > 
> > Alex, I have a strange issue now with either the 'vfio-vga-reset'
> > branches or with the stable 3.9.4 kernel. This is my 'lspci' output:
> > 
> > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > 01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1)
> > 01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
> > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Pitcairn [Radeon HD 7800]
> > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > 
> > The '01:00.0' is my primary device used for Linux and '02:00.0' my
> > secondary for QEMU. Two new different problems:
> > 
> > 1) If the 'nvidia.ko' binary driver is loaded for the first card, QEMU
> > immediately get stuck after startup and hangs with:
> > 
> > 1140  futex(0x7f0ad9b21300, FUTEX_WAIT_PRIVATE, 2, NULL
> > 
> > I have the complete strace output if needed. After that I can only
> > terminate qemu with 'kill -9' and if I start it again the following
> > Oops occurs:
> > 
> > [  655.684121] ------------[ cut here ]------------
> > [  655.684134] WARNING: at lib/list_debug.c:29 __list_add+0x77/0xd0()
> > [  655.684151] Hardware name: GA-990FXA-UD3
> > [  655.684271] list_add corruption. next->prev should be prev (ffffffff81ca3d98), but was           (null). (next=ffff88041bc3fe08).
> > [  655.684477] Modules linked in: vhost_net macvtap macvlan tun arc4 md4 nls_utf8 cifs dns_resolver fscache vfio_pci vfio_iommu_type1 vfio bridge stp llc ip6table_filter ip6_tables it87 hwmon_vid snd_hda_codec_hdmi nvidia(POF) acpi_cpufreq mperf kvm_amd snd_hda_codec_realtek kvm crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec microcode edac_core snd_hwdep fam15h_power snd_seq edac_mce_amd snd_seq_device k10temp r8169 sp5100_tco snd_pcm mii i2c_piix4 snd_page_alloc snd_timer i2c_core snd soundcore mxm_wmi firewire_ohci firewire_core crc_itu_t wmi
> > [  655.685451] Pid: 2097, comm: qemu-system-x86 Tainted: PF          O 3.9.4-200.fc18.x86_64 #1
> > [  655.685642] Call Trace:
> > [  655.685738]  [<ffffffff8105f125>] warn_slowpath_common+0x75/0xa0
> > [  655.685851]  [<ffffffff8105f206>] warn_slowpath_fmt+0x46/0x50
> > [  655.685955]  [<ffffffff81316ef7>] __list_add+0x77/0xd0
> > [  655.686058]  [<ffffffff8108392c>] add_wait_queue+0x3c/0x60
> > [  655.686162]  [<ffffffff813f241d>] vga_get+0xdd/0x190
> > [  655.686266]  [<ffffffff81093e40>] ? try_to_wake_up+0x2d0/0x2d0
> > [  655.686373]  [<ffffffffa01ac625>] vfio_pci_vga_rw+0xb5/0x230 [vfio_pci]
> > [  655.686481]  [<ffffffffa01aa279>] vfio_pci_rw+0x39/0x80 [vfio_pci]
> > [  655.686587]  [<ffffffffa01aa30c>] vfio_pci_read+0x1c/0x20 [vfio_pci]
> > [  655.686701]  [<ffffffffa01a40e3>] vfio_device_fops_read+0x23/0x30 [vfio]
> > [  655.686814]  [<ffffffff811a01b9>] vfs_read+0xa9/0x180
> > [  655.686915]  [<ffffffff811a05ba>] sys_pread64+0x9a/0xb0
> > [  655.687018]  [<ffffffff81669f59>] system_call_fastpath+0x16/0x1b
> > [  655.687123] ---[ end trace a68eabc3660237b1 ]---
> > 
> > This is always reproducible. I know it is the binary driver and maybe
> > nobody cares but it is widely used. :)
> 
> Here is the DEBUG_VFIO output:
> 
> vfio: vfio_initfn(0000:04:00.0) group 14
> vfio: region_add 0 - afffffff [0x7f8698000000]
> vfio: SKIPPING region_add fec00000 - fec00fff
> vfio: SKIPPING region_add fed00000 - fed003ff
> vfio: SKIPPING region_add fee00000 - feefffff
> vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> vfio: Device 0000:04:00.0 region 0:
> vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> vfio: Device 0000:04:00.0 region 1:
> vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> vfio: Device 0000:04:00.0 region 2:
> vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> vfio: Device 0000:04:00.0 region 3:
> vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> vfio: Device 0000:04:00.0 region 4:
> vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> vfio: Device 0000:04:00.0 region 5:
> vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> vfio: Device 0000:04:00.0 ROM:
> vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> vfio: Device 0000:04:00.0 config:
> vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> vfio: vfio_load_rom(0000:04:00.0)
> vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> vfio: vfio_enable_intx(0000:04:00.0)
> vfio: vfio_initfn(0000:04:00.1) group 14
> vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> vfio: Device 0000:04:00.1 region 0:
> vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> vfio: Device 0000:04:00.1 region 1:
> vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> vfio: Device 0000:04:00.1 region 2:
> vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> vfio: Device 0000:04:00.1 region 3:
> vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> vfio: Device 0000:04:00.1 region 4:
> vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> vfio: Device 0000:04:00.1 region 5:
> vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> vfio: Device 0000:04:00.1 ROM:
> vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> vfio: Device 0000:04:00.1 config:
> vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> vfio: vfio_enable_intx(0000:04:00.1)
> vfio: region_del 0 - afffffff
> vfio: region_add 0 - bffff [0x7f8698000000]
> vfio: region_add c0000 - dffff [0x7f88aa200000]
> vfio: region_add e0000 - fffff [0x7f88aa400000]
> vfio: region_add 100000 - afffffff [0x7f8698100000]
> vfio: vfio_pci_reset(0000:04:00.0)
> vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> vfio: vfio_disable_intx(0000:04:00.0)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> vfio: vfio_enable_intx(0000:04:00.0)
> vfio: vfio_pci_reset(0000:04:00.1)
> vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> vfio: vfio_disable_intx(0000:04:00.1)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> vfio: vfio_enable_intx(0000:04:00.1)
> vfio: region_del 0 - bffff
> vfio: region_del c0000 - dffff
> vfio: region_add 0 - c7fff [0x7f8698000000]
> vfio: region_add c8000 - dffff [0x7f88aa208000]
> vfio: region_del 0 - c7fff
> vfio: region_del c8000 - dffff
> vfio: region_add 0 - cffff [0x7f8698000000]
> vfio: region_add d0000 - dffff [0x7f88aa210000]
> vfio: region_del 0 - cffff
> vfio: region_del d0000 - dffff
> vfio: region_add 0 - d7fff [0x7f8698000000]
> vfio: region_add d8000 - dffff [0x7f88aa218000]
> vfio: region_del 0 - d7fff
> vfio: region_del d8000 - dffff
> vfio: region_add 0 - dffff [0x7f8698000000]
> vfio: region_del 0 - dffff
> vfio: region_del e0000 - fffff
> vfio: region_add 0 - e7fff [0x7f8698000000]
> vfio: region_add e8000 - fffff [0x7f88aa408000]
> vfio: region_del 0 - e7fff
> vfio: region_del e8000 - fffff
> vfio: region_add 0 - effff [0x7f8698000000]
> vfio: region_add f0000 - fffff [0x7f88aa410000]
> vfio: region_del 0 - effff
> vfio: region_del f0000 - fffff
> vfio: region_del 100000 - afffffff
> vfio: region_add 0 - afffffff [0x7f8698000000]
> vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> vfio: SKIPPING region_add b0000000 - bfffffff
> vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> vfio: SKIPPING region_add feb40000 - feb4002f
> vfio: SKIPPING region_add feb40800 - feb40807
> vfio: SKIPPING region_add feb41000 - feb4101f
> vfio: SKIPPING region_add feb41800 - feb41807
> vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> vfio: SKIPPING region_add feb42000 - feb42fff
> vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> vfio: region_add e0000000 - efffffff [0x7f8688000000]
> vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> vfio: SKIPPING region_add fea04000 - fea04fff
> vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> vfio: region_del 0 - afffffff
> vfio: region_add 0 - 9ffff [0x7f8698000000]
> vfio: SKIPPING region_add a0000 - bffff
> vfio: region_add c0000 - afffffff [0x7f86980c0000]
> vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000

Here the option ROM was sized

> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)

Then enabled

> vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]

Adding this memory region

> vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> vfio: region_del fea40000 - fea5ffff

Then disabled, removing the memory region.  Presumably between the
enable and disable the contents were read and copied to 0xc0000, which
is where the VGA BIOS is shadowed.

> Here is the strace output from this failure:
> 
> 1110  ioctl(14, KVM_RUN, 0)             = 0
> 1110  pread(20,  <unfinished ...>
> 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
> 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> 1109  _exit(0)                          = ?
> 1109  +++ exited with 0 +++
> 
> From reading the source 'hw/misc/vfio.c' it looks like the following
> in 'vfio_vga_read' never finished:
> 
>     if (pread(vga->fd, &buf, size, offset) != size) {
>         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
>                      __func__, region->offset + addr, size);
>         return (uint64_t)-1;
>     }

I agree.  Every VGA access requires us to lock the VGA resources on the
device, so if we can't get the lock, we stop making progress.  I took a
look at Xorg last night and it seems like it should be taking and
releasing the VGA arbiter lock in a way that would be compatible with
our use.  That's in the xserver, not the actual display hardware driver,
and it wraps access functions in the arbiter support, so should be
transparent to the drivers.  So for nouveau, it seems like it should
work.  For nvidia, we don't really know, it could be locking the device
from the kernel module.

You could instrument vga_get, vga_tryget, and vga_put to figure out
what's happening.  It might be enough to look at /dev/vga_arbiter at
each step in the sequence to reproduce (sudo head
--lines=1 /dev/vga_arbiter).  Thanks,

Alex

> > 
> > 2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
> > as I start qemu all my SATA links get a hard reset and kernel freezes.
> > No SysRQs are working anymore and only reboot helps. If needed I can
> > look if I can get some dumps from this freeze because it writes nothing
> > more to the disks.
> > 
> > But it is getting even more strange. I was putting the secondary card
> > in another PCI slot and then it started to work with nouveau module
> > loaded and passthrough ATI card to QEMU. But this worked only until I
> > started X server with nouveau X driver. As soon as X is running and I
> > started QEMU it hanged again in FUTEX_WAIT_PRIVATE.
> > 
> > 3) Without loading 'nvidia.ko' or 'nouveau.ko' modules it works out of
> > the box with several start/stop cycles. However I have no X in this
> > case. ;)
> > 
> > Any ideas? :)
> > 
> > > > Alex
> > > > 
> > > 
> > > --Maik
> > > 
> > 
> > --Maik
> > 
> 
> --Maik

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

Hi,

Alex Williamson <alex.williamson@redhat.com> wrote:
> On Tue, 2013-05-28 at 20:45 +0200, Maik Broemme wrote:
> > Hi,
> > 
> > Maik Broemme <mbroemme@parallels.com> wrote:
> > > Hi Alex,
> > > 
> > > Maik Broemme <mbroemme@parallels.com> wrote:
> > > > Hi Alex,
> > > > 
> > > > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > > > 
> > > > > Good to hear.  It looks like you have the same motherboard as my AMD
> > > > > test system.  An HD7850 in that system runs quite reliably with the
> > > > > branches above although I do occasionally get VGA palette corruption.
> > > > > 
> > > > 
> > > > Good to know. I'm using a Radeon HD7870 which works fine now. I have the
> > > > same VGA palette corruption occasionally but only until Catalyst driver
> > > > is loaded. So it happens sometimes during VGA init if Windows 7 boot
> > > > logo is shown with very strange colors and went away if Catalyst driver
> > > > is loaded.
> > > > 
> > > > > Are you still require -vga cirrus or do the -vga none, x-vga=on cases
> > > > > work now too?  Thanks,
> > > > > 
> > > > 
> > > > No longer required, -vga none with x-vga=on work on your branches fine
> > > > now. Not sure if there was something more changed because with original
> > > > Fedora 3.9.2 kernel it still doesn't work.
> > > > 
> > > 
> > > Alex, I have a strange issue now with either the 'vfio-vga-reset'
> > > branches or with the stable 3.9.4 kernel. This is my 'lspci' output:
> > > 
> > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > > 01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1)
> > > 01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
> > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Pitcairn [Radeon HD 7800]
> > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > 
> > > The '01:00.0' is my primary device used for Linux and '02:00.0' my
> > > secondary for QEMU. Two new different problems:
> > > 
> > > 1) If the 'nvidia.ko' binary driver is loaded for the first card, QEMU
> > > immediately get stuck after startup and hangs with:
> > > 
> > > 1140  futex(0x7f0ad9b21300, FUTEX_WAIT_PRIVATE, 2, NULL
> > > 
> > > I have the complete strace output if needed. After that I can only
> > > terminate qemu with 'kill -9' and if I start it again the following
> > > Oops occurs:
> > > 
> > > [  655.684121] ------------[ cut here ]------------
> > > [  655.684134] WARNING: at lib/list_debug.c:29 __list_add+0x77/0xd0()
> > > [  655.684151] Hardware name: GA-990FXA-UD3
> > > [  655.684271] list_add corruption. next->prev should be prev (ffffffff81ca3d98), but was           (null). (next=ffff88041bc3fe08).
> > > [  655.684477] Modules linked in: vhost_net macvtap macvlan tun arc4 md4 nls_utf8 cifs dns_resolver fscache vfio_pci vfio_iommu_type1 vfio bridge stp llc ip6table_filter ip6_tables it87 hwmon_vid snd_hda_codec_hdmi nvidia(POF) acpi_cpufreq mperf kvm_amd snd_hda_codec_realtek kvm crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec microcode edac_core snd_hwdep fam15h_power snd_seq edac_mce_amd snd_seq_device k10temp r8169 sp5100_tco snd_pcm mii i2c_piix4 snd_page_alloc snd_timer i2c_core snd soundcore mxm_wmi firewire_ohci firewire_core crc_itu_t wmi
> > > [  655.685451] Pid: 2097, comm: qemu-system-x86 Tainted: PF          O 3.9.4-200.fc18.x86_64 #1
> > > [  655.685642] Call Trace:
> > > [  655.685738]  [<ffffffff8105f125>] warn_slowpath_common+0x75/0xa0
> > > [  655.685851]  [<ffffffff8105f206>] warn_slowpath_fmt+0x46/0x50
> > > [  655.685955]  [<ffffffff81316ef7>] __list_add+0x77/0xd0
> > > [  655.686058]  [<ffffffff8108392c>] add_wait_queue+0x3c/0x60
> > > [  655.686162]  [<ffffffff813f241d>] vga_get+0xdd/0x190
> > > [  655.686266]  [<ffffffff81093e40>] ? try_to_wake_up+0x2d0/0x2d0
> > > [  655.686373]  [<ffffffffa01ac625>] vfio_pci_vga_rw+0xb5/0x230 [vfio_pci]
> > > [  655.686481]  [<ffffffffa01aa279>] vfio_pci_rw+0x39/0x80 [vfio_pci]
> > > [  655.686587]  [<ffffffffa01aa30c>] vfio_pci_read+0x1c/0x20 [vfio_pci]
> > > [  655.686701]  [<ffffffffa01a40e3>] vfio_device_fops_read+0x23/0x30 [vfio]
> > > [  655.686814]  [<ffffffff811a01b9>] vfs_read+0xa9/0x180
> > > [  655.686915]  [<ffffffff811a05ba>] sys_pread64+0x9a/0xb0
> > > [  655.687018]  [<ffffffff81669f59>] system_call_fastpath+0x16/0x1b
> > > [  655.687123] ---[ end trace a68eabc3660237b1 ]---
> > > 
> > > This is always reproducible. I know it is the binary driver and maybe
> > > nobody cares but it is widely used. :)
> > 
> > Here is the DEBUG_VFIO output:
> > 
> > vfio: vfio_initfn(0000:04:00.0) group 14
> > vfio: region_add 0 - afffffff [0x7f8698000000]
> > vfio: SKIPPING region_add fec00000 - fec00fff
> > vfio: SKIPPING region_add fed00000 - fed003ff
> > vfio: SKIPPING region_add fee00000 - feefffff
> > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > vfio: Device 0000:04:00.0 region 0:
> > vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> > vfio: Device 0000:04:00.0 region 1:
> > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > vfio: Device 0000:04:00.0 region 2:
> > vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> > vfio: Device 0000:04:00.0 region 3:
> > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > vfio: Device 0000:04:00.0 region 4:
> > vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> > vfio: Device 0000:04:00.0 region 5:
> > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > vfio: Device 0000:04:00.0 ROM:
> > vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> > vfio: Device 0000:04:00.0 config:
> > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > vfio: vfio_load_rom(0000:04:00.0)
> > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > vfio: vfio_enable_intx(0000:04:00.0)
> > vfio: vfio_initfn(0000:04:00.1) group 14
> > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > vfio: Device 0000:04:00.1 region 0:
> > vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> > vfio: Device 0000:04:00.1 region 1:
> > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 region 2:
> > vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 region 3:
> > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 region 4:
> > vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 region 5:
> > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 ROM:
> > vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> > vfio: Device 0000:04:00.1 config:
> > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > vfio: vfio_enable_intx(0000:04:00.1)
> > vfio: region_del 0 - afffffff
> > vfio: region_add 0 - bffff [0x7f8698000000]
> > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > vfio: vfio_pci_reset(0000:04:00.0)
> > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > vfio: vfio_disable_intx(0000:04:00.0)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > vfio: vfio_enable_intx(0000:04:00.0)
> > vfio: vfio_pci_reset(0000:04:00.1)
> > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > vfio: vfio_disable_intx(0000:04:00.1)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > vfio: vfio_enable_intx(0000:04:00.1)
> > vfio: region_del 0 - bffff
> > vfio: region_del c0000 - dffff
> > vfio: region_add 0 - c7fff [0x7f8698000000]
> > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > vfio: region_del 0 - c7fff
> > vfio: region_del c8000 - dffff
> > vfio: region_add 0 - cffff [0x7f8698000000]
> > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > vfio: region_del 0 - cffff
> > vfio: region_del d0000 - dffff
> > vfio: region_add 0 - d7fff [0x7f8698000000]
> > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > vfio: region_del 0 - d7fff
> > vfio: region_del d8000 - dffff
> > vfio: region_add 0 - dffff [0x7f8698000000]
> > vfio: region_del 0 - dffff
> > vfio: region_del e0000 - fffff
> > vfio: region_add 0 - e7fff [0x7f8698000000]
> > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > vfio: region_del 0 - e7fff
> > vfio: region_del e8000 - fffff
> > vfio: region_add 0 - effff [0x7f8698000000]
> > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > vfio: region_del 0 - effff
> > vfio: region_del f0000 - fffff
> > vfio: region_del 100000 - afffffff
> > vfio: region_add 0 - afffffff [0x7f8698000000]
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > vfio: SKIPPING region_add b0000000 - bfffffff
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > vfio: SKIPPING region_add feb40000 - feb4002f
> > vfio: SKIPPING region_add feb40800 - feb40807
> > vfio: SKIPPING region_add feb41000 - feb4101f
> > vfio: SKIPPING region_add feb41800 - feb41807
> > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > vfio: SKIPPING region_add feb42000 - feb42fff
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > vfio: SKIPPING region_add fea04000 - fea04fff
> > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > vfio: region_del 0 - afffffff
> > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > vfio: SKIPPING region_add a0000 - bffff
> > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> 
> Here the option ROM was sized
> 
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> 
> Then enabled
> 
> > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> 
> Adding this memory region
> 
> > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > vfio: region_del fea40000 - fea5ffff
> 
> Then disabled, removing the memory region.  Presumably between the
> enable and disable the contents were read and copied to 0xc0000, which
> is where the VGA BIOS is shadowed.
> 
> > Here is the strace output from this failure:
> > 
> > 1110  ioctl(14, KVM_RUN, 0)             = 0
> > 1110  pread(20,  <unfinished ...>
> > 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> > 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
> > 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > 1109  _exit(0)                          = ?
> > 1109  +++ exited with 0 +++
> > 
> > From reading the source 'hw/misc/vfio.c' it looks like the following
> > in 'vfio_vga_read' never finished:
> > 
> >     if (pread(vga->fd, &buf, size, offset) != size) {
> >         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> >                      __func__, region->offset + addr, size);
> >         return (uint64_t)-1;
> >     }
> 
> I agree.  Every VGA access requires us to lock the VGA resources on the
> device, so if we can't get the lock, we stop making progress.  I took a
> look at Xorg last night and it seems like it should be taking and
> releasing the VGA arbiter lock in a way that would be compatible with
> our use.  That's in the xserver, not the actual display hardware driver,
> and it wraps access functions in the arbiter support, so should be
> transparent to the drivers.  So for nouveau, it seems like it should
> work.  For nvidia, we don't really know, it could be locking the device
> from the kernel module.
> 
> You could instrument vga_get, vga_tryget, and vga_put to figure out
> what's happening.  It might be enough to look at /dev/vga_arbiter at
> each step in the sequence to reproduce (sudo head
> --lines=1 /dev/vga_arbiter).  Thanks,
> 

I've played a bit more with it and there are some differences in
behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
binary driver I see the following:

[   18.628676] [drm] Initialized drm 1.1.0 20060810
[   18.668038] nvidia: module license 'NVIDIA' taints kernel.
[   18.668107] Disabling lock debugging due to kernel taint
[   18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[   18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
[   18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
[   18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  319.23  Thu May 16 19:36:02 PDT 2013

root@homer:~# head --lines=1 /dev/vga_arbiter
count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
root@homer:~# modprobe nvidia
root@homer:~# head --lines=1 /dev/vga_arbiter
count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
root@homer:~# /usr/local/bin/qemu-system-x86_64 \
        -L /usr/local/share/qemu \
        -L /usr/local/share/qemu \
        -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
        -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
        -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
        -device virtio-blk-pci,drive=drive0,ioeventfd=on \
        -device ioh3420,id=pcie0,multifunction=on \
        -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
        -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
root@homer:~# head --lines=1 /dev/vga_arbiter
count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)

It looks like nvidia binary driver locks some resources. :( Whereas
with nouveau driver the VGA arbitration starts first if I start the
VM with VGA passthrough:

[  178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[  178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
[  178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
[  181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[  181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0

root@homer:~# head --lines=1 /dev/vga_arbiter
count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
root@homer:~# /usr/local/bin/qemu-system-x86_64 \
        -L /usr/local/share/qemu \
        -L /usr/local/share/qemu \
        -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
        -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
        -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
        -device virtio-blk-pci,drive=drive0,ioeventfd=on \
        -device ioh3420,id=pcie0,multifunction=on \
        -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
        -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
root@homer:~# head --lines=1 /dev/vga_arbiter
count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)

> Alex
> 
> > > 
> > > 2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
> > > as I start qemu all my SATA links get a hard reset and kernel freezes.
> > > No SysRQs are working anymore and only reboot helps. If needed I can
> > > look if I can get some dumps from this freeze because it writes nothing
> > > more to the disks.
> > > 
> > > But it is getting even more strange. I was putting the secondary card
> > > in another PCI slot and then it started to work with nouveau module
> > > loaded and passthrough ATI card to QEMU. But this worked only until I
> > > started X server with nouveau X driver. As soon as X is running and I
> > > started QEMU it hanged again in FUTEX_WAIT_PRIVATE.
> > > 
> > > 3) Without loading 'nvidia.ko' or 'nouveau.ko' modules it works out of
> > > the box with several start/stop cycles. However I have no X in this
> > > case. ;)
> > > 
> > > Any ideas? :)
> > > 
> > > > > Alex
> > > > > 
> > > > 
> > > > --Maik
> > > > 
> > > 
> > > --Maik
> > > 
> > 
> > --Maik
> 
> 
> 

Kind regards
Maik

Parallels, Inc.
http://www.parallels.com/

Re: VFIO VGA test branches

[Maik Broemme]

[-- Attachment #1: Type: text/plain, Size: 19584 bytes --]

Hi,

Maik Broemme <mbroemme@parallels.com> wrote:
> > > 
> > > Here is the DEBUG_VFIO output:
> > > 
> > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > vfio: SKIPPING region_add fee00000 - feefffff
> > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.0 region 0:
> > > vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 1:
> > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 2:
> > > vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 3:
> > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 4:
> > > vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> > > vfio: Device 0000:04:00.0 region 5:
> > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 ROM:
> > > vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > vfio: Device 0000:04:00.0 config:
> > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: vfio_load_rom(0000:04:00.0)
> > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.1 region 0:
> > > vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.1 region 1:
> > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 2:
> > > vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 3:
> > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 4:
> > > vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 5:
> > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 ROM:
> > > vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 config:
> > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > vfio: vfio_pci_reset(0000:04:00.0)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.0)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_pci_reset(0000:04:00.1)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - bffff
> > > vfio: region_del c0000 - dffff
> > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > vfio: region_del 0 - c7fff
> > > vfio: region_del c8000 - dffff
> > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > vfio: region_del 0 - cffff
> > > vfio: region_del d0000 - dffff
> > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > vfio: region_del 0 - d7fff
> > > vfio: region_del d8000 - dffff
> > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > vfio: region_del 0 - dffff
> > > vfio: region_del e0000 - fffff
> > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > vfio: region_del 0 - e7fff
> > > vfio: region_del e8000 - fffff
> > > vfio: region_add 0 - effff [0x7f8698000000]
> > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > vfio: region_del 0 - effff
> > > vfio: region_del f0000 - fffff
> > > vfio: region_del 100000 - afffffff
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > vfio: SKIPPING region_add feb40800 - feb40807
> > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > vfio: SKIPPING region_add feb41800 - feb41807
> > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > vfio: SKIPPING region_add a0000 - bffff
> > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > 
> > Here the option ROM was sized
> > 
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> > 
> > Then enabled
> > 
> > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> > 
> > Adding this memory region
> > 
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: region_del fea40000 - fea5ffff
> > 
> > Then disabled, removing the memory region.  Presumably between the
> > enable and disable the contents were read and copied to 0xc0000, which
> > is where the VGA BIOS is shadowed.
> > 
> > > Here is the strace output from this failure:
> > > 
> > > 1110  ioctl(14, KVM_RUN, 0)             = 0
> > > 1110  pread(20,  <unfinished ...>
> > > 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> > > 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
> > > 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > 1109  _exit(0)                          = ?
> > > 1109  +++ exited with 0 +++
> > > 
> > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > in 'vfio_vga_read' never finished:
> > > 
> > >     if (pread(vga->fd, &buf, size, offset) != size) {
> > >         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > >                      __func__, region->offset + addr, size);
> > >         return (uint64_t)-1;
> > >     }
> > 
> > I agree.  Every VGA access requires us to lock the VGA resources on the
> > device, so if we can't get the lock, we stop making progress.  I took a
> > look at Xorg last night and it seems like it should be taking and
> > releasing the VGA arbiter lock in a way that would be compatible with
> > our use.  That's in the xserver, not the actual display hardware driver,
> > and it wraps access functions in the arbiter support, so should be
> > transparent to the drivers.  So for nouveau, it seems like it should
> > work.  For nvidia, we don't really know, it could be locking the device
> > from the kernel module.
> > 
> > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > what's happening.  It might be enough to look at /dev/vga_arbiter at
> > each step in the sequence to reproduce (sudo head
> > --lines=1 /dev/vga_arbiter).  Thanks,
> > 
> 
> I've played a bit more with it and there are some differences in
> behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> binary driver I see the following:
> 
> [   18.628676] [drm] Initialized drm 1.1.0 20060810
> [   18.668038] nvidia: module license 'NVIDIA' taints kernel.
> [   18.668107] Disabling lock debugging due to kernel taint
> [   18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [   18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> [   18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
> [   18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  319.23  Thu May 16 19:36:02 PDT 2013
> 
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# modprobe nvidia
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
>         -L /usr/local/share/qemu \
>         -L /usr/local/share/qemu \
>         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
>         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
>         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
>         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
>         -device ioh3420,id=pcie0,multifunction=on \
>         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
>         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> 
> It looks like nvidia binary driver locks some resources. :( Whereas
> with nouveau driver the VGA arbitration starts first if I start the
> VM with VGA passthrough:
> 
> [  178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> [  178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
> [  178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
> [  181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [  181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> 
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
>         -L /usr/local/share/qemu \
>         -L /usr/local/share/qemu \
>         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
>         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
>         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
>         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
>         -device ioh3420,id=pcie0,multifunction=on \
>         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
>         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
> 

Alex you pointed me in the right direction, many thanks! I got it
working with 'nvidia' binary driver now, but I had to patch it. I've
attached it here just for reference if others want to try the same
and I don't know if it is the proper way. After some testing (Xorg,
DRI, VDPAU, switching between X and text-console) it looks like a stable
workaround for me now until NVIDIA will fix their driver.

--Maik

[-- Attachment #2: NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix.patch --]
[-- Type: text/plain, Size: 1355 bytes --]

diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h
--- NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h	2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h	2013-05-29 18:09:42.382925622 +0200
@@ -151,9 +151,9 @@
 #error "struct file_operations compile test likely failed!"
 #endif
 
-#if defined(CONFIG_VGA_ARB)
-#include <linux/vgaarb.h>
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#include <linux/vgaarb.h>
+//#endif
 
 #if defined(NV_VM_INSERT_PAGE_PRESENT)
 #include <linux/pagemap.h>
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv.c NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c
--- NVIDIA-Linux-x86_64-319.23/kernel/nv.c	2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c	2013-05-29 18:10:01.494277314 +0200
@@ -2914,12 +2914,12 @@
 
     pci_set_master(dev);
 
-#if defined(CONFIG_VGA_ARB)
-#if defined(VGA_DEFAULT_DEVICE)
-    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
-#endif
-    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#if defined(VGA_DEFAULT_DEVICE)
+//    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
+//#endif
+//    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
+//#endif
 
     if (NV_IS_GVI_DEVICE(nv))
     {

Re: [Qemu-devel] VFIO VGA test branches

[Maik Broemme]

[-- Attachment #1: Type: text/plain, Size: 19584 bytes --]

Hi,

Maik Broemme <mbroemme@parallels.com> wrote:
> > > 
> > > Here is the DEBUG_VFIO output:
> > > 
> > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > vfio: SKIPPING region_add fee00000 - feefffff
> > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.0 region 0:
> > > vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 1:
> > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 2:
> > > vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 3:
> > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 4:
> > > vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> > > vfio: Device 0000:04:00.0 region 5:
> > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 ROM:
> > > vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > vfio: Device 0000:04:00.0 config:
> > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: vfio_load_rom(0000:04:00.0)
> > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.1 region 0:
> > > vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.1 region 1:
> > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 2:
> > > vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 3:
> > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 4:
> > > vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 5:
> > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 ROM:
> > > vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 config:
> > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > vfio: vfio_pci_reset(0000:04:00.0)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.0)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_pci_reset(0000:04:00.1)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - bffff
> > > vfio: region_del c0000 - dffff
> > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > vfio: region_del 0 - c7fff
> > > vfio: region_del c8000 - dffff
> > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > vfio: region_del 0 - cffff
> > > vfio: region_del d0000 - dffff
> > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > vfio: region_del 0 - d7fff
> > > vfio: region_del d8000 - dffff
> > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > vfio: region_del 0 - dffff
> > > vfio: region_del e0000 - fffff
> > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > vfio: region_del 0 - e7fff
> > > vfio: region_del e8000 - fffff
> > > vfio: region_add 0 - effff [0x7f8698000000]
> > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > vfio: region_del 0 - effff
> > > vfio: region_del f0000 - fffff
> > > vfio: region_del 100000 - afffffff
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > vfio: SKIPPING region_add feb40800 - feb40807
> > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > vfio: SKIPPING region_add feb41800 - feb41807
> > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > vfio: SKIPPING region_add a0000 - bffff
> > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > 
> > Here the option ROM was sized
> > 
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> > 
> > Then enabled
> > 
> > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> > 
> > Adding this memory region
> > 
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: region_del fea40000 - fea5ffff
> > 
> > Then disabled, removing the memory region.  Presumably between the
> > enable and disable the contents were read and copied to 0xc0000, which
> > is where the VGA BIOS is shadowed.
> > 
> > > Here is the strace output from this failure:
> > > 
> > > 1110  ioctl(14, KVM_RUN, 0)             = 0
> > > 1110  pread(20,  <unfinished ...>
> > > 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> > > 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
> > > 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > 1109  _exit(0)                          = ?
> > > 1109  +++ exited with 0 +++
> > > 
> > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > in 'vfio_vga_read' never finished:
> > > 
> > >     if (pread(vga->fd, &buf, size, offset) != size) {
> > >         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > >                      __func__, region->offset + addr, size);
> > >         return (uint64_t)-1;
> > >     }
> > 
> > I agree.  Every VGA access requires us to lock the VGA resources on the
> > device, so if we can't get the lock, we stop making progress.  I took a
> > look at Xorg last night and it seems like it should be taking and
> > releasing the VGA arbiter lock in a way that would be compatible with
> > our use.  That's in the xserver, not the actual display hardware driver,
> > and it wraps access functions in the arbiter support, so should be
> > transparent to the drivers.  So for nouveau, it seems like it should
> > work.  For nvidia, we don't really know, it could be locking the device
> > from the kernel module.
> > 
> > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > what's happening.  It might be enough to look at /dev/vga_arbiter at
> > each step in the sequence to reproduce (sudo head
> > --lines=1 /dev/vga_arbiter).  Thanks,
> > 
> 
> I've played a bit more with it and there are some differences in
> behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> binary driver I see the following:
> 
> [   18.628676] [drm] Initialized drm 1.1.0 20060810
> [   18.668038] nvidia: module license 'NVIDIA' taints kernel.
> [   18.668107] Disabling lock debugging due to kernel taint
> [   18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [   18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> [   18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
> [   18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  319.23  Thu May 16 19:36:02 PDT 2013
> 
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# modprobe nvidia
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
>         -L /usr/local/share/qemu \
>         -L /usr/local/share/qemu \
>         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
>         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
>         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
>         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
>         -device ioh3420,id=pcie0,multifunction=on \
>         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
>         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> 
> It looks like nvidia binary driver locks some resources. :( Whereas
> with nouveau driver the VGA arbitration starts first if I start the
> VM with VGA passthrough:
> 
> [  178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> [  178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
> [  178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
> [  181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [  181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> 
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
>         -L /usr/local/share/qemu \
>         -L /usr/local/share/qemu \
>         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
>         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
>         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
>         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
>         -device ioh3420,id=pcie0,multifunction=on \
>         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
>         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
> 

Alex you pointed me in the right direction, many thanks! I got it
working with 'nvidia' binary driver now, but I had to patch it. I've
attached it here just for reference if others want to try the same
and I don't know if it is the proper way. After some testing (Xorg,
DRI, VDPAU, switching between X and text-console) it looks like a stable
workaround for me now until NVIDIA will fix their driver.

--Maik

[-- Attachment #2: NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix.patch --]
[-- Type: text/plain, Size: 1355 bytes --]

diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h
--- NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h	2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h	2013-05-29 18:09:42.382925622 +0200
@@ -151,9 +151,9 @@
 #error "struct file_operations compile test likely failed!"
 #endif
 
-#if defined(CONFIG_VGA_ARB)
-#include <linux/vgaarb.h>
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#include <linux/vgaarb.h>
+//#endif
 
 #if defined(NV_VM_INSERT_PAGE_PRESENT)
 #include <linux/pagemap.h>
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv.c NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c
--- NVIDIA-Linux-x86_64-319.23/kernel/nv.c	2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c	2013-05-29 18:10:01.494277314 +0200
@@ -2914,12 +2914,12 @@
 
     pci_set_master(dev);
 
-#if defined(CONFIG_VGA_ARB)
-#if defined(VGA_DEFAULT_DEVICE)
-    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
-#endif
-    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#if defined(VGA_DEFAULT_DEVICE)
+//    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
+//#endif
+//    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
+//#endif
 
     if (NV_IS_GVI_DEVICE(nv))
     {

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Wed, 2013-05-29 at 18:16 +0200, Maik Broemme wrote:
> Hi,
> 
> Maik Broemme <mbroemme@parallels.com> wrote:
> > > > 
> > > > Here is the DEBUG_VFIO output:
> > > > 
> > > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > > vfio: SKIPPING region_add fee00000 - feefffff
> > > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > > vfio: Device 0000:04:00.0 region 0:
> > > > vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> > > > vfio: Device 0000:04:00.0 region 1:
> > > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 region 2:
> > > > vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > > vfio: Device 0000:04:00.0 region 3:
> > > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 region 4:
> > > > vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> > > > vfio: Device 0000:04:00.0 region 5:
> > > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 ROM:
> > > > vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > > vfio: Device 0000:04:00.0 config:
> > > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > > vfio: vfio_load_rom(0000:04:00.0)
> > > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.0)
> > > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > > vfio: Device 0000:04:00.1 region 0:
> > > > vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> > > > vfio: Device 0000:04:00.1 region 1:
> > > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 2:
> > > > vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 3:
> > > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 4:
> > > > vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 5:
> > > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 ROM:
> > > > vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 config:
> > > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.1)
> > > > vfio: region_del 0 - afffffff
> > > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > > vfio: vfio_pci_reset(0000:04:00.0)
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > > vfio: vfio_disable_intx(0000:04:00.0)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.0)
> > > > vfio: vfio_pci_reset(0000:04:00.1)
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > > vfio: vfio_disable_intx(0000:04:00.1)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.1)
> > > > vfio: region_del 0 - bffff
> > > > vfio: region_del c0000 - dffff
> > > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > > vfio: region_del 0 - c7fff
> > > > vfio: region_del c8000 - dffff
> > > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > > vfio: region_del 0 - cffff
> > > > vfio: region_del d0000 - dffff
> > > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > > vfio: region_del 0 - d7fff
> > > > vfio: region_del d8000 - dffff
> > > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > > vfio: region_del 0 - dffff
> > > > vfio: region_del e0000 - fffff
> > > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > > vfio: region_del 0 - e7fff
> > > > vfio: region_del e8000 - fffff
> > > > vfio: region_add 0 - effff [0x7f8698000000]
> > > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > > vfio: region_del 0 - effff
> > > > vfio: region_del f0000 - fffff
> > > > vfio: region_del 100000 - afffffff
> > > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > > vfio: SKIPPING region_add feb40800 - feb40807
> > > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > > vfio: SKIPPING region_add feb41800 - feb41807
> > > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > > vfio: region_del 0 - afffffff
> > > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > > vfio: SKIPPING region_add a0000 - bffff
> > > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > 
> > > Here the option ROM was sized
> > > 
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> > > 
> > > Then enabled
> > > 
> > > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> > > 
> > > Adding this memory region
> > > 
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > > vfio: region_del fea40000 - fea5ffff
> > > 
> > > Then disabled, removing the memory region.  Presumably between the
> > > enable and disable the contents were read and copied to 0xc0000, which
> > > is where the VGA BIOS is shadowed.
> > > 
> > > > Here is the strace output from this failure:
> > > > 
> > > > 1110  ioctl(14, KVM_RUN, 0)             = 0
> > > > 1110  pread(20,  <unfinished ...>
> > > > 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> > > > 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > > 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
> > > > 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > > 1109  _exit(0)                          = ?
> > > > 1109  +++ exited with 0 +++
> > > > 
> > > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > > in 'vfio_vga_read' never finished:
> > > > 
> > > >     if (pread(vga->fd, &buf, size, offset) != size) {
> > > >         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > > >                      __func__, region->offset + addr, size);
> > > >         return (uint64_t)-1;
> > > >     }
> > > 
> > > I agree.  Every VGA access requires us to lock the VGA resources on the
> > > device, so if we can't get the lock, we stop making progress.  I took a
> > > look at Xorg last night and it seems like it should be taking and
> > > releasing the VGA arbiter lock in a way that would be compatible with
> > > our use.  That's in the xserver, not the actual display hardware driver,
> > > and it wraps access functions in the arbiter support, so should be
> > > transparent to the drivers.  So for nouveau, it seems like it should
> > > work.  For nvidia, we don't really know, it could be locking the device
> > > from the kernel module.
> > > 
> > > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > > what's happening.  It might be enough to look at /dev/vga_arbiter at
> > > each step in the sequence to reproduce (sudo head
> > > --lines=1 /dev/vga_arbiter).  Thanks,
> > > 
> > 
> > I've played a bit more with it and there are some differences in
> > behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> > binary driver I see the following:
> > 
> > [   18.628676] [drm] Initialized drm 1.1.0 20060810
> > [   18.668038] nvidia: module license 'NVIDIA' taints kernel.
> > [   18.668107] Disabling lock debugging due to kernel taint
> > [   18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> > [   18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> > [   18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
> > [   18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  319.23  Thu May 16 19:36:02 PDT 2013
> > 
> > root@homer:~# head --lines=1 /dev/vga_arbiter
> > count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> > root@homer:~# modprobe nvidia
> > root@homer:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> > root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> >         -L /usr/local/share/qemu \
> >         -L /usr/local/share/qemu \
> >         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> >         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> >         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> >         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> >         -device ioh3420,id=pcie0,multifunction=on \
> >         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> >         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> > root@homer:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> > 
> > It looks like nvidia binary driver locks some resources. :( Whereas
> > with nouveau driver the VGA arbitration starts first if I start the
> > VM with VGA passthrough:
> > 
> > [  178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> > [  178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
> > [  178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
> > [  181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> > [  181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> > 
> > root@homer:~# head --lines=1 /dev/vga_arbiter
> > count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> > root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> >         -L /usr/local/share/qemu \
> >         -L /usr/local/share/qemu \
> >         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> >         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> >         -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> >         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> >         -device ioh3420,id=pcie0,multifunction=on \
> >         -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> >         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> > root@homer:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
> > 
> 
> Alex you pointed me in the right direction, many thanks! I got it
> working with 'nvidia' binary driver now, but I had to patch it. I've
> attached it here just for reference if others want to try the same
> and I don't know if it is the proper way. After some testing (Xorg,
> DRI, VDPAU, switching between X and text-console) it looks like a stable
> workaround for me now until NVIDIA will fix their driver.

Hmm, the code doesn't make much sense to me.  They never do a vga_put,
so they must not realize that the vga_tryget is actually locking vga
arbitration.  It looks like they just want to point VGA routing to the
default device and indicate they don't use VGA I/O.  I don't really see
the point of the vga_tryget though since by declaring that they don't
use VGA I/O they give up all locks.  A simpler fix would be to simply
skip the vga_tryget call.  Or check the return value:

if (!vga_tryget(...))
    vga_put(...)

The VGA arbitration code is lazy, so VGA routing will still point to the
default devices, but it won't be locked there.  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

On Sun, 2013-05-19 at 23:26 +0400, Maik Broemme wrote:
> Hi Knut,
> 
> Knut Omang <knuto@ifi.uio.no> wrote:
> > 
> > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > Hi all,
> > > > 
> > > > Perfect timing from my perspective, thanks Alex!
> > > > 
> > > > I spent the better part of the weekend testing your branches on a new system 
> > > > I just put together for this purpose, results below..
> > > > 
> > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > ...
> > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > 
> > > > System setup: 
> > > > 
> > > > - Fedora 18 on
> > > > - Gigabyte Z77X-UD5H motherboard
> > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > - 2 discrete graphics cards:
> > > > 
> > > > lspci | egrep 'VGA|Audio'
> > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 
> > > > Short summary:
> > > > 
> > > > - Once I got past a few time consuming obstacles explained below
> > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > >      on both discrete graphics cards 
> > > >      (though so far only one at at time and with some minor issues, see below)
> > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > 
> > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > support.
> > 
> > In my first tries, Windows were just using a generic
> > VGA driver, which still seems to work perfect with reboots and everything 
> > and in full screen resolution (1920x1200).
> > However after installing the Catalyst AMD driver stack, upon boot
> > Windows 7 now consequently get a BSOD from the graphics driver
> > with the message:
> > 
> > "Attempt to reset the display driver and recover from timeout failed"
> > - a picture of the BSOD screen attached.
> > 
> > I attach the corresponding vfio log where I added some timing code to
> > make it easier to see when the BSOD happens (with 2 seconds of silence
> > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > the log.
> > 
> > Similar behaviour both just after reboot/power cycle of the host and
> > subsequent VM boot attempts.
> > 
> > This is still with the HD7700 as passed through device, but after a
> > motherboard firmware upgrade (to F14) which did not seem to affect the
> > observed behaviour on Windows prior to Catalyst install or with Linux
> > guest, neither did it fix the bug in selecting primary devices as I 
> > was hoping for.
> > 
> > Let me know if you have ideas for further debugging this,
> > 
> 
> I had a similar problem a couple of days ago and posted it in this list.
> I got similar BSOD and tested already the following configurations:
> 
> 1) machine: q35 / kvm: yes / vga: none   / x-vga: on  = qemu freeze
> 2) machine: q35 / kvm: no  / vga: none   / x-vga: on  = qemu freeze
>    (with errors below)
> 3) machine: q35 / kvm: yes / vga: none   / x-vga: off = qemu runs with
>    100% CPU due to no VGA init (no picture)
> 4) machine: q35 / kvm: yes / vga: cirrus / x-vga: off = qemu runs with
>    BOSD on loading atikmpag.sys
> 5) machine: pc  / kvm: yes / vga: cirrus / x-vga: off = qemu runs fine
> 
> However I've re-run the BSOD case already with the following branches
> from Alex:
> 
> git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> 
> Also with latest seabios and it worked so far. No more BSOD and reboot
> of VM was also possible without suspend / resume the host between.

Hmm, seems you have been more lucky with the choice of chipset/CPU than
me - all my tests are also with these branches but on Intel...

Knut

> > Thanks,
> > 
> > Knut
> > 
> > > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > > 
> > > > Details:
> > > > 
> > > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > > >    - this caused the system to freeze (with color artifacts on the console)
> > > > 
> > > > - In my naivity and because of the "i" in ifgx I tried both with 
> > > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > > 
> > > I'm not sure why this is, looks like the code only tries to turn it off
> > > when only graphics is under the remapping device.  We'd probably need to
> > > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > > 
> > > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > > 
> > > > - iommu groups:
> > > > 
> > > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > > total 0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > > 
> > > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > > 
> > > Ick.  Intel has been pretty good about advertising ACS support on their
> > > root ports.  I wonder if this is an oversight or if they are actually
> > > not isolated from each other.
> > > 
> > > > # lspci -n
> > > > ...
> > > > 01:00.0 0300: 1002:683f
> > > > 01:00.1 0403: 1002:aab0
> > > > 02:00.0 0300: 1002:6779
> > > > 02:00.1 0403: 1002:aa98
> > > > ...
> > > > 
> > > > modprobe vfio_pci
> > > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > 
> > > > # lsusb 
> > > > ...
> > > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > > ...
> > > > 
> > > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > > 
> > > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > > 
> > > > 
> > > > qemu-kvm -M q35 \
> > > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > > >   -device ide-cd,drive=cd,bus=ide.1 \
> > > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > > >   -enable-kvm \
> > > >   -device usb-host,hostbus=1,hostaddr=8 \
> > > >   -device usb-host,hostbus=1,hostaddr=4
> > > > 
> > > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > > 
> > > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > > vica versa?  The boot VGA ROM is handled in a special way and what you
> > > really get is the shadow copy, which isn't what we want.
> > > 
> > > > Anyway, supplying it to vfio did not seem to make any difference.
> > > > 
> > > > find /sys -name rom
> > > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > > ...
> > > > 
> > > > Some observations and remaining unresolved issues:
> > > > 
> > > > - VFIO patch:
> > > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > > >   I patched up accordingly with 
> > > > 
> > > > 
> > > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > > index da0e5f9..a361d06 100644
> > > > --- a/hw/misc/vfio.c
> > > > +++ b/hw/misc/vfio.c
> > > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > > >                                    addr + quirk->data.base_offset, size);
> > > >  
> > > > -    if (data == quirk->data.address_match) {
> > > > +    if (1 || data == quirk->data.address_match) {
> > > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > > >      }
> > > > 
> > > > 
> > > >   This of course did not help much until I actually got the iommu 
> > > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > > >   but what I have observed after I got it working is that if 
> > > >   I disable the patch above, things are not that well: the Fedora VM 
> > > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > > >   switching to X.
> > > 
> > > Hmm, I think we'd probably have better luck making that unconditional
> > > until we have reason to do otherwise.
> > > 
> > > > - The fact that the iommu group now extends across all my available graphics 
> > > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > > >   the other card since the vfio_pci driver needs to hold it.
> > > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > > >   Might it be a good idea to have an override option (exception list or similar?) 
> > > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > > >    - allow functionality over security in such case? This of course is further complicated
> > > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > > 
> > > We have a quirk in the kernel that enables us to witelist devices, but
> > > yes, there is no flexibility in this w/o modifying the code and
> > > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > > the example above w/ pci_dev_dma_source - function can just return 1)
> > > 
> > > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > > >   fixes for this in newer BIOSes.
> > > > 
> > > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > > >   Linux guests to try to get some sound through the HDMI audio device. 
> > > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > > >   PulseAudio has problems communicating with the passed through device (?), 
> > > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > > >   config space is accessed ok.
> > > > 
> > > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > > 
> > > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > > >   could be extended/modified to only affect iommu operation on selected device(s),
> > > >   if at all possible..
> > > 
> > > Let's see what we can discover from your DMAR.  Also send along sudo
> > > lspci -vvv.  Thanks,
> > > 
> > > Alex
> > > 
> > 
> > 
> 
> --Maik

Re: VFIO VGA test branches

[Alex Williamson]

On Sun, 2013-05-19 at 17:35 +0200, Knut Omang wrote:
> On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > Hi all,
> > > 
> > > Perfect timing from my perspective, thanks Alex!
> > > 
> > > I spent the better part of the weekend testing your branches on a new system 
> > > I just put together for this purpose, results below..
> > > 
> > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > ...
> > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > 
> > > System setup: 
> > > 
> > > - Fedora 18 on
> > > - Gigabyte Z77X-UD5H motherboard
> > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > - 2 discrete graphics cards:
> > > 
> > > lspci | egrep 'VGA|Audio'
> > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > 
> > > Short summary:
> > > 
> > > - Once I got past a few time consuming obstacles explained below
> > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > >      on both discrete graphics cards 
> > >      (though so far only one at at time and with some minor issues, see below)
> > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > 
> > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > support.
> 
> In my first tries, Windows were just using a generic
> VGA driver, which still seems to work perfect with reboots and everything 
> and in full screen resolution (1920x1200).
> However after installing the Catalyst AMD driver stack, upon boot
> Windows 7 now consequently get a BSOD from the graphics driver
> with the message:
> 
> "Attempt to reset the display driver and recover from timeout failed"
> - a picture of the BSOD screen attached.

I've seen that BSOD before, but I don't know how to reproduce it.  It
seems like I haven't seen it with the PCI bus reset code.  I'm running
version 13.1 of the catalyst driver, you?

> I attach the corresponding vfio log where I added some timing code to
> make it easier to see when the BSOD happens (with 2 seconds of silence
> in the log before the VM reboots, I believe this is at 09:28:32-34 in
> the log.

Yep, looks like that's where windows starts the BSOD.

> Similar behaviour both just after reboot/power cycle of the host and
> subsequent VM boot attempts.
> 
> This is still with the HD7700 as passed through device, but after a
> motherboard firmware upgrade (to F14) which did not seem to affect the
> observed behaviour on Windows prior to Catalyst install or with Linux
> guest, neither did it fix the bug in selecting primary devices as I 
> was hoping for.
> 
> Let me know if you have ideas for further debugging this,

I don't have any great ideas since I don't know how to reproduce the
timeout.  Double/triple check that you're using the correct
vfio-vga-reset branches in both qemu and kernel

# grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
# grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c

I didn't see anything telling in your DMAR either.  The system seems to
have just one DRHD that includes everything, so I'm not sure why you saw
any behavior change from igfx_off.  Thanks,

Alex

> > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > 
> > > Details:
> > > 
> > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > >    - this caused the system to freeze (with color artifacts on the console)
> > > 
> > > - In my naivity and because of the "i" in ifgx I tried both with 
> > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > 
> > I'm not sure why this is, looks like the code only tries to turn it off
> > when only graphics is under the remapping device.  We'd probably need to
> > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > 
> > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > 
> > > - iommu groups:
> > > 
> > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > total 0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > 
> > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > 
> > Ick.  Intel has been pretty good about advertising ACS support on their
> > root ports.  I wonder if this is an oversight or if they are actually
> > not isolated from each other.
> > 
> > > # lspci -n
> > > ...
> > > 01:00.0 0300: 1002:683f
> > > 01:00.1 0403: 1002:aab0
> > > 02:00.0 0300: 1002:6779
> > > 02:00.1 0403: 1002:aa98
> > > ...
> > > 
> > > modprobe vfio_pci
> > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > 
> > > # lsusb 
> > > ...
> > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > ...
> > > 
> > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > 
> > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > 
> > > 
> > > qemu-kvm -M q35 \
> > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > >   -device ide-cd,drive=cd,bus=ide.1 \
> > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > >   -enable-kvm \
> > >   -device usb-host,hostbus=1,hostaddr=8 \
> > >   -device usb-host,hostbus=1,hostaddr=4
> > > 
> > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > 
> > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > vica versa?  The boot VGA ROM is handled in a special way and what you
> > really get is the shadow copy, which isn't what we want.
> > 
> > > Anyway, supplying it to vfio did not seem to make any difference.
> > > 
> > > find /sys -name rom
> > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > ...
> > > 
> > > Some observations and remaining unresolved issues:
> > > 
> > > - VFIO patch:
> > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > >   I patched up accordingly with 
> > > 
> > > 
> > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > index da0e5f9..a361d06 100644
> > > --- a/hw/misc/vfio.c
> > > +++ b/hw/misc/vfio.c
> > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > >                                    addr + quirk->data.base_offset, size);
> > >  
> > > -    if (data == quirk->data.address_match) {
> > > +    if (1 || data == quirk->data.address_match) {
> > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > >      }
> > > 
> > > 
> > >   This of course did not help much until I actually got the iommu 
> > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > >   but what I have observed after I got it working is that if 
> > >   I disable the patch above, things are not that well: the Fedora VM 
> > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > >   switching to X.
> > 
> > Hmm, I think we'd probably have better luck making that unconditional
> > until we have reason to do otherwise.
> > 
> > > - The fact that the iommu group now extends across all my available graphics 
> > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > >   the other card since the vfio_pci driver needs to hold it.
> > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > >   Might it be a good idea to have an override option (exception list or similar?) 
> > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > >    - allow functionality over security in such case? This of course is further complicated
> > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > 
> > We have a quirk in the kernel that enables us to witelist devices, but
> > yes, there is no flexibility in this w/o modifying the code and
> > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > the example above w/ pci_dev_dma_source - function can just return 1)
> > 
> > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > >   fixes for this in newer BIOSes.
> > > 
> > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > >   Linux guests to try to get some sound through the HDMI audio device. 
> > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > >   PulseAudio has problems communicating with the passed through device (?), 
> > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > >   config space is accessed ok.
> > > 
> > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > 
> > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > >   could be extended/modified to only affect iommu operation on selected device(s),
> > >   if at all possible..
> > 
> > Let's see what we can discover from your DMAR.  Also send along sudo
> > lspci -vvv.  Thanks,
> > 
> > Alex
> > 
> 
> 

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Sun, 2013-05-19 at 17:35 +0200, Knut Omang wrote:
> On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > Hi all,
> > > 
> > > Perfect timing from my perspective, thanks Alex!
> > > 
> > > I spent the better part of the weekend testing your branches on a new system 
> > > I just put together for this purpose, results below..
> > > 
> > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > ...
> > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > 
> > > System setup: 
> > > 
> > > - Fedora 18 on
> > > - Gigabyte Z77X-UD5H motherboard
> > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > - 2 discrete graphics cards:
> > > 
> > > lspci | egrep 'VGA|Audio'
> > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > 
> > > Short summary:
> > > 
> > > - Once I got past a few time consuming obstacles explained below
> > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > >      on both discrete graphics cards 
> > >      (though so far only one at at time and with some minor issues, see below)
> > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > 
> > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > support.
> 
> In my first tries, Windows were just using a generic
> VGA driver, which still seems to work perfect with reboots and everything 
> and in full screen resolution (1920x1200).
> However after installing the Catalyst AMD driver stack, upon boot
> Windows 7 now consequently get a BSOD from the graphics driver
> with the message:
> 
> "Attempt to reset the display driver and recover from timeout failed"
> - a picture of the BSOD screen attached.

I've seen that BSOD before, but I don't know how to reproduce it.  It
seems like I haven't seen it with the PCI bus reset code.  I'm running
version 13.1 of the catalyst driver, you?

> I attach the corresponding vfio log where I added some timing code to
> make it easier to see when the BSOD happens (with 2 seconds of silence
> in the log before the VM reboots, I believe this is at 09:28:32-34 in
> the log.

Yep, looks like that's where windows starts the BSOD.

> Similar behaviour both just after reboot/power cycle of the host and
> subsequent VM boot attempts.
> 
> This is still with the HD7700 as passed through device, but after a
> motherboard firmware upgrade (to F14) which did not seem to affect the
> observed behaviour on Windows prior to Catalyst install or with Linux
> guest, neither did it fix the bug in selecting primary devices as I 
> was hoping for.
> 
> Let me know if you have ideas for further debugging this,

I don't have any great ideas since I don't know how to reproduce the
timeout.  Double/triple check that you're using the correct
vfio-vga-reset branches in both qemu and kernel

# grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
# grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c

I didn't see anything telling in your DMAR either.  The system seems to
have just one DRHD that includes everything, so I'm not sure why you saw
any behavior change from igfx_off.  Thanks,

Alex

> > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > 
> > > Details:
> > > 
> > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > >    - this caused the system to freeze (with color artifacts on the console)
> > > 
> > > - In my naivity and because of the "i" in ifgx I tried both with 
> > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > 
> > I'm not sure why this is, looks like the code only tries to turn it off
> > when only graphics is under the remapping device.  We'd probably need to
> > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > 
> > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > 
> > > - iommu groups:
> > > 
> > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > total 0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > 
> > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > 
> > Ick.  Intel has been pretty good about advertising ACS support on their
> > root ports.  I wonder if this is an oversight or if they are actually
> > not isolated from each other.
> > 
> > > # lspci -n
> > > ...
> > > 01:00.0 0300: 1002:683f
> > > 01:00.1 0403: 1002:aab0
> > > 02:00.0 0300: 1002:6779
> > > 02:00.1 0403: 1002:aa98
> > > ...
> > > 
> > > modprobe vfio_pci
> > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > 
> > > # lsusb 
> > > ...
> > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > ...
> > > 
> > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > 
> > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > 
> > > 
> > > qemu-kvm -M q35 \
> > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > >   -device ide-cd,drive=cd,bus=ide.1 \
> > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > >   -enable-kvm \
> > >   -device usb-host,hostbus=1,hostaddr=8 \
> > >   -device usb-host,hostbus=1,hostaddr=4
> > > 
> > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > 
> > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > vica versa?  The boot VGA ROM is handled in a special way and what you
> > really get is the shadow copy, which isn't what we want.
> > 
> > > Anyway, supplying it to vfio did not seem to make any difference.
> > > 
> > > find /sys -name rom
> > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > ...
> > > 
> > > Some observations and remaining unresolved issues:
> > > 
> > > - VFIO patch:
> > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > >   I patched up accordingly with 
> > > 
> > > 
> > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > index da0e5f9..a361d06 100644
> > > --- a/hw/misc/vfio.c
> > > +++ b/hw/misc/vfio.c
> > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > >                                    addr + quirk->data.base_offset, size);
> > >  
> > > -    if (data == quirk->data.address_match) {
> > > +    if (1 || data == quirk->data.address_match) {
> > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > >      }
> > > 
> > > 
> > >   This of course did not help much until I actually got the iommu 
> > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > >   but what I have observed after I got it working is that if 
> > >   I disable the patch above, things are not that well: the Fedora VM 
> > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > >   switching to X.
> > 
> > Hmm, I think we'd probably have better luck making that unconditional
> > until we have reason to do otherwise.
> > 
> > > - The fact that the iommu group now extends across all my available graphics 
> > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > >   the other card since the vfio_pci driver needs to hold it.
> > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > >   Might it be a good idea to have an override option (exception list or similar?) 
> > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > >    - allow functionality over security in such case? This of course is further complicated
> > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > 
> > We have a quirk in the kernel that enables us to witelist devices, but
> > yes, there is no flexibility in this w/o modifying the code and
> > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > the example above w/ pci_dev_dma_source - function can just return 1)
> > 
> > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > >   fixes for this in newer BIOSes.
> > > 
> > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > >   Linux guests to try to get some sound through the HDMI audio device. 
> > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > >   PulseAudio has problems communicating with the passed through device (?), 
> > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > >   config space is accessed ok.
> > > 
> > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > 
> > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > >   could be extended/modified to only affect iommu operation on selected device(s),
> > >   if at all possible..
> > 
> > Let's see what we can discover from your DMAR.  Also send along sudo
> > lspci -vvv.  Thanks,
> > 
> > Alex
> > 
> 
> 

Re: VFIO VGA test branches

[Knut Omang]

[-- Attachment #1: Type: text/plain, Size: 15801 bytes --]

On Sun, 2013-05-19 at 22:15 -0600, Alex Williamson wrote:
> On Sun, 2013-05-19 at 17:35 +0200, Knut Omang wrote:
> > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > Hi all,
> > > > 
> > > > Perfect timing from my perspective, thanks Alex!
> > > > 
> > > > I spent the better part of the weekend testing your branches on a new system 
> > > > I just put together for this purpose, results below..
> > > > 
> > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > ...
> > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > 
> > > > System setup: 
> > > > 
> > > > - Fedora 18 on
> > > > - Gigabyte Z77X-UD5H motherboard
> > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > - 2 discrete graphics cards:
> > > > 
> > > > lspci | egrep 'VGA|Audio'
> > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 
> > > > Short summary:
> > > > 
> > > > - Once I got past a few time consuming obstacles explained below
> > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > >      on both discrete graphics cards 
> > > >      (though so far only one at at time and with some minor issues, see below)
> > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > 
> > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > support.
> > 
> > In my first tries, Windows were just using a generic
> > VGA driver, which still seems to work perfect with reboots and everything 
> > and in full screen resolution (1920x1200).
> > However after installing the Catalyst AMD driver stack, upon boot
> > Windows 7 now consequently get a BSOD from the graphics driver
> > with the message:
> > 
> > "Attempt to reset the display driver and recover from timeout failed"
> > - a picture of the BSOD screen attached.
> 
> I've seen that BSOD before, but I don't know how to reproduce it.  It
> seems like I haven't seen it with the PCI bus reset code.  I'm running
> version 13.1 of the catalyst driver, you?

I first tried with the install CD that came with the card - v.13-045
then upgraded to the latest from AMD, catalyst v.13.4 which appears to
be driver v.12.104 - similar behaviour for both. This was with a plain
Windows 7 install from my SP1 DVD. 

With most recommended windows updates and the latest catalyst driver,
the BSOD is gone but instead I see the initial VGA boot screen and the
windows logo, then syncs but no display and then reboot into recovery
mode. (If I try all updates, Windows seems never to be able to recover
from the last reboot)

I have tried without kvm and also with vnc or spice graphics in addition
but in those cases it seems Windows is not able to allocate MMIO
resources for both adapters so I haven't been able to test the catalyst
driver as a secondary windows display.

> > I attach the corresponding vfio log where I added some timing code to
> > make it easier to see when the BSOD happens (with 2 seconds of silence
> > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > the log.
> 
> Yep, looks like that's where windows starts the BSOD.
> 
> > Similar behaviour both just after reboot/power cycle of the host and
> > subsequent VM boot attempts.
> > 
> > This is still with the HD7700 as passed through device, but after a
> > motherboard firmware upgrade (to F14) which did not seem to affect the
> > observed behaviour on Windows prior to Catalyst install or with Linux
> > guest, neither did it fix the bug in selecting primary devices as I 
> > was hoping for.
> > 
> > Let me know if you have ideas for further debugging this,
> 
> I don't have any great ideas since I don't know how to reproduce the
> timeout.  Double/triple check that you're using the correct
> vfio-vga-reset branches in both qemu and kernel
> 
> # grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
> # grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c

[Matches in both..]
I do believe I have used the right branches all along.

> I didn't see anything telling in your DMAR either.  The system seems to
> have just one DRHD that includes everything, so I'm not sure why you saw
> any behavior change from igfx_off.  Thanks,

After the firmware upgrade, I tried again with the integrated graphics
enabled, this time with more success - I am now able to get a GUI fedora
console on the integrated graphics, but see some colorful artifacts
there during the VGA startup on one of the Radeon cards, which goes away
with a toggle to another console and back.

Seems I have slightly mislead you with the DMAR table - sorry about that
- the table I posted was with the igfx disabled, with the igfx enabled I
see one more hardware unit dedicated to the igfx if I am able to
interpret it right (attached)

Both the HD7700 and the HD6450 behave very similar and both still starts
and displays Windows fine if I disable the Catalyst driver.

Knut

> Alex
> 
> > > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > > 
> > > > Details:
> > > > 
> > > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > > >    - this caused the system to freeze (with color artifacts on the console)
> > > > 
> > > > - In my naivity and because of the "i" in ifgx I tried both with 
> > > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > > 
> > > I'm not sure why this is, looks like the code only tries to turn it off
> > > when only graphics is under the remapping device.  We'd probably need to
> > > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > > 
> > > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > > 
> > > > - iommu groups:
> > > > 
> > > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > > total 0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > > 
> > > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > > 
> > > Ick.  Intel has been pretty good about advertising ACS support on their
> > > root ports.  I wonder if this is an oversight or if they are actually
> > > not isolated from each other.
> > > 
> > > > # lspci -n
> > > > ...
> > > > 01:00.0 0300: 1002:683f
> > > > 01:00.1 0403: 1002:aab0
> > > > 02:00.0 0300: 1002:6779
> > > > 02:00.1 0403: 1002:aa98
> > > > ...
> > > > 
> > > > modprobe vfio_pci
> > > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > 
> > > > # lsusb 
> > > > ...
> > > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > > ...
> > > > 
> > > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > > 
> > > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > > 
> > > > 
> > > > qemu-kvm -M q35 \
> > > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > > >   -device ide-cd,drive=cd,bus=ide.1 \
> > > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > > >   -enable-kvm \
> > > >   -device usb-host,hostbus=1,hostaddr=8 \
> > > >   -device usb-host,hostbus=1,hostaddr=4
> > > > 
> > > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > > 
> > > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > > vica versa?  The boot VGA ROM is handled in a special way and what you
> > > really get is the shadow copy, which isn't what we want.
> > > 
> > > > Anyway, supplying it to vfio did not seem to make any difference.
> > > > 
> > > > find /sys -name rom
> > > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > > ...
> > > > 
> > > > Some observations and remaining unresolved issues:
> > > > 
> > > > - VFIO patch:
> > > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > > >   I patched up accordingly with 
> > > > 
> > > > 
> > > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > > index da0e5f9..a361d06 100644
> > > > --- a/hw/misc/vfio.c
> > > > +++ b/hw/misc/vfio.c
> > > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > > >                                    addr + quirk->data.base_offset, size);
> > > >  
> > > > -    if (data == quirk->data.address_match) {
> > > > +    if (1 || data == quirk->data.address_match) {
> > > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > > >      }
> > > > 
> > > > 
> > > >   This of course did not help much until I actually got the iommu 
> > > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > > >   but what I have observed after I got it working is that if 
> > > >   I disable the patch above, things are not that well: the Fedora VM 
> > > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > > >   switching to X.
> > > 
> > > Hmm, I think we'd probably have better luck making that unconditional
> > > until we have reason to do otherwise.
> > > 
> > > > - The fact that the iommu group now extends across all my available graphics 
> > > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > > >   the other card since the vfio_pci driver needs to hold it.
> > > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > > >   Might it be a good idea to have an override option (exception list or similar?) 
> > > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > > >    - allow functionality over security in such case? This of course is further complicated
> > > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > > 
> > > We have a quirk in the kernel that enables us to witelist devices, but
> > > yes, there is no flexibility in this w/o modifying the code and
> > > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > > the example above w/ pci_dev_dma_source - function can just return 1)
> > > 
> > > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > > >   fixes for this in newer BIOSes.
> > > > 
> > > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > > >   Linux guests to try to get some sound through the HDMI audio device. 
> > > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > > >   PulseAudio has problems communicating with the passed through device (?), 
> > > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > > >   config space is accessed ok.
> > > > 
> > > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > > 
> > > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > > >   could be extended/modified to only affect iommu operation on selected device(s),
> > > >   if at all possible..
> > > 
> > > Let's see what we can discover from your DMAR.  Also send along sudo
> > > lspci -vvv.  Thanks,
> > > 
> > > Alex
> > > 
> > 
> > 
> 
> 
> 


[-- Attachment #2: DMAR_igfx.dsl --]
[-- Type: text/x-dsl, Size: 5266 bytes --]

/*
 * Intel ACPI Component Architecture
 * AML Disassembler version 20100528
 *
 * Disassembly of DMAR_igfx.raw, Mon May 20 19:31:19 2013
 *
 * ACPI Data Table [DMAR]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 0000  4]                    Signature : "DMAR"    /* DMA Remapping table */
[004h 0004  4]                 Table Length : 000000B8
[008h 0008  1]                     Revision : 01
[009h 0009  1]                     Checksum : 19
[00Ah 0010  6]                       Oem ID : "INTEL "
[010h 0016  8]                 Oem Table ID : "SNB "
[018h 0024  4]                 Oem Revision : 00000001
[01Ch 0028  4]              Asl Compiler ID : "INTL"
[020h 0032  4]        Asl Compiler Revision : 00000001

[024h 0036  1]           Host Address Width : 23
[025h 0037  1]                        Flags : 01

[030h 0048  2]                Subtable Type : 0000 <Hardware Unit Definition>
[032h 0050  2]                       Length : 0018
[034h 0052  1]                        Flags : 00
[035h 0053  1]                     Reserved : 00
[036h 0054  2]           PCI Segment Number : 0000
[038h 0056  8]        Register Base Address : 00000000FED90000

[040h 0064  1]      Device Scope Entry Type : 01
[041h 0065  1]                 Entry Length : 08
[042h 0066  2]                     Reserved : 0000
[044h 0068  1]               Enumeration ID : 00
[045h 0069  1]               PCI Bus Number : 00
[046h 0070  2]                     PCI Path : [02, 00]

[048h 0072  2]                Subtable Type : 0000 <Hardware Unit Definition>
[04Ah 0074  2]                       Length : 0020
[04Ch 0076  1]                        Flags : 01
[04Dh 0077  1]                     Reserved : 00
[04Eh 0078  2]           PCI Segment Number : 0000
[050h 0080  8]        Register Base Address : 00000000FED91000

[058h 0088  1]      Device Scope Entry Type : 03
[059h 0089  1]                 Entry Length : 08
[05Ah 0090  2]                     Reserved : 0000
[05Ch 0092  1]               Enumeration ID : 02
[05Dh 0093  1]               PCI Bus Number : F0
[05Eh 0094  2]                     PCI Path : [1F, 00]

[060h 0096  1]      Device Scope Entry Type : 04
[061h 0097  1]                 Entry Length : 08
[062h 0098  2]                     Reserved : 0000
[064h 0100  1]               Enumeration ID : 00
[065h 0101  1]               PCI Bus Number : F0
[066h 0102  2]                     PCI Path : [0F, 00]

[068h 0104  2]                Subtable Type : 0001 <Reserved Memory Region>
[06Ah 0106  2]                       Length : 0030
[06Ch 0108  2]                     Reserved : 0000
[06Eh 0110  2]           PCI Segment Number : 0000
[070h 0112  8]                 Base Address : 00000000AD86C000
[078h 0120  8]          End Address (limit) : 00000000AD896FFF

[080h 0128  1]      Device Scope Entry Type : 01
[081h 0129  1]                 Entry Length : 08
[082h 0130  2]                     Reserved : 0000
[084h 0132  1]               Enumeration ID : 00
[085h 0133  1]               PCI Bus Number : 00
[086h 0134  2]                     PCI Path : [1D, 00]

[088h 0136  1]      Device Scope Entry Type : 01
[089h 0137  1]                 Entry Length : 08
[08Ah 0138  2]                     Reserved : 0000
[08Ch 0140  1]               Enumeration ID : 00
[08Dh 0141  1]               PCI Bus Number : 00
[08Eh 0142  2]                     PCI Path : [1A, 00]

[090h 0144  1]      Device Scope Entry Type : 01
[091h 0145  1]                 Entry Length : 08
[092h 0146  2]                     Reserved : 0000
[094h 0148  1]               Enumeration ID : 00
[095h 0149  1]               PCI Bus Number : 00
[096h 0150  2]                     PCI Path : [14, 00]

[098h 0152  2]                Subtable Type : 0001 <Reserved Memory Region>
[09Ah 0154  2]                       Length : 0020
[09Ch 0156  2]                     Reserved : 0000
[09Eh 0158  2]           PCI Segment Number : 0000
[0A0h 0160  8]                 Base Address : 00000000AF800000
[0A8h 0168  8]          End Address (limit) : 00000000BF9FFFFF

[0B0h 0176  1]      Device Scope Entry Type : 01
[0B1h 0177  1]                 Entry Length : 08
[0B2h 0178  2]                     Reserved : 0000
[0B4h 0180  1]               Enumeration ID : 00
[0B5h 0181  1]               PCI Bus Number : 00
[0B6h 0182  2]                     PCI Path : [02, 00]

Raw Table Data

  0000: 44 4D 41 52 B8 00 00 00 01 19 49 4E 54 45 4C 20  DMAR......INTEL 
  0010: 53 4E 42 20 00 00 00 00 01 00 00 00 49 4E 54 4C  SNB ........INTL
  0020: 01 00 00 00 23 01 00 00 00 00 00 00 00 00 00 00  ....#...........
  0030: 00 00 18 00 00 00 00 00 00 00 D9 FE 00 00 00 00  ................
  0040: 01 08 00 00 00 00 02 00 00 00 20 00 01 00 00 00  .......... .....
  0050: 00 10 D9 FE 00 00 00 00 03 08 00 00 02 F0 1F 00  ................
  0060: 04 08 00 00 00 F0 0F 00 01 00 30 00 00 00 00 00  ..........0.....
  0070: 00 C0 86 AD 00 00 00 00 FF 6F 89 AD 00 00 00 00  .........o......
  0080: 01 08 00 00 00 00 1D 00 01 08 00 00 00 00 1A 00  ................
  0090: 01 08 00 00 00 00 14 00 01 00 20 00 00 00 00 00  .......... .....
  00A0: 00 00 80 AF 00 00 00 00 FF FF 9F BF 00 00 00 00  ................
  00B0: 01 08 00 00 00 00 02 00                          ........

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

[-- Attachment #1: Type: text/plain, Size: 15801 bytes --]

On Sun, 2013-05-19 at 22:15 -0600, Alex Williamson wrote:
> On Sun, 2013-05-19 at 17:35 +0200, Knut Omang wrote:
> > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > Hi all,
> > > > 
> > > > Perfect timing from my perspective, thanks Alex!
> > > > 
> > > > I spent the better part of the weekend testing your branches on a new system 
> > > > I just put together for this purpose, results below..
> > > > 
> > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > ...
> > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > 
> > > > System setup: 
> > > > 
> > > > - Fedora 18 on
> > > > - Gigabyte Z77X-UD5H motherboard
> > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > - 2 discrete graphics cards:
> > > > 
> > > > lspci | egrep 'VGA|Audio'
> > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 
> > > > Short summary:
> > > > 
> > > > - Once I got past a few time consuming obstacles explained below
> > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > >      on both discrete graphics cards 
> > > >      (though so far only one at at time and with some minor issues, see below)
> > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > 
> > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > support.
> > 
> > In my first tries, Windows were just using a generic
> > VGA driver, which still seems to work perfect with reboots and everything 
> > and in full screen resolution (1920x1200).
> > However after installing the Catalyst AMD driver stack, upon boot
> > Windows 7 now consequently get a BSOD from the graphics driver
> > with the message:
> > 
> > "Attempt to reset the display driver and recover from timeout failed"
> > - a picture of the BSOD screen attached.
> 
> I've seen that BSOD before, but I don't know how to reproduce it.  It
> seems like I haven't seen it with the PCI bus reset code.  I'm running
> version 13.1 of the catalyst driver, you?

I first tried with the install CD that came with the card - v.13-045
then upgraded to the latest from AMD, catalyst v.13.4 which appears to
be driver v.12.104 - similar behaviour for both. This was with a plain
Windows 7 install from my SP1 DVD. 

With most recommended windows updates and the latest catalyst driver,
the BSOD is gone but instead I see the initial VGA boot screen and the
windows logo, then syncs but no display and then reboot into recovery
mode. (If I try all updates, Windows seems never to be able to recover
from the last reboot)

I have tried without kvm and also with vnc or spice graphics in addition
but in those cases it seems Windows is not able to allocate MMIO
resources for both adapters so I haven't been able to test the catalyst
driver as a secondary windows display.

> > I attach the corresponding vfio log where I added some timing code to
> > make it easier to see when the BSOD happens (with 2 seconds of silence
> > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > the log.
> 
> Yep, looks like that's where windows starts the BSOD.
> 
> > Similar behaviour both just after reboot/power cycle of the host and
> > subsequent VM boot attempts.
> > 
> > This is still with the HD7700 as passed through device, but after a
> > motherboard firmware upgrade (to F14) which did not seem to affect the
> > observed behaviour on Windows prior to Catalyst install or with Linux
> > guest, neither did it fix the bug in selecting primary devices as I 
> > was hoping for.
> > 
> > Let me know if you have ideas for further debugging this,
> 
> I don't have any great ideas since I don't know how to reproduce the
> timeout.  Double/triple check that you're using the correct
> vfio-vga-reset branches in both qemu and kernel
> 
> # grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
> # grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c

[Matches in both..]
I do believe I have used the right branches all along.

> I didn't see anything telling in your DMAR either.  The system seems to
> have just one DRHD that includes everything, so I'm not sure why you saw
> any behavior change from igfx_off.  Thanks,

After the firmware upgrade, I tried again with the integrated graphics
enabled, this time with more success - I am now able to get a GUI fedora
console on the integrated graphics, but see some colorful artifacts
there during the VGA startup on one of the Radeon cards, which goes away
with a toggle to another console and back.

Seems I have slightly mislead you with the DMAR table - sorry about that
- the table I posted was with the igfx disabled, with the igfx enabled I
see one more hardware unit dedicated to the igfx if I am able to
interpret it right (attached)

Both the HD7700 and the HD6450 behave very similar and both still starts
and displays Windows fine if I disable the Catalyst driver.

Knut

> Alex
> 
> > > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > > 
> > > > Details:
> > > > 
> > > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > > >    - this caused the system to freeze (with color artifacts on the console)
> > > > 
> > > > - In my naivity and because of the "i" in ifgx I tried both with 
> > > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > > 
> > > I'm not sure why this is, looks like the code only tries to turn it off
> > > when only graphics is under the remapping device.  We'd probably need to
> > > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > > 
> > > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > > 
> > > > - iommu groups:
> > > > 
> > > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > > total 0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > > 
> > > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > > 
> > > Ick.  Intel has been pretty good about advertising ACS support on their
> > > root ports.  I wonder if this is an oversight or if they are actually
> > > not isolated from each other.
> > > 
> > > > # lspci -n
> > > > ...
> > > > 01:00.0 0300: 1002:683f
> > > > 01:00.1 0403: 1002:aab0
> > > > 02:00.0 0300: 1002:6779
> > > > 02:00.1 0403: 1002:aa98
> > > > ...
> > > > 
> > > > modprobe vfio_pci
> > > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > 
> > > > # lsusb 
> > > > ...
> > > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > > ...
> > > > 
> > > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > > 
> > > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > > 
> > > > 
> > > > qemu-kvm -M q35 \
> > > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > > >   -device ide-cd,drive=cd,bus=ide.1 \
> > > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > > >   -enable-kvm \
> > > >   -device usb-host,hostbus=1,hostaddr=8 \
> > > >   -device usb-host,hostbus=1,hostaddr=4
> > > > 
> > > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > > 
> > > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > > vica versa?  The boot VGA ROM is handled in a special way and what you
> > > really get is the shadow copy, which isn't what we want.
> > > 
> > > > Anyway, supplying it to vfio did not seem to make any difference.
> > > > 
> > > > find /sys -name rom
> > > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > > ...
> > > > 
> > > > Some observations and remaining unresolved issues:
> > > > 
> > > > - VFIO patch:
> > > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > > >   I patched up accordingly with 
> > > > 
> > > > 
> > > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > > index da0e5f9..a361d06 100644
> > > > --- a/hw/misc/vfio.c
> > > > +++ b/hw/misc/vfio.c
> > > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > > >                                    addr + quirk->data.base_offset, size);
> > > >  
> > > > -    if (data == quirk->data.address_match) {
> > > > +    if (1 || data == quirk->data.address_match) {
> > > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > > >      }
> > > > 
> > > > 
> > > >   This of course did not help much until I actually got the iommu 
> > > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > > >   but what I have observed after I got it working is that if 
> > > >   I disable the patch above, things are not that well: the Fedora VM 
> > > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > > >   switching to X.
> > > 
> > > Hmm, I think we'd probably have better luck making that unconditional
> > > until we have reason to do otherwise.
> > > 
> > > > - The fact that the iommu group now extends across all my available graphics 
> > > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > > >   the other card since the vfio_pci driver needs to hold it.
> > > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > > >   Might it be a good idea to have an override option (exception list or similar?) 
> > > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > > >    - allow functionality over security in such case? This of course is further complicated
> > > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > > 
> > > We have a quirk in the kernel that enables us to witelist devices, but
> > > yes, there is no flexibility in this w/o modifying the code and
> > > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > > the example above w/ pci_dev_dma_source - function can just return 1)
> > > 
> > > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > > >   fixes for this in newer BIOSes.
> > > > 
> > > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > > >   Linux guests to try to get some sound through the HDMI audio device. 
> > > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > > >   PulseAudio has problems communicating with the passed through device (?), 
> > > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > > >   config space is accessed ok.
> > > > 
> > > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > > 
> > > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > > >   could be extended/modified to only affect iommu operation on selected device(s),
> > > >   if at all possible..
> > > 
> > > Let's see what we can discover from your DMAR.  Also send along sudo
> > > lspci -vvv.  Thanks,
> > > 
> > > Alex
> > > 
> > 
> > 
> 
> 
> 


[-- Attachment #2: DMAR_igfx.dsl --]
[-- Type: text/x-dsl, Size: 5266 bytes --]

/*
 * Intel ACPI Component Architecture
 * AML Disassembler version 20100528
 *
 * Disassembly of DMAR_igfx.raw, Mon May 20 19:31:19 2013
 *
 * ACPI Data Table [DMAR]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 0000  4]                    Signature : "DMAR"    /* DMA Remapping table */
[004h 0004  4]                 Table Length : 000000B8
[008h 0008  1]                     Revision : 01
[009h 0009  1]                     Checksum : 19
[00Ah 0010  6]                       Oem ID : "INTEL "
[010h 0016  8]                 Oem Table ID : "SNB "
[018h 0024  4]                 Oem Revision : 00000001
[01Ch 0028  4]              Asl Compiler ID : "INTL"
[020h 0032  4]        Asl Compiler Revision : 00000001

[024h 0036  1]           Host Address Width : 23
[025h 0037  1]                        Flags : 01

[030h 0048  2]                Subtable Type : 0000 <Hardware Unit Definition>
[032h 0050  2]                       Length : 0018
[034h 0052  1]                        Flags : 00
[035h 0053  1]                     Reserved : 00
[036h 0054  2]           PCI Segment Number : 0000
[038h 0056  8]        Register Base Address : 00000000FED90000

[040h 0064  1]      Device Scope Entry Type : 01
[041h 0065  1]                 Entry Length : 08
[042h 0066  2]                     Reserved : 0000
[044h 0068  1]               Enumeration ID : 00
[045h 0069  1]               PCI Bus Number : 00
[046h 0070  2]                     PCI Path : [02, 00]

[048h 0072  2]                Subtable Type : 0000 <Hardware Unit Definition>
[04Ah 0074  2]                       Length : 0020
[04Ch 0076  1]                        Flags : 01
[04Dh 0077  1]                     Reserved : 00
[04Eh 0078  2]           PCI Segment Number : 0000
[050h 0080  8]        Register Base Address : 00000000FED91000

[058h 0088  1]      Device Scope Entry Type : 03
[059h 0089  1]                 Entry Length : 08
[05Ah 0090  2]                     Reserved : 0000
[05Ch 0092  1]               Enumeration ID : 02
[05Dh 0093  1]               PCI Bus Number : F0
[05Eh 0094  2]                     PCI Path : [1F, 00]

[060h 0096  1]      Device Scope Entry Type : 04
[061h 0097  1]                 Entry Length : 08
[062h 0098  2]                     Reserved : 0000
[064h 0100  1]               Enumeration ID : 00
[065h 0101  1]               PCI Bus Number : F0
[066h 0102  2]                     PCI Path : [0F, 00]

[068h 0104  2]                Subtable Type : 0001 <Reserved Memory Region>
[06Ah 0106  2]                       Length : 0030
[06Ch 0108  2]                     Reserved : 0000
[06Eh 0110  2]           PCI Segment Number : 0000
[070h 0112  8]                 Base Address : 00000000AD86C000
[078h 0120  8]          End Address (limit) : 00000000AD896FFF

[080h 0128  1]      Device Scope Entry Type : 01
[081h 0129  1]                 Entry Length : 08
[082h 0130  2]                     Reserved : 0000
[084h 0132  1]               Enumeration ID : 00
[085h 0133  1]               PCI Bus Number : 00
[086h 0134  2]                     PCI Path : [1D, 00]

[088h 0136  1]      Device Scope Entry Type : 01
[089h 0137  1]                 Entry Length : 08
[08Ah 0138  2]                     Reserved : 0000
[08Ch 0140  1]               Enumeration ID : 00
[08Dh 0141  1]               PCI Bus Number : 00
[08Eh 0142  2]                     PCI Path : [1A, 00]

[090h 0144  1]      Device Scope Entry Type : 01
[091h 0145  1]                 Entry Length : 08
[092h 0146  2]                     Reserved : 0000
[094h 0148  1]               Enumeration ID : 00
[095h 0149  1]               PCI Bus Number : 00
[096h 0150  2]                     PCI Path : [14, 00]

[098h 0152  2]                Subtable Type : 0001 <Reserved Memory Region>
[09Ah 0154  2]                       Length : 0020
[09Ch 0156  2]                     Reserved : 0000
[09Eh 0158  2]           PCI Segment Number : 0000
[0A0h 0160  8]                 Base Address : 00000000AF800000
[0A8h 0168  8]          End Address (limit) : 00000000BF9FFFFF

[0B0h 0176  1]      Device Scope Entry Type : 01
[0B1h 0177  1]                 Entry Length : 08
[0B2h 0178  2]                     Reserved : 0000
[0B4h 0180  1]               Enumeration ID : 00
[0B5h 0181  1]               PCI Bus Number : 00
[0B6h 0182  2]                     PCI Path : [02, 00]

Raw Table Data

  0000: 44 4D 41 52 B8 00 00 00 01 19 49 4E 54 45 4C 20  DMAR......INTEL 
  0010: 53 4E 42 20 00 00 00 00 01 00 00 00 49 4E 54 4C  SNB ........INTL
  0020: 01 00 00 00 23 01 00 00 00 00 00 00 00 00 00 00  ....#...........
  0030: 00 00 18 00 00 00 00 00 00 00 D9 FE 00 00 00 00  ................
  0040: 01 08 00 00 00 00 02 00 00 00 20 00 01 00 00 00  .......... .....
  0050: 00 10 D9 FE 00 00 00 00 03 08 00 00 02 F0 1F 00  ................
  0060: 04 08 00 00 00 F0 0F 00 01 00 30 00 00 00 00 00  ..........0.....
  0070: 00 C0 86 AD 00 00 00 00 FF 6F 89 AD 00 00 00 00  .........o......
  0080: 01 08 00 00 00 00 1D 00 01 08 00 00 00 00 1A 00  ................
  0090: 01 08 00 00 00 00 14 00 01 00 20 00 00 00 00 00  .......... .....
  00A0: 00 00 80 AF 00 00 00 00 FF FF 9F BF 00 00 00 00  ................
  00B0: 01 08 00 00 00 00 02 00                          ........

Re: [Qemu-devel] VFIO VGA test branches

[Knut Omang]

On Mon, 2013-05-20 at 23:11 +0200, Knut Omang wrote:
> On Sun, 2013-05-19 at 22:15 -0600, Alex Williamson wrote:
> > On Sun, 2013-05-19 at 17:35 +0200, Knut Omang wrote:
> > > On Mon, 2013-05-13 at 16:23 -0600, Alex Williamson wrote:
> > > > On Mon, 2013-05-13 at 22:55 +0200, Knut Omang wrote:
> > > > > Hi all,
> > > > > 
> > > > > Perfect timing from my perspective, thanks Alex!
> > > > > 
> > > > > I spent the better part of the weekend testing your branches on a new system 
> > > > > I just put together for this purpose, results below..
> > > > > 
> > > > > On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote:
> > > > > ...
> > > > > > git://github.com/awilliam/linux-vfio.git vfio-vga-reset
> > > > > > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset
> > > > > 
> > > > > System setup: 
> > > > > 
> > > > > - Fedora 18 on
> > > > > - Gigabyte Z77X-UD5H motherboard
> > > > > - Intel Core i7 3770 (Ivy bridge w/integrated graphics)
> > > > > - 2 discrete graphics cards:
> > > > > 
> > > > > lspci | egrep 'VGA|Audio'
> > > > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
> > > > > 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
> > > > > 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
> > > > > 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series]
> > > > > 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series]
> > > > > 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > > 
> > > > > Short summary:
> > > > > 
> > > > > - Once I got past a few time consuming obstacles explained below
> > > > >    - the graphics part of the graphics/hdmi audio passthrough seems to work perfect
> > > > >      on both discrete graphics cards 
> > > > >      (though so far only one at at time and with some minor issues, see below)
> > > > >    - no success with the hdmi audio yet (ideas for further investigation appreciated!)
> > > > 
> > > > I've had hdmi audio working with an HD7850, but only in Windows (7) and
> > > > it was using legacy interrupts for some reason instead of MSI.  I wonder
> > > > if Liux guests might work with snd_hda_intel.enable_msi=0.  I'm not sure
> > > > what's wrong with MSI, but it seems to be new with the PCI bus reset
> > > > support.
> > > 
> > > In my first tries, Windows were just using a generic
> > > VGA driver, which still seems to work perfect with reboots and everything 
> > > and in full screen resolution (1920x1200).
> > > However after installing the Catalyst AMD driver stack, upon boot
> > > Windows 7 now consequently get a BSOD from the graphics driver
> > > with the message:
> > > 
> > > "Attempt to reset the display driver and recover from timeout failed"
> > > - a picture of the BSOD screen attached.
> > 
> > I've seen that BSOD before, but I don't know how to reproduce it.  It
> > seems like I haven't seen it with the PCI bus reset code.  I'm running
> > version 13.1 of the catalyst driver, you?
> 
> I first tried with the install CD that came with the card - v.13-045
> then upgraded to the latest from AMD, catalyst v.13.4 which appears to
> be driver v.12.104 - similar behaviour for both. This was with a plain
> Windows 7 install from my SP1 DVD. 
> 
> With most recommended windows updates and the latest catalyst driver,
> the BSOD is gone but instead I see the initial VGA boot screen and the
> windows logo, then syncs but no display and then reboot into recovery
> mode. (If I try all updates, Windows seems never to be able to recover
> from the last reboot)
> 
> I have tried without kvm and also with vnc or spice graphics in addition
> but in those cases it seems Windows is not able to allocate MMIO
> resources for both adapters so I haven't been able to test the catalyst
> driver as a secondary windows display.
> 
> > > I attach the corresponding vfio log where I added some timing code to
> > > make it easier to see when the BSOD happens (with 2 seconds of silence
> > > in the log before the VM reboots, I believe this is at 09:28:32-34 in
> > > the log.
> > 
> > Yep, looks like that's where windows starts the BSOD.
> > 
> > > Similar behaviour both just after reboot/power cycle of the host and
> > > subsequent VM boot attempts.
> > > 
> > > This is still with the HD7700 as passed through device, but after a
> > > motherboard firmware upgrade (to F14) which did not seem to affect the
> > > observed behaviour on Windows prior to Catalyst install or with Linux
> > > guest, neither did it fix the bug in selecting primary devices as I 
> > > was hoping for.
> > > 
> > > Let me know if you have ideas for further debugging this,
> > 
> > I don't have any great ideas since I don't know how to reproduce the
> > timeout.  Double/triple check that you're using the correct
> > vfio-vga-reset branches in both qemu and kernel
> > 
> > # grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
> > # grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c
> 
> [Matches in both..]
> I do believe I have used the right branches all along.
> 
> > I didn't see anything telling in your DMAR either.  The system seems to
> > have just one DRHD that includes everything, so I'm not sure why you saw
> > any behavior change from igfx_off.  Thanks,
> 
> After the firmware upgrade, I tried again with the integrated graphics
> enabled, this time with more success - I am now able to get a GUI fedora
> console on the integrated graphics, but see some colorful artifacts
> there during the VGA startup on one of the Radeon cards, which goes away
> with a toggle to another console and back.
> 
> Seems I have slightly mislead you with the DMAR table - sorry about that
> - the table I posted was with the igfx disabled, with the igfx enabled I
> see one more hardware unit dedicated to the igfx if I am able to
> interpret it right (attached)

I noticed this warning in the host log - I suppose it is unrelated but
thought I'd mention it just in case there is some side effect I do not
understand here:

[    0.538124] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.538619] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.538676] ------------[ cut here ]------------
[    0.538681] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x58/0x80()
[    0.538683] Hardware name: To be filled by O.E.M.
[    0.538685] Modules linked in:
[    0.538687] Pid: 1, comm: swapper/0 Not tainted 3.9.0+ #1
[    0.538689] Call Trace:
[    0.538694]  [<ffffffff8105ed2f>] warn_slowpath_common+0x7f/0xc0
[    0.538697]  [<ffffffff8105ed8a>] warn_slowpath_null+0x1a/0x20
[    0.538699]  [<ffffffff8132dc28>] pci_find_upstream_pcie_bridge+0x58/0x80
[    0.538703]  [<ffffffff8152e26b>] intel_iommu_add_device+0x4b/0x1f0
[    0.538706]  [<ffffffff81525b30>] ? bus_set_iommu+0x60/0x60
[    0.538708]  [<ffffffff81525b63>] add_iommu_group+0x33/0x60
[    0.538712]  [<ffffffff813f38fd>] bus_for_each_dev+0x5d/0xa0
[    0.538714]  [<ffffffff81525b1b>] bus_set_iommu+0x4b/0x60
[    0.538718]  [<ffffffff81d47d61>] intel_iommu_init+0xa72/0xb9a
[    0.538722]  [<ffffffff81d0db94>] ? memblock_find_dma_reserve+0x13d/0x13d
[    0.538724]  [<ffffffff81d0dba7>] pci_iommu_init+0x13/0x3e
[    0.538727]  [<ffffffff8100215a>] do_one_initcall+0x12a/0x180
[    0.538730]  [<ffffffff81d0603b>] kernel_init_freeable+0x150/0x1df
[    0.538732]  [<ffffffff81d0588d>] ? do_early_param+0x8c/0x8c
[    0.538736]  [<ffffffff81646580>] ? rest_init+0x80/0x80
[    0.538738]  [<ffffffff8164658e>] kernel_init+0xe/0xf0
[    0.538742]  [<ffffffff8166af6c>] ret_from_fork+0x7c/0xb0
[    0.538744]  [<ffffffff81646580>] ? rest_init+0x80/0x80
[    0.538749] ---[ end trace f4e8b5168095f9c1 ]---


> Both the HD7700 and the HD6450 behave very similar and both still starts
> and displays Windows fine if I disable the Catalyst driver.
> 
> Knut
> 
> > Alex
> > 
> > > > > - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA
> > > > >   with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, 
> > > > > 
> > > > > Details:
> > > > > 
> > > > > - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon
> > > > >    (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display
> > > > >    - this caused the system to freeze (with color artifacts on the console)
> > > > > 
> > > > > - In my naivity and because of the "i" in ifgx I tried both with 
> > > > >   'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' 
> > > > >   and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected 
> > > > >   that iommu support was turned off for **all** graphics cards with igfx_off
> > > > 
> > > > I'm not sure why this is, looks like the code only tries to turn it off
> > > > when only graphics is under the remapping device.  We'd probably need to
> > > > see the DMAR to know more (/sys/firmware/acpi/tables/DMAR).
> > > > 
> > > > > - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on':
> > > > > 
> > > > > - iommu groups:
> > > > > 
> > > > > ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
> > > > > total 0
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
> > > > > lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
> > > > > 
> > > > > - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group:
> > > > 
> > > > Ick.  Intel has been pretty good about advertising ACS support on their
> > > > root ports.  I wonder if this is an oversight or if they are actually
> > > > not isolated from each other.
> > > > 
> > > > > # lspci -n
> > > > > ...
> > > > > 01:00.0 0300: 1002:683f
> > > > > 01:00.1 0403: 1002:aab0
> > > > > 02:00.0 0300: 1002:6779
> > > > > 02:00.1 0403: 1002:aa98
> > > > > ...
> > > > > 
> > > > > modprobe vfio_pci
> > > > > echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
> > > > > echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind
> > > > > echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > > echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > > echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > > echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > > > 
> > > > > # lsusb 
> > > > > ...
> > > > > Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard
> > > > > Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse]
> > > > > ...
> > > > > 
> > > > > - I also applied your suggested patch to the quirk function in VFIO (see below)
> > > > > 
> > > > > - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, 
> > > > >   notice the cd and disk device descriptions and the bus parameter - I struggled a while with that 
> > > > >   until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!)
> > > > > 
> > > > > 
> > > > > qemu-kvm -M q35 \
> > > > >   -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \
> > > > >   -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \
> > > > >   -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \
> > > > >   -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \
> > > > >   -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \
> > > > >   -drive file=/dev/sr0,index=2,media=cdrom,id=cd \
> > > > >   -drive file=ivm03.img,index=0,media=disk,id=ivm03 \
> > > > >   -device ide-drive,drive=ivm03,bus=ide.0 \
> > > > >   -device ide-cd,drive=cd,bus=ide.1 \
> > > > >   -net nic,vlan=0,model=virtio -net tap,vlan=0 \
> > > > >   -enable-kvm \
> > > > >   -device usb-host,hostbus=1,hostaddr=8 \
> > > > >   -device usb-host,hostbus=1,hostaddr=4
> > > > > 
> > > > > - Both the graphics card seemshould really support ACS on s to have a rom but only the HD6450 let itself to "scraping". 
> > > > 
> > > > Did you try scraping the HD6450 while the HD7700 was the boot VGA and
> > > > vica versa?  The boot VGA ROM is handled in a special way and what you
> > > > really get is the shadow copy, which isn't what we want.
> > > > 
> > > > > Anyway, supplying it to vfio did not seem to make any difference.
> > > > > 
> > > > > find /sys -name rom
> > > > > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom
> > > > > /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom
> > > > > ...
> > > > > 
> > > > > Some observations and remaining unresolved issues:
> > > > > 
> > > > > - VFIO patch:
> > > > >   Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com
> > > > >   reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to
> > > > >   vfio_ati_3c3_quirk_read and repeated logs: 
> > > > >     vfio: vfio_vga_read(0x3c3, 1) = 0x0
> > > > >   I patched up accordingly with 
> > > > > 
> > > > > 
> > > > > diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> > > > > index da0e5f9..a361d06 100644
> > > > > --- a/hw/misc/vfio.c
> > > > > +++ b/hw/misc/vfio.c
> > > > > @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque,
> > > > >      uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI],
> > > > >                                    addr + quirk->data.base_offset, size);
> > > > >  
> > > > > -    if (data == quirk->data.address_match) {
> > > > > +    if (1 || data == quirk->data.address_match) {
> > > > >          data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size);
> > > > >          DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data);
> > > > >      }
> > > > > 
> > > > > 
> > > > >   This of course did not help much until I actually got the iommu 
> > > > >   enabled for the radeons (similar "repeated patters" as deniv reported)
> > > > >   but what I have observed after I got it working is that if 
> > > > >   I disable the patch above, things are not that well: the Fedora VM 
> > > > >   comes up with VGA and the Fedora boot screen, then goes blank when 
> > > > >   switching to X.
> > > > 
> > > > Hmm, I think we'd probably have better luck making that unconditional
> > > > until we have reason to do otherwise.
> > > > 
> > > > > - The fact that the iommu group now extends across all my available graphics 
> > > > >   devices now makes it difficult to  get the radeon (or catalyst) driver use to 
> > > > >   the other card since the vfio_pci driver needs to hold it.
> > > > >   Not a complete showstopper since the vesa driver comes up with 1024x768..
> > > > >   Might it be a good idea to have an override option (exception list or similar?) 
> > > > >   to allow the vfio_pci to be less restrictive about owning the whole group 
> > > > >    - allow functionality over security in such case? This of course is further complicated
> > > > >   by the need for graphics drivers to be disabled/enabled already at the kernel prompt..
> > > > 
> > > > We have a quirk in the kernel that enables us to witelist devices, but
> > > > yes, there is no flexibility in this w/o modifying the code and
> > > > rebuilding.  (see drivers/pci/quirks.c:pci_dev_acs_enabled and follow
> > > > the example above w/ pci_dev_dma_source - function can just return 1)
> > > > 
> > > > > - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, 
> > > > >   The BIOS offers (undocumented) a full range of selections of which PCIe 
> > > > >   (or PCIe 1x) graphics card to use as primary, but any other selection 
> > > > >   than the first PCIe 16x slot has no effect and the motherboard reverts 
> > > > >   to the first slot, so to be able to test both cards, I had to put the card under test
> > > > >   into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible 
> > > > >   fixes for this in newer BIOSes.
> > > > > 
> > > > > - The ultimate goal is to try to consolidate some older Windows desktops as "seats" 
> > > > >   on the new system, using the discrete graphics with HDMI/Displayport audio. 
> > > > >   With the HD7700 moved to the second PCIe slot I tested both Windows and 
> > > > >   Linux guests to try to get some sound through the HDMI audio device. 
> > > > >   Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), 
> > > > >   the system settings -> multimedia dialogue never opens up which seems to indicate that 
> > > > >   PulseAudio has problems communicating with the passed through device (?), 
> > > > >   any hints/pointers here appreciated. From the vfio log it seems at least
> > > > >   config space is accessed ok.
> > > > > 
> > > > > - There also seems to be issues with radeon and intel_iommu=on - if I try 
> > > > >   to enable modesetting and normal X support for the radeon cards, X fails to start.
> > > > > 
> > > > > - It would be nice if the integrated graphics could be used as the host primary display - 
> > > > >   I would be happy if someone has any hints as to if/how the ifgx_off option 
> > > > >   could be extended/modified to only affect iommu operation on selected device(s),
> > > > >   if at all possible..
> > > > 
> > > > Let's see what we can discover from your DMAR.  Also send along sudo
> > > > lspci -vvv.  Thanks,
> > > > 
> > > > Alex
> > > > 
> > > 
> > > 
> > 
> > 
> > 
> 

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Tue, 2013-05-28 at 07:33 +0200, Knut Omang wrote:
> 
> I noticed this warning in the host log - I suppose it is unrelated but
> thought I'd mention it just in case there is some side effect I do not
> understand here:
> 
> [    0.538124] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
> [    0.538619] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
> [    0.538676] ------------[ cut here ]------------
> [    0.538681] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x58/0x80()
> [    0.538683] Hardware name: To be filled by O.E.M.
> [    0.538685] Modules linked in:
> [    0.538687] Pid: 1, comm: swapper/0 Not tainted 3.9.0+ #1
> [    0.538689] Call Trace:
> [    0.538694]  [<ffffffff8105ed2f>] warn_slowpath_common+0x7f/0xc0
> [    0.538697]  [<ffffffff8105ed8a>] warn_slowpath_null+0x1a/0x20
> [    0.538699]  [<ffffffff8132dc28>] pci_find_upstream_pcie_bridge+0x58/0x80
> [    0.538703]  [<ffffffff8152e26b>] intel_iommu_add_device+0x4b/0x1f0
> [    0.538706]  [<ffffffff81525b30>] ? bus_set_iommu+0x60/0x60
> [    0.538708]  [<ffffffff81525b63>] add_iommu_group+0x33/0x60
> [    0.538712]  [<ffffffff813f38fd>] bus_for_each_dev+0x5d/0xa0
> [    0.538714]  [<ffffffff81525b1b>] bus_set_iommu+0x4b/0x60
> [    0.538718]  [<ffffffff81d47d61>] intel_iommu_init+0xa72/0xb9a
> [    0.538722]  [<ffffffff81d0db94>] ? memblock_find_dma_reserve+0x13d/0x13d
> [    0.538724]  [<ffffffff81d0dba7>] pci_iommu_init+0x13/0x3e
> [    0.538727]  [<ffffffff8100215a>] do_one_initcall+0x12a/0x180
> [    0.538730]  [<ffffffff81d0603b>] kernel_init_freeable+0x150/0x1df
> [    0.538732]  [<ffffffff81d0588d>] ? do_early_param+0x8c/0x8c
> [    0.538736]  [<ffffffff81646580>] ? rest_init+0x80/0x80
> [    0.538738]  [<ffffffff8164658e>] kernel_init+0xe/0xf0
> [    0.538742]  [<ffffffff8166af6c>] ret_from_fork+0x7c/0xb0
> [    0.538744]  [<ffffffff81646580>] ? rest_init+0x80/0x80
> [    0.538749] ---[ end trace f4e8b5168095f9c1 ]---


There's a bug for this:

https://bugzilla.kernel.org/show_bug.cgi?id=44881


Chances are your system includes one of the non-compliant PCIe-to-PCI
bridges that doesn't include a PCIe capability.  So long as you're not
assigning anything behind that bridge, it shouldn't matter, but I think
we'll setup the wrong grouping and use the wrong source ID for devices
behind it.  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Justin Gottula]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

Hi,

The kernel won't compile with CONFIG_HOTPLUG_PCI=m:

drivers/pci/hotplug/pci_hotplug_core.c:548:5: error: redefinition of
‘pci_hp_reset_slot’
 int pci_hp_reset_slot(struct hotplug_slot *hotplug, int probe)
     ^
In file included from drivers/pci/hotplug/pci_hotplug_core.c:41:0:
include/linux/pci_hotplug.h:141:19: note: previous definition of
‘pci_hp_reset_slot’ was here
 static inline int pci_hp_reset_slot(struct hotplug_slot *slot, int probe)
                   ^
make[3]: *** [drivers/pci/hotplug/pci_hotplug_core.o] Error 1


Justin

[-- Attachment #2: Type: text/html, Size: 703 bytes --]

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Fri, 2013-05-10 at 14:31 -0700, Justin Gottula wrote:
> Hi,
> 
> The kernel won't compile with CONFIG_HOTPLUG_PCI=m:
> 
> drivers/pci/hotplug/pci_hotplug_core.c:548:5: error: redefinition of
> ‘pci_hp_reset_slot’
>  int pci_hp_reset_slot(struct hotplug_slot *hotplug, int probe)
>      ^
> In file included from drivers/pci/hotplug/pci_hotplug_core.c:41:0:
> include/linux/pci_hotplug.h:141:19: note: previous definition of
> ‘pci_hp_reset_slot’ was here
>  static inline int pci_hp_reset_slot(struct hotplug_slot *slot, int probe)
>                    ^
> make[3]: *** [drivers/pci/hotplug/pci_hotplug_core.o] Error 1

Oops, thanks.  Hopefully you can use =y or off for now.  These trees
aren't yet ready for upstream.  Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Justin Gottula]

[-- Attachment #1.1: Type: text/plain, Size: 5509 bytes --]

Hi Alex,

VGA passthrough is working great here, with the exception of device reset.

In short, everything works the first time the guest runs. But the second
time I start the guest, before anything comes on the screen, the host
grinds to a halt and freezes (gradually, until after a few moments, magic
sysrq doesn't even work). No 'reduced performance' here, just a completely
frozen system.

Suspending and waking the host in between guest runs, while inconvenient,
completely avoids the problem. No more freezes and full graphics
performance in the guest. So my guess is that the PCI reset from software
just isn't happening for some reason.

The passthru devices are being assigned to pci-stub and vfio-pci as they
should be. Secondary passthrough (-vga cirrus or -vga std) doesn't change
much: things still work, and reset still doesn't. One small difference is
that the freeze on second boot is delayed until Windows initializes the
secondary graphics adapter, since the device isn't touched by the BIOS
prior to that.

Overriding the video BIOS doesn't seem to change anything. Passing through
just the VGA device (excluding the HDMI audio device) doesn't seem to make
much of a difference either.

- hardware
ASUS M5A99X EVO (AMD 990X/SB950; the IVRS is broken and overriden)
AMD Radeon HD 5750 (for the host)
AMD Radeon HD 7870 (for passthru)

- software: host
linux (Joerg Roedel's iommu tree) with linux-vfio merged in
qemu (latest git with vfio)
seabios (latest git)

- software: guest
windows 8
amd catalyst 13.5b2
virtio drivers

- lspci (abbreviated)
00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory
Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (PCI express gpp port B)
00:03.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
bridge (PCI express gpp port C)
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller
(rev 42)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
Juniper [Radeon HD 5700 Series]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Juniper HDMI
Audio [Radeon HD 5700 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
Pitcairn [Radeon HD 7800]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape
Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

- lspci -n (abbreviated)
01:00.0 0300: 1002:68be
01:00.1 0403: 1002:aa58
02:00.0 0300: 1002:6818
02:00.1 0403: 1002:aab0

- lspci -t (abbreviated)
-[0000:00]-+-00.0
           +-00.2
           +-02.0-[01]--+-00.0
           |            \-00.1
           +-03.0-[02]--+-00.0
                        \-00.1

- kernel cmdline
/vmlinuz-linux-iommu initrd=/initramfs-linux-iommu.img
root=/dev/CorsairVG/ArchLinux rootflags=subvol=root
systemd.unit=graphical.target debug nomodeset vga=804 acpi.debug_level=0x2
acpi.debug_layer=0xFFFFFFFF amd_iommu_dump
vfio_iommu_type1.allow_unsafe_interrupts=1 ivrs_ioapic[9]=00:14.0
ivrs_ioapic[10]=00:00.1

- qemu options
qemu-system-x86_64 -enable-kvm -name Windows8 \
 -M q35 -nodefconfig -readconfig /pool/KVM/Windows8/q35-chipset.cfg \
 -m 4096 -balloon none \
 -rtc base=localtime \
 -cpu host -smp 8,sockets=1,cores=4,threads=2 \
 -bios /usr/share/qemu/bios.bin \
 -vga none \
 -drive
if=virtio,format=raw,discard=on,cache=none,file=/dev/CorsairVG/Windows \
 -drive if=virtio,format=raw,file=/pool/KVM/Windows8/WinData.ntfs.img \
 -drive id=cdrom,media=cdrom,format=raw,file=/dev/null \
 -device ide-cd,bus=ide.0,drive=cdrom \
 -boot order=dc,menu=on \
 -net nic,model=virtio,macaddr=00:55:aa:00:00:01 -net bridge,br=vm_br \
 -soundhw hda \
 -usbdevice tablet \
 -device
vfio-pci,host=02:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on
\
 -device vfio-pci,host=02:00.1,bus=ich9-pcie-port-1,addr=0.1

- dmesg | egrep -i '(iommu|ioapic|pci-stub|vfio)' | grep -vi 'command line'
[    0.000000] ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI
0-23
[    0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI
24-55
[    0.223213] AMD-Vi:   DEV_SPECIAL(IOAPIC[0])         devid: 00:14.0
[    0.223219] AMD-Vi:   DEV_SPECIAL(IOAPIC[255])               devid:
00:00.1
[    0.600863] ACPI: Using IOAPIC for interrupt routing
[    2.033551] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.191528] VFIO - User Level meta-driver version: 0.3
[    3.601002] pci-stub: add 1002:6818 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    3.607895] pci-stub 0000:02:00.0: claimed by stub
[    3.614688] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    3.621640] pci-stub 0000:02:00.1: claimed by stub
[ 5137.969990] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[ 5137.995842] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[ 5137.995849] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
[ 5166.727114] vfio-pci 0000:02:00.0: irq 93 for MSI/MSI-X
(this is from before the second boot attempt)

- last two lines from dmesg before the freeze (netcat'd to another box)
Clocksource tsc unstable (delta = -416526709 ns)
Switching to clocksource hpet

- output with DEBUG_VFIO: lots and lots, see attachment

If you need any more information, I'll be glad to provide it.

Justin

[-- Attachment #1.2: Type: text/html, Size: 5988 bytes --]

[-- Attachment #2: qemu-logs.tar.gz --]
[-- Type: application/x-gzip, Size: 294655 bytes --]

Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Fri, 2013-05-17 at 01:09 -0700, Justin Gottula wrote:
> Hi Alex,
> 
> VGA passthrough is working great here, with the exception of device reset.
> 
> In short, everything works the first time the guest runs. But the second
> time I start the guest, before anything comes on the screen, the host
> grinds to a halt and freezes (gradually, until after a few moments, magic
> sysrq doesn't even work). No 'reduced performance' here, just a completely
> frozen system.
> 
> Suspending and waking the host in between guest runs, while inconvenient,
> completely avoids the problem. No more freezes and full graphics
> performance in the guest. So my guess is that the PCI reset from software
> just isn't happening for some reason.
> 
> The passthru devices are being assigned to pci-stub and vfio-pci as they
> should be. Secondary passthrough (-vga cirrus or -vga std) doesn't change
> much: things still work, and reset still doesn't. One small difference is
> that the freeze on second boot is delayed until Windows initializes the
> secondary graphics adapter, since the device isn't touched by the BIOS
> prior to that.
> 
> Overriding the video BIOS doesn't seem to change anything. Passing through
> just the VGA device (excluding the HDMI audio device) doesn't seem to make
> much of a difference either.
> 
> - hardware
> ASUS M5A99X EVO (AMD 990X/SB950; the IVRS is broken and overriden)
> AMD Radeon HD 5750 (for the host)
> AMD Radeon HD 7870 (for passthru)
> 
> - software: host
> linux (Joerg Roedel's iommu tree) with linux-vfio merged in
> qemu (latest git with vfio)

Are you dependent on Joerg's tree for the IVRS fixup?  It would be
preferable to start with just my vfio-vga-reset branches before adding
more variables.  Also, be sure you're using the correct branch to get
the PCI bus reset code.  You can verify with something like:

grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c

I have seen timer messages from the host and they can cause the host to
become very unresponsive.  I'm not sure I've seen a full freeze though.
In one case I've also seen this with the PCI bus reset code where the
bus didn't return to a working state (the reason I haven't reposted the
PCI changes upstream yet).  When I saw this it was in a system where
resetting another slot works quite well, so when I get a chance to look
at it again, I'll probably start with seeing if the problem is unique to
the slot.  If you can manage to get the system to run lspci once it's in
this state you can tell if the bus is still in reset if the devices
report ref FF and get an unknown header type 7F for those devices.

> seabios (latest git)
> 
> - software: guest
> windows 8
> amd catalyst 13.5b2
> virtio drivers
> 
> - lspci (abbreviated)
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
> bridge (external gfx0 port B) (rev 02)
> 00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory
> Management Unit (IOMMU)
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
> bridge (PCI express gpp port B)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI
> bridge (PCI express gpp port C)
> 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller
> (rev 42)
> 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
> Juniper [Radeon HD 5700 Series]
> 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Juniper HDMI
> Audio [Radeon HD 5700 Series]
> 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI
> Pitcairn [Radeon HD 7800]
> 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape
> Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> - lspci -n (abbreviated)
> 01:00.0 0300: 1002:68be
> 01:00.1 0403: 1002:aa58
> 02:00.0 0300: 1002:6818
> 02:00.1 0403: 1002:aab0
> 
> - lspci -t (abbreviated)
> -[0000:00]-+-00.0
>            +-00.2
>            +-02.0-[01]--+-00.0
>            |            \-00.1
>            +-03.0-[02]--+-00.0
>                         \-00.1
> 
> - kernel cmdline
> /vmlinuz-linux-iommu initrd=/initramfs-linux-iommu.img
> root=/dev/CorsairVG/ArchLinux rootflags=subvol=root
> systemd.unit=graphical.target debug nomodeset vga=804 acpi.debug_level=0x2
> acpi.debug_layer=0xFFFFFFFF amd_iommu_dump
> vfio_iommu_type1.allow_unsafe_interrupts=1 ivrs_ioapic[9]=00:14.0
> ivrs_ioapic[10]=00:00.1
> 
> - qemu options
> qemu-system-x86_64 -enable-kvm -name Windows8 \
>  -M q35 -nodefconfig -readconfig /pool/KVM/Windows8/q35-chipset.cfg \
>  -m 4096 -balloon none \
>  -rtc base=localtime \
>  -cpu host -smp 8,sockets=1,cores=4,threads=2 \
>  -bios /usr/share/qemu/bios.bin \
>  -vga none \
>  -drive
> if=virtio,format=raw,discard=on,cache=none,file=/dev/CorsairVG/Windows \
>  -drive if=virtio,format=raw,file=/pool/KVM/Windows8/WinData.ntfs.img \
>  -drive id=cdrom,media=cdrom,format=raw,file=/dev/null \
>  -device ide-cd,bus=ide.0,drive=cdrom \
>  -boot order=dc,menu=on \
>  -net nic,model=virtio,macaddr=00:55:aa:00:00:01 -net bridge,br=vm_br \
>  -soundhw hda \
>  -usbdevice tablet \
>  -device
> vfio-pci,host=02:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on
> \
>  -device vfio-pci,host=02:00.1,bus=ich9-pcie-port-1,addr=0.1
> 
> - dmesg | egrep -i '(iommu|ioapic|pci-stub|vfio)' | grep -vi 'command line'
> [    0.000000] ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0])
> [    0.000000] IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI
> 0-23
> [    0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24])
> [    0.000000] IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI
> 24-55
> [    0.223213] AMD-Vi:   DEV_SPECIAL(IOAPIC[0])         devid: 00:14.0
> [    0.223219] AMD-Vi:   DEV_SPECIAL(IOAPIC[255])               devid:
> 00:00.1
> [    0.600863] ACPI: Using IOAPIC for interrupt routing
> [    2.033551] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
> [    2.191528] VFIO - User Level meta-driver version: 0.3
> [    3.601002] pci-stub: add 1002:6818 sub=FFFFFFFF:FFFFFFFF
> cls=00000000/00000000
> [    3.607895] pci-stub 0000:02:00.0: claimed by stub
> [    3.614688] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF
> cls=00000000/00000000
> [    3.621640] pci-stub 0000:02:00.1: claimed by stub

2:00.0/1 is added to pci-stub here, but used by vfio-pci below.  Is
pci-stub just temporary to keep radeon from binding to it?

> [ 5137.969990] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
> [ 5137.995842] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
> [ 5137.995849] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
> [ 5166.727114] vfio-pci 0000:02:00.0: irq 93 for MSI/MSI-X
> (this is from before the second boot attempt)
> 
> - last two lines from dmesg before the freeze (netcat'd to another box)
> Clocksource tsc unstable (delta = -416526709 ns)
> Switching to clocksource hpet
> 
> - output with DEBUG_VFIO: lots and lots, see attachment
> 
> If you need any more information, I'll be glad to provide it.

Thanks,

Alex

Re: [Qemu-devel] VFIO VGA test branches

[Justin Gottula]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On Sun, May 19, 2013 at 8:44 PM, Alex Williamson <alex.williamson@redhat.com
> wrote:

> Also, be sure you're using the correct branch to get
> the PCI bus reset code.  You can verify with something like:
>
> grep VFIO_DEVICE_PCI_BUS_RESET qemu.git/hw/misc/vfio.c
> grep VFIO_DEVICE_PCI_BUS_RESET linux.git/drivers/vfio/pci/vfio_pci.c
>

Oops, that's embarrassing. I'm pretty sure I pulled the wrong qemu branch.

With that goof eliminated, everything is working flawlessly: the host
doesn't freeze up anymore, and I no longer need to suspend between VM runs.
Very nice!

Thanks for the patches and the help,

Justin

[-- Attachment #2: Type: text/html, Size: 1004 bytes --]

Re: [Qemu-devel] VFIO VGA test branches

[Michael Nelson]

[-- Attachment #1: Type: text/plain, Size: 3107 bytes --]

Here is some more data for vfio-vga.

I have tried NVIDIA (MSI) GT610 and ATI (MSI) 7850 cards separately in
primary and secondary (to Cirrus) modes. The ATI can boot as primary in VGA
mode, but installing Catalyst causes the VM to reboot repeatedly on bootup.
The NVIDIA doesn't get past POST as primary. Neither card works as a
secondary with their drivers; they each complain that there aren't enough
resources available (code 12) in Windows device manager.

I have tried a few different KVM device settings, but feel like I am poking
in the dark at this point. What I am currently using is at the bottom of
the e-mail. I have URLs pointing to the various compressed logs since they
are fairly large. If there is any other data I can provide, please let me
know, I would really like to get this working :).

Machine:

Motherboard: Intel DQ67SW (BIOS: WQ6710H.86A.0052.2011.0520.1802 05/20/2011)
CPU: i5-2400
Kernel: 3.9 - current vfio-vga-reset branch
Qemu: 1.4.50 - current vfio-vga-reset branch
Guest: Windows Server 2008 R2 64-bit

Qemu has this patch added to vfio_ati_3c3_quirk_read():
+    if (1 || data == quirk->data.address_match) {

NVIDIA:

When it is primary, the NVIDIA card ends up with a QEMU fatal error within
a couple seconds of startup (the EIP seems to vary slightly). I have seen
this with 3 different NVidia cards (Quadro FX 580, GeForce 8800 GTS, and a
brand new GT610):

qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000000a0000

EAX=00009e48 EBX=00000000 ECX=0000b79d EDX=000003d4
ESI=000000e2 EDI=0000823a EBP=00004918 ESP=00008234
EIP=0009ffca EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
[…]

QEMU output when card is primary is at:
http://www.tnld.net/~mikenel/vga/qemu-nv-primary.txt.bz2

QEMU output when card is secondary is at:
http://www.tnld.net/~mikenel/vga/qemu-nv-secondary.txt.bz2

ATI:

The ATI card is able to boot Windows in VGA mode once I add "if (1)" to the
3C3 quirk in QEMU. Once I install the Catalyst drivers and reboot, the
machine reboots itself 10-15 seconds after the Windows boot loader
finishes. I am not sure if it's blue screening and I am not seeing it --
there is no output.

QEMU output when card is primary is at:
http://www.tnld.net/~mikenel/vga/qemu-ati-primary.txt.bz2

QEMU output when card is secondary is at:
http://www.tnld.net/~mikenel/vga/qemu-ati-secondary.txt.bz2

KVM command line:

~/qemu/usr/local/bin/qemu-system-x86_64 \
-nodefconfig -readconfig /root/q35-chipset.cfg \
-enable-kvm \
-M q35 \
-cpu host \
-L ~/seabios-out -L ~/qemu/usr/local/share/qemu \
-m 1024 \
-drive file=/dev/vgsys/vm.delete,if=virtio -boot order=cad,menu=on \
-vga cirrus \
-device vfio-pci,host=00:1d.0,addr=3.0,bus=pcie.0 \
-netdev tap,ifname=vm_rest_lan,id=hostnet0,vhost=on \
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:00:59:03,addr=05.00 \
-vnc 0.0.0.0:1 \
-device
vfio-pci,host=01:00.0,addr=0.0,bus=ich9-pcie-port-1,multifunction=on,x-vga=on
\
-device vfio-pci,host=01:00.1,addr=0.1,bus=ich9-pcie-port-1 \

Thanks,
-mike

[-- Attachment #2: Type: text/html, Size: 4248 bytes --]