AMD integrated graphics passthrough to KVM guest

AMD integrated graphics passthrough to KVM guest

[Gustav Sorenson]

Hello list,

I hope this is the right place for my question. If it is not, I'd be
very happy if you pointed me in the right direction.

I have an AMD A10 6800K APU with integrated graphics and I'm trying to
pass that graphics adapter to a KVM guest.
My mainboard is an ASRock FM2A75 Pro4 with latest firmware, so that it
(supposedly) supports IOMMU.

Since this is the first time I'm attempting this, I followed this
guide: https://bbs.archlinux.org/viewtopic.php?id=162768

I'm running Debian wheezy amd64, mainline kernel 3.10 with the
kernel-vfio-vga-reset patch mentioned in the link above, and qemu
1.5.1 with a corresponding patch (again, see link above.)
Kernel config options I set that I assume are relevant:
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

When I run

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
2,sockets=1,cores=2,threads=1 -bios /home/gustav/myroot/bios.bin -vga
none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=00:01.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
-device vfio-pci,host=00:01.1,bus=root.1,addr=00.1

I receive the message

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed
(Device or resource busy).  VGA may not work.

Indeed the display output doesn't change, i.e. the login promt (on the
console, there is no X installed) of the host doesn't vanish, and I
don't get SeaBIOS output from the KVM guest.

Next, I took a really old PCIe graphics card, installed it and set it
to be my primary graphics adapter in BIOS (or rather UEFI?), so that I
now saw POST and kernel messages via the dedicated GPU. Surprisingly
to me, I was able to pass-through the dedicated PCIe card to a KVM
guest, but still not the now supposedly unused integrated GPU, still
getting the same message when I tried.
After changing the primary graphics adapter to be the integrated one,
I still wasn't able to pass through the integrated GPU, but was able
to see SeaBIOS output if I passed the dedicated card to the guest.
However in this case, the graphics output was a bit strange in that
the space bewteen characters and lines was unusually large and I saw
only what seemed to be part of the whole screen output.

Since my integrated GPU is more powerful than the dedicated one, I'd
like to know what I can try to pass it to a guest, or what I can do to
find out why "Device or resource busy" is reported.
Please let me know if you need any more data. Also please note that
the computer is not in any kind of "production" use, so it wouldn't be
a problem if some operation resulted in an unusable operating system,
as long as there is no hardware damage.

Any input is very highly appreciated.
Thanks!

Re: AMD integrated graphics passthrough to KVM guest

[Alex Williamson]

On Fri, 2013-07-12 at 16:48 +0200, Gustav Sorenson wrote:
> Hello list,
> 
> I hope this is the right place for my question. If it is not, I'd be
> very happy if you pointed me in the right direction.
> 
> I have an AMD A10 6800K APU with integrated graphics and I'm trying to
> pass that graphics adapter to a KVM guest.
> My mainboard is an ASRock FM2A75 Pro4 with latest firmware, so that it
> (supposedly) supports IOMMU.
> 
> Since this is the first time I'm attempting this, I followed this
> guide: https://bbs.archlinux.org/viewtopic.php?id=162768
> 
> I'm running Debian wheezy amd64, mainline kernel 3.10 with the
> kernel-vfio-vga-reset patch mentioned in the link above, and qemu
> 1.5.1 with a corresponding patch (again, see link above.)
> Kernel config options I set that I assume are relevant:
> CONFIG_VFIO_IOMMU_TYPE1=y
> CONFIG_VFIO=y
> CONFIG_VFIO_PCI=y
> CONFIG_VFIO_PCI_VGA=y
> 
> When I run
> 
> qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
> 2,sockets=1,cores=2,threads=1 -bios /home/gustav/myroot/bios.bin -vga
> none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
> -device vfio-pci,host=00:01.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
> -device vfio-pci,host=00:01.1,bus=root.1,addr=00.1
> 
> I receive the message
> 
> qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed
> (Device or resource busy).  VGA may not work.

One problem with integrated graphics is that it's on the root complex.
That means a) there's no visible bridge from which to reset it and b)
there are lots of other devices on the bus, so we couldn't reset it even
if we knew how.

> Indeed the display output doesn't change, i.e. the login promt (on the
> console, there is no X installed) of the host doesn't vanish, and I
> don't get SeaBIOS output from the KVM guest.

VGA assignment should only even be attempted on non-primary displays at
this point (unless of course you want to work on adding support).

> Next, I took a really old PCIe graphics card, installed it and set it
> to be my primary graphics adapter in BIOS (or rather UEFI?), so that I
> now saw POST and kernel messages via the dedicated GPU. Surprisingly
> to me, I was able to pass-through the dedicated PCIe card to a KVM
> guest, but still not the now supposedly unused integrated GPU, still
> getting the same message when I tried.

What is the plugin graphics card?

> After changing the primary graphics adapter to be the integrated one,
> I still wasn't able to pass through the integrated GPU, but was able
> to see SeaBIOS output if I passed the dedicated card to the guest.
> However in this case, the graphics output was a bit strange in that
> the space bewteen characters and lines was unusually large and I saw
> only what seemed to be part of the whole screen output.
> 
> Since my integrated GPU is more powerful than the dedicated one, I'd
> like to know what I can try to pass it to a guest, or what I can do to
> find out why "Device or resource busy" is reported.
> Please let me know if you need any more data. Also please note that
> the computer is not in any kind of "production" use, so it wouldn't be
> a problem if some operation resulted in an unusable operating system,
> as long as there is no hardware damage.

GPUs have all sorts quirks, none of which are openly documented by the
vendors.  I've only tried plugin Radeon cards and it's entirely possible
that an integrated Radeon has a completely different set of quirks.
Also, if it's anything like Intel IGD, the GPU is actually spread across
multiple components of the chipset, so not as easy to pass through as a
dedicated GPU.  Please provide 'sudo lspci -vvv'.  Thanks,

Alex

Re: AMD integrated graphics passthrough to KVM guest

[Gustav Sorenson]

Hello Alex,

thank you very much for your answer.

> VGA assignment should only even be attempted on non-primary displays at
> this point (unless of course you want to work on adding support).

I assume when you say "non-primary," you refer to graphics adapters
that are not used to display POST and kernel messages?
Although I know C, I've never worked on hardware-related tasks, so I'm
afraid I won't be of much help in adding support, unless you think
it's a task that is suitable for a newcomer. In any case, I'd be happy
to help by testing suggestions by you or others.

>> Next, I took a really old PCIe graphics card, installed it and set it
>> to be my primary graphics adapter in BIOS (or rather UEFI?), so that I
>> now saw POST and kernel messages via the dedicated GPU. Surprisingly
>> to me, I was able to pass-through the dedicated PCIe card to a KVM
>> guest, but still not the now supposedly unused integrated GPU, still
>> getting the same message when I tried.
>
> What is the plugin graphics card?

I've pulled that card out of an old computer that I haven't bought.
It's manufactured by MSI, but other than that, I haven't found make or
model names. lspci says it's an "ATI RV515 [Radeon X1300]."

> Please provide 'sudo lspci -vvv'.  Thanks,
lspci -vvv with the graphics card built into the first PCIe slot:
http://pastebin.com/92Q6uFwa
lspci -vvv without the graphics card: http://pastebin.com/RAAsXxF3

Thanks!

Re: AMD integrated graphics passthrough to KVM guest

[Alex Williamson]

On Fri, 2013-07-12 at 18:13 +0200, Gustav Sorenson wrote:
> Hello Alex,
> 
> thank you very much for your answer.
> 
> > VGA assignment should only even be attempted on non-primary displays at
> > this point (unless of course you want to work on adding support).
> 
> I assume when you say "non-primary," you refer to graphics adapters
> that are not used to display POST and kernel messages?

Yes

> Although I know C, I've never worked on hardware-related tasks, so I'm
> afraid I won't be of much help in adding support, unless you think
> it's a task that is suitable for a newcomer. In any case, I'd be happy
> to help by testing suggestions by you or others.
> 
> >> Next, I took a really old PCIe graphics card, installed it and set it
> >> to be my primary graphics adapter in BIOS (or rather UEFI?), so that I
> >> now saw POST and kernel messages via the dedicated GPU. Surprisingly
> >> to me, I was able to pass-through the dedicated PCIe card to a KVM
> >> guest, but still not the now supposedly unused integrated GPU, still
> >> getting the same message when I tried.
> >
> > What is the plugin graphics card?
> 
> I've pulled that card out of an old computer that I haven't bought.
> It's manufactured by MSI, but other than that, I haven't found make or
> model names. lspci says it's an "ATI RV515 [Radeon X1300]."

Ok, I have an X500 and I've never been able to get it to work correctly
for passthrough.  It has additional quirks required beyond what either
my HD5450 or HD7850 require.  Also, these old devices have a weird
co-processor secondary function (see 01:00.1 in your lspci).  It
probably needs to be co-assigned, but I'm not really sure what it does.
FWIW, I don't think it's worth bothering to add support for anything
older than a Radeon HD5xxx (I'd say HD6xxx, but I happen to have the
HD5450 and it works, so...)

> > Please provide 'sudo lspci -vvv'.  Thanks,
> lspci -vvv with the graphics card built into the first PCIe slot:
> http://pastebin.com/92Q6uFwa
> lspci -vvv without the graphics card: http://pastebin.com/RAAsXxF3

One reason you're probably not getting any output is because the device
doesn't have a ROM (ie. there's no video BIOS to run to get seabios to
POST the device).  You'll either need to find a VBIOS for the device or
use it as a secondary graphics device in the guest.  In some cases you
can rip this out of the system ROM or ACPI tables.

I don't think that using the vfio-vga-reset branches is helping you
since it's a root complex device and we can't do a reset even if we knew
how and if you use it as a secondary device, you might not even need the
x-vga=on option for VFIO (ie. VFIO would handle this just like any
regular PCI device).  Thanks,

Alex

Re: AMD integrated graphics passthrough to KVM guest

[Gustav Sorenson]

Hello,

> Ok, I have an X500 and I've never been able to get it to work correctly
> for passthrough.  It has additional quirks required beyond what either
> my HD5450 or HD7850 require.  Also, these old devices have a weird
> co-processor secondary function (see 01:00.1 in your lspci).  It
> probably needs to be co-assigned, but I'm not really sure what it does.
> FWIW, I don't think it's worth bothering to add support for anything
> older than a Radeon HD5xxx (I'd say HD6xxx, but I happen to have the
> HD5450 and it works, so...)

You're probably right that for old cards the demand isn't high enough
to justify the time spent. I don't plan to use the X1300 actively
anyway.

My plan was to initially buy the hardware as I have now, get VGA
passthrough of the integrated GPU to work, and then add another more
powerful (HD6xxx most probably) PCIe device, so that I could have a
headless host and two 3D-accelerated VMs.
Now that this seemingly won't work, I'd be happy if at least I could
run the IGP on the host and pass the not-yet-bought dedicated GPU to a
guest.
So, in order to not again discover that my plans won't work out after
I already bought the hardware, could you please tell me whether you
have any problems with your graphics cards in guests? Xen users seem
to suffer degraded performance after their guest systems shut
down/reboot. Is this the case with KVM and VFIO as well? Any other
issues, or models I should/should not buy?
Of course I've done some research on my own, but the VGA
virtualization topic seems to be quite volatile right now, and I find
contradicting information.

> One reason you're probably not getting any output is because the device
> doesn't have a ROM (ie. there's no video BIOS to run to get seabios to
> POST the device).  You'll either need to find a VBIOS for the device or
> use it as a secondary graphics device in the guest.  In some cases you
> can rip this out of the system ROM or ACPI tables.
>
> I don't think that using the vfio-vga-reset branches is helping you
> since it's a root complex device and we can't do a reset even if we knew
> how and if you use it as a secondary device, you might not even need the
> x-vga=on option for VFIO (ie. VFIO would handle this just like any
> regular PCI device).  Thanks,

Are you implying that I should be able to use the integrated GPU (if
it's not the primary, I suppose) as a secondary for a guest? If so, do
you expect 3D acceleration to work?

Thank you very much for your help, it's highly appreciated.

Re: AMD integrated graphics passthrough to KVM guest

[Alex Williamson]

On Fri, 2013-07-12 at 20:18 +0200, Gustav Sorenson wrote:
> Hello,
> 
> > Ok, I have an X500 and I've never been able to get it to work correctly
> > for passthrough.  It has additional quirks required beyond what either
> > my HD5450 or HD7850 require.  Also, these old devices have a weird
> > co-processor secondary function (see 01:00.1 in your lspci).  It
> > probably needs to be co-assigned, but I'm not really sure what it does.
> > FWIW, I don't think it's worth bothering to add support for anything
> > older than a Radeon HD5xxx (I'd say HD6xxx, but I happen to have the
> > HD5450 and it works, so...)
> 
> You're probably right that for old cards the demand isn't high enough
> to justify the time spent. I don't plan to use the X1300 actively
> anyway.
> 
> My plan was to initially buy the hardware as I have now, get VGA
> passthrough of the integrated GPU to work, and then add another more
> powerful (HD6xxx most probably) PCIe device, so that I could have a
> headless host and two 3D-accelerated VMs.
> Now that this seemingly won't work, I'd be happy if at least I could
> run the IGP on the host and pass the not-yet-bought dedicated GPU to a
> guest.
> So, in order to not again discover that my plans won't work out after
> I already bought the hardware, could you please tell me whether you
> have any problems with your graphics cards in guests? Xen users seem
> to suffer degraded performance after their guest systems shut
> down/reboot. Is this the case with KVM and VFIO as well? Any other
> issues, or models I should/should not buy?
> Of course I've done some research on my own, but the VGA
> virtualization topic seems to be quite volatile right now, and I find
> contradicting information.

I'm reluctant to recommend anything because there are several unresolved
issues and none of the current consumer-class cards claim to work in
virtual environments.  We have issues with VGA routing, which can cause
problems with host drives which don't make use of the in-kernel VGA
arbiter (ex. vga console).  We need to continue to work on the kinks out
of PCI secondary bus resets.  All consumer-class cards have undocumented
backdoors to config space, which we're hoping are only accessed via CPU,
which we can quirk, and not via the GPU, which we can't.  On the Nvidia
side, the nouveau project has discovered many of these, so I expect many
Nvidia cards work and there are user reports to confirm so.  On the
Radeon side, I've discovered many of the quirks and have an HD7850 that
seems to work pretty well.  By well I mean, don't do anything that would
cause VGA access on the primary display and the PCI bus sometimes
doesn't come back if the guest is reset without a clean shutdown.

For performance, I expect there will be a hit vs running on baremetal,
but I don't know the extent.  I don't know if we suffer from the same
problem you mention for Xen, but I don't see why it would be the case
either (perhaps the link retrains at a lower speed?).


> > One reason you're probably not getting any output is because the device
> > doesn't have a ROM (ie. there's no video BIOS to run to get seabios to
> > POST the device).  You'll either need to find a VBIOS for the device or
> > use it as a secondary graphics device in the guest.  In some cases you
> > can rip this out of the system ROM or ACPI tables.
> >
> > I don't think that using the vfio-vga-reset branches is helping you
> > since it's a root complex device and we can't do a reset even if we knew
> > how and if you use it as a secondary device, you might not even need the
> > x-vga=on option for VFIO (ie. VFIO would handle this just like any
> > regular PCI device).  Thanks,
> 
> Are you implying that I should be able to use the integrated GPU (if
> it's not the primary, I suppose) as a secondary for a guest? If so, do
> you expect 3D acceleration to work?

Yes, with the correction of s/should/might/.  3D acceleration might also
work.  Generally if you assign a passthrough graphics as a secondary
device it will only work once you load the vendor driver stack (ex. AMD
Catalyst), at which point the emulated graphics is disabled and the
passthrough graphics becomes the primary for the guest.  Thanks,

Alex

Re: AMD integrated graphics passthrough to KVM guest

[Gustav Sorenson]

Hello again,

> For performance, I expect there will be a hit vs running on baremetal,
> but I don't know the extent.  I don't know if we suffer from the same
> problem you mention for Xen, but I don't see why it would be the case
> either (perhaps the link retrains at a lower speed?).

When using xen, at least some users experience that after a guest
which used VGA-passthrough powered down, the host has to be rebooted
as well before the guest is started again, since otherwise, the
graphics card is in some kind of half-initialized state (or so I read)
and won't do 3D acceleration.
Also, nVidia is said to be a lot more problematic than AMD in xen-land.

> Yes, with the correction of s/should/might/.  3D acceleration might also
> work.  Generally if you assign a passthrough graphics as a secondary
> device it will only work once you load the vendor driver stack (ex. AMD
> Catalyst), at which point the emulated graphics is disabled and the
> passthrough graphics becomes the primary for the guest.  Thanks,

I just tried with Windows 7. When the host's integrated CPU (set as
non-primary display) was used as secondary in the guest, driver
installation caused a bluescreen.
I also tried the equivalent with Linux Mint, but when the emulated
std-vga and the passthrough'd integrated GPU both are present, both
screens show garbage. When passing through the integrated GPU already
during Mint installation, the livecd makes the screens flicker, and
then I'm told X failed to detect screen resolutions. I haven't looked
further into this.

The latter I did with a command line like this:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
2,sockets=1,cores=2,threads=1 -bios /home/gustav/myroot/bios.bin -vga
std -vnc :0 -device
ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=00:01.0,bus=root.1,addr=00.0,multifunction=on
-device vfio-pci,host=00:01.1,bus=root.1,addr=00.1 -device
ahci,bus=pcie.0,id=ahci -drive
file=/home/gustav/linuxmint-15-xfce-dvd-64bit-rc.iso,id=isocd -device
ide-cd,bus=ahci.1,drive=isocd

While the guest was running, there were lines saying "vcpu0 ignored
rdmsr", "vcpu0 ignored wrmsr", "vcpu0 unimplemented HWCR wrmsr" and
vcpu0 unimplemented perfctr wrmsr" in dmesg.
Relevant dmesg section if the integrated GPU is set as the host
primary: http://pastebin.com/DjALp678
Relevant dmesg section if the dedicated GPU is set as the host
primary: http://pastebin.com/7UX3xQtM

I don't know whether this is relevant.

Thank you very much for your help!

Re: AMD integrated graphics passthrough to KVM guest

[Gustav Sorenson]

Hello Alex,

I'm thinking about buying a HD 7850 to go with my integrated graphics
card, so that I can pass the 7850 to a KVM guest and keep the
integrated GPU for the host. However, I'd like to use the proprietary
AMD drivers for the host to get reasonable 3D performance there (the
open source driver that ships with Mint 15 didn't get me far; maybe it
should and I've done something wrong?). I've read here:
https://bbs.archlinux.org/viewtopic.php?pid=1273412#p1273412
that this scenario caused problems. Is this still the case?
Which GPU do you use on the host / guest, and which driver combinations work?

Or should I just grab an nVidia instead?

Of course I'd be happy to hear reports from others as well. :)

Thank you so much for your help!

Re: AMD integrated graphics passthrough to KVM guest

[Alex Williamson]

On Sat, 2013-07-13 at 21:45 +0200, Gustav Sorenson wrote:
> Hello Alex,
> 
> I'm thinking about buying a HD 7850 to go with my integrated graphics
> card, so that I can pass the 7850 to a KVM guest and keep the
> integrated GPU for the host. However, I'd like to use the proprietary
> AMD drivers for the host to get reasonable 3D performance there (the
> open source driver that ships with Mint 15 didn't get me far; maybe it
> should and I've done something wrong?). I've read here:
> https://bbs.archlinux.org/viewtopic.php?pid=1273412#p1273412
> that this scenario caused problems. Is this still the case?

That's actually the first I've heard of that problem.  I don't see why
it wouldn't still be the case, fglrx is broken if that report is
accurate.

> Which GPU do you use on the host / guest, and which driver combinations work?

I'm not a good example, I don't do anything on the host VGA at all.

> Or should I just grab an nVidia instead?

If you plan to run fglrx in the host, it would certainly help to work
around the above issue, but I don't have much direct experience with
anything newer than an 8400GS.

> Of course I'd be happy to hear reports from others as well. :)

Yep, I'm astonished at the number of users trying to make this work.
I'm sure someone has some knowledge they can share.  Thanks,

Alex