Re: AMD KVM Pci Passthrough reports device busy

From: Andreas Hartmann <andihartmann@01019freenet.de>
To: Alex Williamson <alex.williamson@redhat.com>, Joerg.Roedel@amd.com
Cc: Chris Sanders <sanders.chris@gmail.com>, kvm@vger.kernel.org
Subject: Re: AMD KVM Pci Passthrough reports device busy
Date: Tue, 5 Jun 2012 12:39:32 +0200	[thread overview]
Message-ID: <201206051039.q55AdXda002609@mail.maya.org> (raw)
In-Reply-To: <1338867845.23475.168.camel@bling.home>

Hello Alex,

thanks for your efforts!

Maybe, you already know that I'm suffering the same problem :-( with a
GA-990XA-UD3 (990X chipset).

On Mon, 04 Jun 2012 21:44:05 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:
[...]
> I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> installed to reproduce.  The sad answer is that it's nearly impossible
> to assign PCI devices on these systems due to the aliasing of devices
> below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> assign).  If you boot with amd_iommu_dump, you'll see some output like
> this:
> 
> AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7

here:
AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7

This means according to your description, the following devices have to
be unbound here, too (0000:06:07.0 is the device I want to passthrough):

0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0

These are the following additional devices:

00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])

Among them is the sound device and the USB device - no good idea to
disable them on a desktop.

Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
I'm getting errors after rebooting the system (the filesystems haven't
been cleanly unmounted during shutdown).

Anyway, I would have tested it, but I'm getting a compile error while
compiling qemu. It complains about missing pci/header.h while
processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
should it come from?

[...]

> The downside is that VFIO is strict about
> multifunction devices supporting ACS to prevent peer-to-peer between
> domains, so will require all of the 14.x devices to be bound to pci-stub
> as well.  On my system, this includes an smbus controller, audio device,
> lpc controller, and usb device.  If AMD could confirm this device
> doesn't allow peer-to-peer between functions, we could relax this
> requirement a bit.

Please Joerg, could you take a look at this problem?

Thanks,
kind regards,
Andreas