Re: AMD KVM Pci Passthrough reports device busy

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 2012-06-05 at 17:17 +0200, Andreas Hartmann wrote:
> On Tue, 05 Jun 2012 08:27:05 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> 
> > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote:
> > > Hello Alex,
> > > 
> > > thanks for your efforts!
> > > 
> > > Maybe, you already know that I'm suffering the same problem :-( with a
> > > GA-990XA-UD3 (990X chipset).
> > 
> > My system is a GA-990FXA-UD3, so just a slightly different version of
> > the chipset.
> > 
> > > On Mon, 04 Jun 2012 21:44:05 -0600
> > > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > [...]
> > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> > > > installed to reproduce.  The sad answer is that it's nearly impossible
> > > > to assign PCI devices on these systems due to the aliasing of devices
> > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> > > > assign).  If you boot with amd_iommu_dump, you'll see some output like
> > > > this:
> > > > 
> > > > AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> > > > AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7
> > > 
> > > here:
> > > AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> > > AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> > > 
> > > 
> > > This means according to your description, the following devices have to
> > > be unbound here, too (0000:06:07.0 is the device I want to passthrough):
> > > 
> > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0
> > 
> > Yep.
> > 
> > > These are the following additional devices:
> > > 
> > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
> > 
> > My system shows this for lspci -n -s 14.
> > 
> > 00:14.0 0c05: 1002:4385 (rev 42)
> > 00:14.2 0403: 1002:4383 (rev 40)
> > 00:14.3 0601: 1002:439d (rev 40)
> > 00:14.4 0604: 1002:4384 (rev 40)
> > 00:14.5 0c03: 1002:4399
> 
> My system:
> 00:14.0 0c05: 1002:4385 (rev 42)
> 00:14.1 0101: 1002:439c (rev 40)
> 00:14.2 0403: 1002:4383 (rev 40)
> 00:14.3 0601: 1002:439d (rev 40)
> 00:14.4 0604: 1002:4384 (rev 40)
> 00:14.5 0c03: 1002:4399
> 
> You miss 00.14.1!? But the others are equal.

Yeah, my system doesn't expose the IDE, *shrug*

> > Can you confirm yours matches and fill in 14.1 so we have the
> > vendor:device IDs for these.
> 
> Yes.

Thanks

> > > Among them is the sound device and the USB device - no good idea to
> > > disable them on a desktop.
> > 
> > You can always blacklist drivers if the device is unique or unbind
> > devices in an rc.local type script.  I'm hoping that Joerg or someone
> > else from AMD can tell us this devices does not allow internal
> > peer-to-peer between functions.  Then we can add it to the device
> > specific ACS checks and you wouldn't need to worry about displacing all
> > the other 14.x functions.
> 
> This would be cool.
> 
> > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> > > I'm getting errors after rebooting the system (the filesystems haven't
> > > been cleanly unmounted during shutdown).
> > 
> > Did you have a disk off that controller?  My system doesn't expose
> > function 1, but is the same otherwise.
> 
> Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the
> reason is another?
> 
> > > Anyway, I would have tested it, but I'm getting a compile error while
> > > compiling qemu. It complains about missing pci/header.h while
> > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> > > should it come from?
> > 
> > It is from pciutils-devel for me.  I'll see what I'm getting out of
> > there and try to remove it.
> 
> Meanwhile I found it :-). Thanks for your hint!

Great.

> > > > The downside is that VFIO is strict about
> > > > multifunction devices supporting ACS to prevent peer-to-peer between
> > > > domains, so will require all of the 14.x devices to be bound to pci-stub
> > > > as well.  On my system, this includes an smbus controller, audio device,
> > > > lpc controller, and usb device.  If AMD could confirm this device
> > > > doesn't allow peer-to-peer between functions, we could relax this
> > > > requirement a bit.
> > > 
> > > Please Joerg, could you take a look at this problem?
> > 
> > Joerg, the question is whether the multifunction device above allows
> > peer-to-peer between functions that could bypass the iommu.  If not, we
> > can make it the first entry in device specific acs enabled function I
> > proposed:
> > 
> > https://lkml.org/lkml/2012/5/30/438
> > 
> > and it would greatly simplify assigning PCI devices on these systems
> > with VFIO.  Thanks,
> 
> I tried to run qemu-system-x86_64 but got this error on startup:
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
> **
> ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)
> 
> 
> I started qemu-system-x86_64 with this option among others 
> 
> -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5
> 
> after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.
> 
> 
> Could you please tell me, why the operation isn't permitted? I started
> qemu-system-x86_64 as root.

Check dmesg, it's probably complaining about no interrupt remapping
support.  If so:

# modprobe -r vfio_iommu_type1
# modprobe vfio_iommu_type1 allow_unsafe_interrupts=1

Then try again.  Thanks,

Alex