Re: AMD KVM Pci Passthrough reports device busy

[Andreas Hartmann <andihartmann@01019freenet.de>]

On Tue, 05 Jun 2012 08:27:05 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote:
> > Hello Alex,
> > 
> > thanks for your efforts!
> > 
> > Maybe, you already know that I'm suffering the same problem :-( with a
> > GA-990XA-UD3 (990X chipset).
> 
> My system is a GA-990FXA-UD3, so just a slightly different version of
> the chipset.
> 
> > On Mon, 04 Jun 2012 21:44:05 -0600
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> > [...]
> > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> > > installed to reproduce.  The sad answer is that it's nearly impossible
> > > to assign PCI devices on these systems due to the aliasing of devices
> > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> > > assign).  If you boot with amd_iommu_dump, you'll see some output like
> > > this:
> > > 
> > > AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> > > AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7
> > 
> > here:
> > AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> > AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> > 
> > 
> > This means according to your description, the following devices have to
> > be unbound here, too (0000:06:07.0 is the device I want to passthrough):
> > 
> > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0
> 
> Yep.
> 
> > These are the following additional devices:
> > 
> > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
> 
> My system shows this for lspci -n -s 14.
> 
> 00:14.0 0c05: 1002:4385 (rev 42)
> 00:14.2 0403: 1002:4383 (rev 40)
> 00:14.3 0601: 1002:439d (rev 40)
> 00:14.4 0604: 1002:4384 (rev 40)
> 00:14.5 0c03: 1002:4399

My system:
00:14.0 0c05: 1002:4385 (rev 42)
00:14.1 0101: 1002:439c (rev 40)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399

You miss 00.14.1!? But the others are equal.


> Can you confirm yours matches and fill in 14.1 so we have the
> vendor:device IDs for these.

Yes.

> > Among them is the sound device and the USB device - no good idea to
> > disable them on a desktop.
> 
> You can always blacklist drivers if the device is unique or unbind
> devices in an rc.local type script.  I'm hoping that Joerg or someone
> else from AMD can tell us this devices does not allow internal
> peer-to-peer between functions.  Then we can add it to the device
> specific ACS checks and you wouldn't need to worry about displacing all
> the other 14.x functions.

This would be cool.

> > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> > I'm getting errors after rebooting the system (the filesystems haven't
> > been cleanly unmounted during shutdown).
> 
> Did you have a disk off that controller?  My system doesn't expose
> function 1, but is the same otherwise.

Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the
reason is another?

> > Anyway, I would have tested it, but I'm getting a compile error while
> > compiling qemu. It complains about missing pci/header.h while
> > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> > should it come from?
> 
> It is from pciutils-devel for me.  I'll see what I'm getting out of
> there and try to remove it.

Meanwhile I found it :-). Thanks for your hint!

> > > The downside is that VFIO is strict about
> > > multifunction devices supporting ACS to prevent peer-to-peer between
> > > domains, so will require all of the 14.x devices to be bound to pci-stub
> > > as well.  On my system, this includes an smbus controller, audio device,
> > > lpc controller, and usb device.  If AMD could confirm this device
> > > doesn't allow peer-to-peer between functions, we could relax this
> > > requirement a bit.
> > 
> > Please Joerg, could you take a look at this problem?
> 
> Joerg, the question is whether the multifunction device above allows
> peer-to-peer between functions that could bypass the iommu.  If not, we
> can make it the first entry in device specific acs enabled function I
> proposed:
> 
> https://lkml.org/lkml/2012/5/30/438
> 
> and it would greatly simplify assigning PCI devices on these systems
> with VFIO.  Thanks,

I tried to run qemu-system-x86_64 but got this error on startup:

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
**
ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)


I started qemu-system-x86_64 with this option among others 

-device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5

after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.


Could you please tell me, why the operation isn't permitted? I started
qemu-system-x86_64 as root.



Thanks,
kind regards,
Andreas Hartmann