* AMD KVM Pci Passthrough reports device busy @ 2012-06-04 21:11 Chris Sanders 2012-06-05 3:44 ` Alex Williamson 0 siblings, 1 reply; 30+ messages in thread From: Chris Sanders @ 2012-06-04 21:11 UTC (permalink / raw) To: kvm Hello, I've been working for several days trying to get Pci Passthrough to work. So far the #kvm IRC channel has helped me with a few suggestions, though that hasn't yet solved the problem. I'm running CentOS 6.2 and was suggested I try compiling 3.2.18 kernel form kernel.org. This has changed a few of the messages but the guest still fails to start. Grepping for AMD-VI produces: # dmesg | grep AMD-Vi AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 AMD-Vi: Lazy IO/TLB flushing enabled After boot I'm running the following script echo "unbind pci-pci bridge" echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id echo "unbind pci device" echo "0000:03:07.0" > /sys/bus/pci/drivers/ivtv/unbind echo "4444 0803" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:03:07.0 > /sys/bus/pci/devices/0000\:03\:07.0/driver/unbind echo 0000:03:07.0 > /sys/bus/pci/drivers/pci-stub/bind This is lspci -n showing my device behind the Pci-Pci bridge -[0000:00]-+-00.0 +-00.2 +-02.0-[01]--+-00.0 | \-00.1 +-09.0-[02]----00.0 +-11.0 +-12.0 +-12.2 +-13.0 +-13.2 +-14.0 +-14.1 +-14.3 +-14.4-[03]----07.0 +-14.5 +-15.0-[04]-- +-16.0 +-16.2 +-18.0 +-18.1 +-18.2 +-18.3 +-18.4 \-18.5 My kvm command and error are: # /usr/libexec/qemu-kvm -m 3048 -net none -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/vg_hdd/lv_sagetv,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device pci-assign,host=03:07.0 Failed to assign device "(null)" : Device or resource busy qemu-kvm: -device pci-assign,host=03:07.0: Device 'pci-assign' could not be initialized dmesg after attempting to start shows: fuse init (API version 7.17) ivtv 0000:03:07.0: BAR 0: can't reserve [mem 0xf8000000-0xfbffffff pref] kvm_vm_ioctl_assign_device: Could not get access to device regions tda9887 5-0043: destroying instance tuner-simple 5-0061: destroying instance ivtv 0000:03:07.0: PCI INT A disabled ivtv: Removed Hauppauge WinTV PVR-350 pci-stub 0000:03:07.0: claimed by stub pci-stub 0000:03:07.0: claimed by stub pci-stub 0000:03:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pci-stub 0000:03:07.0: restoring config space at offset 0x1 (was 0x2100000, writing 0x2100002) assign device 0:3:7.0 failed pci-stub 0000:03:07.0: PCI INT A disabled This is on the 3.2.18 kernel amd_iommu=1 clear_emulator_capabilities = 0 relaxed_acs_check = 1 Any help would be greatly appreciated. I've just recently bought this hardware specifically for this purpose and if the motherboard has a bad IOMMU implementation I'd like to figure it out in time to exchange it. This is an GA-970A-D3 Thanks, Chris P.S. I'm not subscribed to the list so if this list requires is be sure to cc me on responses (sanders.chris@gmail.com) ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-04 21:11 AMD KVM Pci Passthrough reports device busy Chris Sanders @ 2012-06-05 3:44 ` Alex Williamson 2012-06-05 10:39 ` Andreas Hartmann 0 siblings, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-05 3:44 UTC (permalink / raw) To: Chris Sanders; +Cc: kvm On Mon, 2012-06-04 at 16:11 -0500, Chris Sanders wrote: > Hello, I've been working for several days trying to get Pci > Passthrough to work. So far the #kvm IRC channel has helped me with a > few suggestions, though that hasn't yet solved the problem. I'm > running CentOS 6.2 and was suggested I try compiling 3.2.18 kernel > form kernel.org. This has changed a few of the messages but the guest > still fails to start. > > Grepping for AMD-VI produces: > # dmesg | grep AMD-Vi > AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 > AMD-Vi: Lazy IO/TLB flushing enabled > > After boot I'm running the following script > echo "unbind pci-pci bridge" > echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id > echo "unbind pci device" > echo "0000:03:07.0" > /sys/bus/pci/drivers/ivtv/unbind > echo "4444 0803" > /sys/bus/pci/drivers/pci-stub/new_id > echo 0000:03:07.0 > /sys/bus/pci/devices/0000\:03\:07.0/driver/unbind > echo 0000:03:07.0 > /sys/bus/pci/drivers/pci-stub/bind > > This is lspci -n showing my device behind the Pci-Pci bridge > -[0000:00]-+-00.0 > +-00.2 > +-02.0-[01]--+-00.0 > | \-00.1 > +-09.0-[02]----00.0 > +-11.0 > +-12.0 > +-12.2 > +-13.0 > +-13.2 > +-14.0 > +-14.1 > +-14.3 > +-14.4-[03]----07.0 > +-14.5 > +-15.0-[04]-- > +-16.0 > +-16.2 > +-18.0 > +-18.1 > +-18.2 > +-18.3 > +-18.4 > \-18.5 > > My kvm command and error are: > # /usr/libexec/qemu-kvm -m 3048 -net none -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive > file=/dev/vg_hdd/lv_sagetv,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native > -device pci-assign,host=03:07.0 > Failed to assign device "(null)" : Device or resource busy > qemu-kvm: -device pci-assign,host=03:07.0: Device 'pci-assign' could > not be initialized I have a setup with an AMD 990FX system and a spare PVR-350 card that I installed to reproduce. The sad answer is that it's nearly impossible to assign PCI devices on these systems due to the aliasing of devices below the PCIe-to-PCI bridge (PCIe devices are much, much easier to assign). If you boot with amd_iommu_dump, you'll see some output like this: AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 AMD-Vi: DEV_RANGE_END devid: 05:1f.7 This says the devices on bus 5 (my bus 5 is equivalent to your bus 3) are all aliased to device 14.4: 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) What that means is that the IOMMU can't distinguish devices behind the PCI-to-PCI bridge so all devices are grouped as an alias to device 14.4. You would hopefully not care about this, you don't have any other devices anyway. Unfortunately amd_iommu pre-allocates IOMMU domains for every device, so it's already allocated a domain for device 14.4 and adds device 03:07.0 into it. Unbinding 03:07.0 from the ivtv driver detaches that devices from the domain, but when we go to assign it to a guest we create a new domain. Assigning 03:07.0 into that new domain fails because the device is an alias for 00:14.4, which still has a different domain. One way to get around this would be to also assign the bridge to the guest, but we don't support and actually reject assigning bridges :( This works a bit better on Intel VT-d systems because domains are dynamically allocated. Thus for streaming DMA, the domain is only created when the driver attempts to setup a DMA transaction. When the driver is unbound, the domain is destroyed thus allowing us to setup a new domain for device assignment. If you don't mind running non-upstream code, VFIO is a re-write of device assignment for Qemu that is aware of such alias problems and actually works in this case. The downside is that VFIO is strict about multifunction devices supporting ACS to prevent peer-to-peer between domains, so will require all of the 14.x devices to be bound to pci-stub as well. On my system, this includes an smbus controller, audio device, lpc controller, and usb device. If AMD could confirm this device doesn't allow peer-to-peer between functions, we could relax this requirement a bit. VFIO kernel and qemu can be found here: git://github.com/awilliam/linux-vfio.git (iommu-group-vfio-next-20120529) git://github.com/awilliam/qemu-vfio.git (iommu-group-vfio) See Documentation/vfio.txt for description. The major difference is the setup of drivers. At a minimum, the device you want to assign to the guest needs to be bound to the vfio-pci driver, much in the same way you bind devices to pci-stub. All other devices in the group should be bound to pci-stub. You can find the other devices in the group by following the iommu_group link in sysfs, ex: /sys/bus/pci/devices/0000:03:07.0/iommu_group/devices Once you have that, simply replace pci-assign with vfio-pci on the qemu command line. I'm actively trying to get this code upstream and hoping we can do it asap, so barring it getting rejected, this would hopefully eventually make mainline. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 3:44 ` Alex Williamson @ 2012-06-05 10:39 ` Andreas Hartmann 2012-06-05 14:27 ` Alex Williamson 2012-06-06 1:32 ` sheng qiu 0 siblings, 2 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 10:39 UTC (permalink / raw) To: Alex Williamson, Joerg.Roedel; +Cc: Chris Sanders, kvm Hello Alex, thanks for your efforts! Maybe, you already know that I'm suffering the same problem :-( with a GA-990XA-UD3 (990X chipset). On Mon, 04 Jun 2012 21:44:05 -0600 Alex Williamson <alex.williamson@redhat.com> wrote: [...] > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > installed to reproduce. The sad answer is that it's nearly impossible > to assign PCI devices on these systems due to the aliasing of devices > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > assign). If you boot with amd_iommu_dump, you'll see some output like > this: > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 here: AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 AMD-Vi: DEV_RANGE_END devid: 06:1f.7 This means according to your description, the following devices have to be unbound here, too (0000:06:07.0 is the device I want to passthrough): 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 These are the following additional devices: 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) Among them is the sound device and the USB device - no good idea to disable them on a desktop. Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but I'm getting errors after rebooting the system (the filesystems haven't been cleanly unmounted during shutdown). Anyway, I would have tested it, but I'm getting a compile error while compiling qemu. It complains about missing pci/header.h while processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where should it come from? [...] > The downside is that VFIO is strict about > multifunction devices supporting ACS to prevent peer-to-peer between > domains, so will require all of the 14.x devices to be bound to pci-stub > as well. On my system, this includes an smbus controller, audio device, > lpc controller, and usb device. If AMD could confirm this device > doesn't allow peer-to-peer between functions, we could relax this > requirement a bit. Please Joerg, could you take a look at this problem? Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 10:39 ` Andreas Hartmann @ 2012-06-05 14:27 ` Alex Williamson 2012-06-05 15:17 ` Andreas Hartmann 2012-06-06 10:11 ` Joerg Roedel 2012-06-06 1:32 ` sheng qiu 1 sibling, 2 replies; 30+ messages in thread From: Alex Williamson @ 2012-06-05 14:27 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote: > Hello Alex, > > thanks for your efforts! > > Maybe, you already know that I'm suffering the same problem :-( with a > GA-990XA-UD3 (990X chipset). My system is a GA-990FXA-UD3, so just a slightly different version of the chipset. > On Mon, 04 Jun 2012 21:44:05 -0600 > Alex Williamson <alex.williamson@redhat.com> wrote: > [...] > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > > installed to reproduce. The sad answer is that it's nearly impossible > > to assign PCI devices on these systems due to the aliasing of devices > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > > assign). If you boot with amd_iommu_dump, you'll see some output like > > this: > > > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > here: > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > This means according to your description, the following devices have to > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 Yep. > These are the following additional devices: > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) My system shows this for lspci -n -s 14. 00:14.0 0c05: 1002:4385 (rev 42) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 Can you confirm yours matches and fill in 14.1 so we have the vendor:device IDs for these. > Among them is the sound device and the USB device - no good idea to > disable them on a desktop. You can always blacklist drivers if the device is unique or unbind devices in an rc.local type script. I'm hoping that Joerg or someone else from AMD can tell us this devices does not allow internal peer-to-peer between functions. Then we can add it to the device specific ACS checks and you wouldn't need to worry about displacing all the other 14.x functions. > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > I'm getting errors after rebooting the system (the filesystems haven't > been cleanly unmounted during shutdown). Did you have a disk off that controller? My system doesn't expose function 1, but is the same otherwise. > Anyway, I would have tested it, but I'm getting a compile error while > compiling qemu. It complains about missing pci/header.h while > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > should it come from? It is from pciutils-devel for me. I'll see what I'm getting out of there and try to remove it. > > The downside is that VFIO is strict about > > multifunction devices supporting ACS to prevent peer-to-peer between > > domains, so will require all of the 14.x devices to be bound to pci-stub > > as well. On my system, this includes an smbus controller, audio device, > > lpc controller, and usb device. If AMD could confirm this device > > doesn't allow peer-to-peer between functions, we could relax this > > requirement a bit. > > Please Joerg, could you take a look at this problem? Joerg, the question is whether the multifunction device above allows peer-to-peer between functions that could bypass the iommu. If not, we can make it the first entry in device specific acs enabled function I proposed: https://lkml.org/lkml/2012/5/30/438 and it would greatly simplify assigning PCI devices on these systems with VFIO. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 14:27 ` Alex Williamson @ 2012-06-05 15:17 ` Andreas Hartmann 2012-06-05 15:48 ` Alex Williamson 2012-06-05 15:58 ` Andreas Hartmann 2012-06-06 10:11 ` Joerg Roedel 1 sibling, 2 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 15:17 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm On Tue, 05 Jun 2012 08:27:05 -0600 Alex Williamson <alex.williamson@redhat.com> wrote: > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote: > > Hello Alex, > > > > thanks for your efforts! > > > > Maybe, you already know that I'm suffering the same problem :-( with a > > GA-990XA-UD3 (990X chipset). > > My system is a GA-990FXA-UD3, so just a slightly different version of > the chipset. > > > On Mon, 04 Jun 2012 21:44:05 -0600 > > Alex Williamson <alex.williamson@redhat.com> wrote: > > [...] > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > > > installed to reproduce. The sad answer is that it's nearly impossible > > > to assign PCI devices on these systems due to the aliasing of devices > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > > > assign). If you boot with amd_iommu_dump, you'll see some output like > > > this: > > > > > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > > > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > > > here: > > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > > > > This means according to your description, the following devices have to > > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 > > Yep. > > > These are the following additional devices: > > > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > My system shows this for lspci -n -s 14. > > 00:14.0 0c05: 1002:4385 (rev 42) > 00:14.2 0403: 1002:4383 (rev 40) > 00:14.3 0601: 1002:439d (rev 40) > 00:14.4 0604: 1002:4384 (rev 40) > 00:14.5 0c03: 1002:4399 My system: 00:14.0 0c05: 1002:4385 (rev 42) 00:14.1 0101: 1002:439c (rev 40) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 You miss 00.14.1!? But the others are equal. > Can you confirm yours matches and fill in 14.1 so we have the > vendor:device IDs for these. Yes. > > Among them is the sound device and the USB device - no good idea to > > disable them on a desktop. > > You can always blacklist drivers if the device is unique or unbind > devices in an rc.local type script. I'm hoping that Joerg or someone > else from AMD can tell us this devices does not allow internal > peer-to-peer between functions. Then we can add it to the device > specific ACS checks and you wouldn't need to worry about displacing all > the other 14.x functions. This would be cool. > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > > I'm getting errors after rebooting the system (the filesystems haven't > > been cleanly unmounted during shutdown). > > Did you have a disk off that controller? My system doesn't expose > function 1, but is the same otherwise. Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the reason is another? > > Anyway, I would have tested it, but I'm getting a compile error while > > compiling qemu. It complains about missing pci/header.h while > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > > should it come from? > > It is from pciutils-devel for me. I'll see what I'm getting out of > there and try to remove it. Meanwhile I found it :-). Thanks for your hint! > > > The downside is that VFIO is strict about > > > multifunction devices supporting ACS to prevent peer-to-peer between > > > domains, so will require all of the 14.x devices to be bound to pci-stub > > > as well. On my system, this includes an smbus controller, audio device, > > > lpc controller, and usb device. If AMD could confirm this device > > > doesn't allow peer-to-peer between functions, we could relax this > > > requirement a bit. > > > > Please Joerg, could you take a look at this problem? > > Joerg, the question is whether the multifunction device above allows > peer-to-peer between functions that could bypass the iommu. If not, we > can make it the first entry in device specific acs enabled function I > proposed: > > https://lkml.org/lkml/2012/5/30/438 > > and it would greatly simplify assigning PCI devices on these systems > with VFIO. Thanks, I tried to run qemu-system-x86_64 but got this error on startup: qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 ** ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) I started qemu-system-x86_64 with this option among others -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. Could you please tell me, why the operation isn't permitted? I started qemu-system-x86_64 as root. Thanks, kind regards, Andreas Hartmann ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 15:17 ` Andreas Hartmann @ 2012-06-05 15:48 ` Alex Williamson 2012-06-05 15:58 ` Andreas Hartmann 1 sibling, 0 replies; 30+ messages in thread From: Alex Williamson @ 2012-06-05 15:48 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm On Tue, 2012-06-05 at 17:17 +0200, Andreas Hartmann wrote: > On Tue, 05 Jun 2012 08:27:05 -0600 > Alex Williamson <alex.williamson@redhat.com> wrote: > > > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote: > > > Hello Alex, > > > > > > thanks for your efforts! > > > > > > Maybe, you already know that I'm suffering the same problem :-( with a > > > GA-990XA-UD3 (990X chipset). > > > > My system is a GA-990FXA-UD3, so just a slightly different version of > > the chipset. > > > > > On Mon, 04 Jun 2012 21:44:05 -0600 > > > Alex Williamson <alex.williamson@redhat.com> wrote: > > > [...] > > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > > > > installed to reproduce. The sad answer is that it's nearly impossible > > > > to assign PCI devices on these systems due to the aliasing of devices > > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > > > > assign). If you boot with amd_iommu_dump, you'll see some output like > > > > this: > > > > > > > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > > > > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > > > > > here: > > > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > > > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > > > > > > > This means according to your description, the following devices have to > > > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > > > > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 > > > > Yep. > > > > > These are the following additional devices: > > > > > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > > > My system shows this for lspci -n -s 14. > > > > 00:14.0 0c05: 1002:4385 (rev 42) > > 00:14.2 0403: 1002:4383 (rev 40) > > 00:14.3 0601: 1002:439d (rev 40) > > 00:14.4 0604: 1002:4384 (rev 40) > > 00:14.5 0c03: 1002:4399 > > My system: > 00:14.0 0c05: 1002:4385 (rev 42) > 00:14.1 0101: 1002:439c (rev 40) > 00:14.2 0403: 1002:4383 (rev 40) > 00:14.3 0601: 1002:439d (rev 40) > 00:14.4 0604: 1002:4384 (rev 40) > 00:14.5 0c03: 1002:4399 > > You miss 00.14.1!? But the others are equal. Yeah, my system doesn't expose the IDE, *shrug* > > Can you confirm yours matches and fill in 14.1 so we have the > > vendor:device IDs for these. > > Yes. Thanks > > > Among them is the sound device and the USB device - no good idea to > > > disable them on a desktop. > > > > You can always blacklist drivers if the device is unique or unbind > > devices in an rc.local type script. I'm hoping that Joerg or someone > > else from AMD can tell us this devices does not allow internal > > peer-to-peer between functions. Then we can add it to the device > > specific ACS checks and you wouldn't need to worry about displacing all > > the other 14.x functions. > > This would be cool. > > > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > > > I'm getting errors after rebooting the system (the filesystems haven't > > > been cleanly unmounted during shutdown). > > > > Did you have a disk off that controller? My system doesn't expose > > function 1, but is the same otherwise. > > Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the > reason is another? > > > > Anyway, I would have tested it, but I'm getting a compile error while > > > compiling qemu. It complains about missing pci/header.h while > > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > > > should it come from? > > > > It is from pciutils-devel for me. I'll see what I'm getting out of > > there and try to remove it. > > Meanwhile I found it :-). Thanks for your hint! Great. > > > > The downside is that VFIO is strict about > > > > multifunction devices supporting ACS to prevent peer-to-peer between > > > > domains, so will require all of the 14.x devices to be bound to pci-stub > > > > as well. On my system, this includes an smbus controller, audio device, > > > > lpc controller, and usb device. If AMD could confirm this device > > > > doesn't allow peer-to-peer between functions, we could relax this > > > > requirement a bit. > > > > > > Please Joerg, could you take a look at this problem? > > > > Joerg, the question is whether the multifunction device above allows > > peer-to-peer between functions that could bypass the iommu. If not, we > > can make it the first entry in device specific acs enabled function I > > proposed: > > > > https://lkml.org/lkml/2012/5/30/438 > > > > and it would greatly simplify assigning PCI devices on these systems > > with VFIO. Thanks, > > I tried to run qemu-system-x86_64 but got this error on startup: > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 > ** > ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) > > > I started qemu-system-x86_64 with this option among others > > -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 > > after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. > > > Could you please tell me, why the operation isn't permitted? I started > qemu-system-x86_64 as root. Check dmesg, it's probably complaining about no interrupt remapping support. If so: # modprobe -r vfio_iommu_type1 # modprobe vfio_iommu_type1 allow_unsafe_interrupts=1 Then try again. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 15:17 ` Andreas Hartmann 2012-06-05 15:48 ` Alex Williamson @ 2012-06-05 15:58 ` Andreas Hartmann 2012-06-05 16:19 ` Alex Williamson 1 sibling, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 15:58 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, Chris Sanders, kvm Andreas Hartmann wrote: [...] > I tried to run qemu-system-x86_64 but got this error on startup: > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 > ** > ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) > > > I started qemu-system-x86_64 with this option among others > > -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 > > after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. > > > Could you please tell me, why the operation isn't permitted? I started > qemu-system-x86_64 as root. I straced the call with strace and got the following error: ... 8048 open("/usr/local/share/qemu/pxe-virtio.rom", O_RDONLY) = 14 8048 lseek(14, 0, SEEK_END) = 60416 8048 lseek(14, 0, SEEK_SET) = 0 8048 read(14, "U\252v\351\217\0z\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\177\0 \0<\0\365\274\266\16"..., 60416) = 60416 8048 close(14) = 0 8048 stat("/sys/bus/pci/devices/0000:06:07.0/", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 8048 readlink("/sys/bus/pci/devices/0000:06:07.0/iommu_group", "../../../../kernel/iommu_groups/9"..., 4096) = 33 8048 open("/dev/vfio/9", O_RDWR) = 14 8048 ioctl(14, 0x3b67, 0x7fff237d5ac0) = 0 8048 open("/dev/vfio/vfio", O_RDWR) = 15 8048 ioctl(15, 0x3b64, 0xf) = 0 8048 ioctl(15, 0x3b65, 0x1) = 1 8048 ioctl(14, 0x3b68, 0x7fff237d5ad8) = 0 8048 ioctl(15, 0x3b66, 0x1) = -1 EPERM (Operation not permitted) ... /dev/vfio # ls -l total 0 crw-rw-rw- 1 root root 250, 1 Jun 5 17:42 9 crw-rw-rw- 1 root root 250, 0 Jun 5 17:41 vfio Kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 15:58 ` Andreas Hartmann @ 2012-06-05 16:19 ` Alex Williamson 2012-06-05 16:55 ` Andreas Hartmann 0 siblings, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-05 16:19 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm On Tue, 2012-06-05 at 17:58 +0200, Andreas Hartmann wrote: > Andreas Hartmann wrote: > [...] > > I tried to run qemu-system-x86_64 but got this error on startup: > > > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted > > > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 > > > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 > > ** > > ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) > > > > > > I started qemu-system-x86_64 with this option among others > > > > -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 > > > > after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. > > > > > > Could you please tell me, why the operation isn't permitted? I started > > qemu-system-x86_64 as root. > > I straced the call with strace and got the following error: > > ... > 8048 open("/usr/local/share/qemu/pxe-virtio.rom", O_RDONLY) = 14 > 8048 lseek(14, 0, SEEK_END) = 60416 > 8048 lseek(14, 0, SEEK_SET) = 0 > 8048 read(14, "U\252v\351\217\0z\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\177\0 > \0<\0\365\274\266\16"..., 60416) = 60416 > 8048 close(14) = 0 > 8048 stat("/sys/bus/pci/devices/0000:06:07.0/", {st_mode=S_IFDIR|0755, > st_size=0, ...}) = 0 > 8048 readlink("/sys/bus/pci/devices/0000:06:07.0/iommu_group", > "../../../../kernel/iommu_groups/9"..., 4096) = 33 > 8048 open("/dev/vfio/9", O_RDWR) = 14 > 8048 ioctl(14, 0x3b67, 0x7fff237d5ac0) = 0 > 8048 open("/dev/vfio/vfio", O_RDWR) = 15 > 8048 ioctl(15, 0x3b64, 0xf) = 0 > 8048 ioctl(15, 0x3b65, 0x1) = 1 > 8048 ioctl(14, 0x3b68, 0x7fff237d5ad8) = 0 > 8048 ioctl(15, 0x3b66, 0x1) = -1 EPERM (Operation not permitted) > ... Yep, I think the previous suggestion about reloading vfio_iommu_type1 with allow_unsafe_interrupts=1 will solve it. It has nothing to do with file permissions, you're getting EPERM at the point where we set the iommu type, which enables access to devices. By default we want an iommu which protects against malicious MSI attacks, which requires interrupt remapping on x86. Joerg has been working to add this for AMD since the hardware supports it, but for now, the above options lets us bypass the check. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 16:19 ` Alex Williamson @ 2012-06-05 16:55 ` Andreas Hartmann 2012-06-05 18:43 ` Alex Williamson 2012-06-06 8:12 ` Andreas Hartmann 0 siblings, 2 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 16:55 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm Alex Williamson wrote: [...] > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > with allow_unsafe_interrupts=1 will solve it. Yes! Works now. Success!!!!! Works means: Device is seen in VM. I couldn't test it up to now, because I don't have any driver in the VM for this device. > It has nothing to do with > file permissions, you're getting EPERM at the point where we set the > iommu type, which enables access to devices. By default we want an > iommu which protects against malicious MSI attacks, which requires > interrupt remapping on x86. Joerg has been working to add this for AMD > since the hardware supports it, but for now, the above options lets us > bypass the check. Thanks, Now, I have to check my problem with the filesystem first, which isn't cleanly unmounted. I suspect a few missing options for kernel compile. Next will be fglrx :-) for 3.4 Hmmm, is there a chance to get your extension isolated to get it compiled for 3.1, too? Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 16:55 ` Andreas Hartmann @ 2012-06-05 18:43 ` Alex Williamson 2012-06-05 20:37 ` Andreas Hartmann 2012-06-06 8:12 ` Andreas Hartmann 1 sibling, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-05 18:43 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote: > Alex Williamson wrote: > [...] > > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > > with allow_unsafe_interrupts=1 will solve it. > > Yes! Works now. Success!!!!! > > Works means: Device is seen in VM. I couldn't test it up to now, because > I don't have any driver in the VM for this device. I tested it as far as doing 'cat /dev/video0 > /tmp/test'. The file grows, the device gets interrupts and `file` thinks it's a video (no input connected to the card). > > It has nothing to do with > > file permissions, you're getting EPERM at the point where we set the > > iommu type, which enables access to devices. By default we want an > > iommu which protects against malicious MSI attacks, which requires > > interrupt remapping on x86. Joerg has been working to add this for AMD > > since the hardware supports it, but for now, the above options lets us > > bypass the check. Thanks, > > Now, I have to check my problem with the filesystem first, which isn't > cleanly unmounted. I suspect a few missing options for kernel compile. > Next will be fglrx :-) for 3.4 > > Hmmm, is there a chance to get your extension isolated to get it > compiled for 3.1, too? The patches are all layered on top of that branch, so you should be able to easily 'git show' to get patches or cherry-pick them across trees. I don't think there should be too much trouble backporting them. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 18:43 ` Alex Williamson @ 2012-06-05 20:37 ` Andreas Hartmann 2012-06-05 21:09 ` Alex Williamson 0 siblings, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 20:37 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm Alex Williamson wrote: > On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote: >> Alex Williamson wrote: >> [...] >>> Yep, I think the previous suggestion about reloading vfio_iommu_type1 >>> with allow_unsafe_interrupts=1 will solve it. >> >> Yes! Works now. Success!!!!! >> >> Works means: Device is seen in VM. I couldn't test it up to now, because >> I don't have any driver in the VM for this device. Well, I've got another problem now with 3.4: I can't pass through my PCIe device any more, which works fine without any problem with 3.1.10. I'm getting the following error in 3.4: Failed to assign irq for "hostdev0": Input/output error Perhaps you are assigning a device that shares an IRQ with another device? qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized options kvm allow_unsafe_assigned_interrupts=1 is set. There are share IRQ's - but that's the same in 3.1.10. This VM is started with libvirt (virsh) the old fashioned way with the old qemu-kvm tool. Doesn't this work any more? BTW: I think libvirt / virsh isn't aware yet of the new vfio way, isn't it? Thanks, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 20:37 ` Andreas Hartmann @ 2012-06-05 21:09 ` Alex Williamson 2012-06-05 22:02 ` Andreas Hartmann 0 siblings, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-05 21:09 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm On Tue, 2012-06-05 at 22:37 +0200, Andreas Hartmann wrote: > Alex Williamson wrote: > > On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote: > >> Alex Williamson wrote: > >> [...] > >>> Yep, I think the previous suggestion about reloading vfio_iommu_type1 > >>> with allow_unsafe_interrupts=1 will solve it. > >> > >> Yes! Works now. Success!!!!! > >> > >> Works means: Device is seen in VM. I couldn't test it up to now, because > >> I don't have any driver in the VM for this device. > > Well, I've got another problem now with 3.4: I can't pass through my > PCIe device any more, which works fine without any problem with 3.1.10. > I'm getting the following error in 3.4: > > Failed to assign irq for "hostdev0": Input/output error > Perhaps you are assigning a device that shares an IRQ with another device? > qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized > > options kvm allow_unsafe_assigned_interrupts=1 is set. > > There are share IRQ's - but that's the same in 3.1.10. > > This VM is started with libvirt (virsh) the old fashioned way with the > old qemu-kvm tool. Doesn't this work any more? Does this help: https://lkml.org/lkml/2012/6/1/261 Otherwise, dmesg and device info please. > BTW: I think libvirt / virsh isn't aware yet of the new vfio way, isn't > it? Correct, need to get it into the kernel and qemu first. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 21:09 ` Alex Williamson @ 2012-06-05 22:02 ` Andreas Hartmann 0 siblings, 0 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-05 22:02 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm Alex Williamson wrote: > On Tue, 2012-06-05 at 22:37 +0200, Andreas Hartmann wrote: >> Alex Williamson wrote: >>> On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote: >>>> Alex Williamson wrote: >>>> [...] >>>>> Yep, I think the previous suggestion about reloading vfio_iommu_type1 >>>>> with allow_unsafe_interrupts=1 will solve it. >>>> >>>> Yes! Works now. Success!!!!! >>>> >>>> Works means: Device is seen in VM. I couldn't test it up to now, because >>>> I don't have any driver in the VM for this device. >> >> Well, I've got another problem now with 3.4: I can't pass through my >> PCIe device any more, which works fine without any problem with 3.1.10. >> I'm getting the following error in 3.4: >> >> Failed to assign irq for "hostdev0": Input/output error >> Perhaps you are assigning a device that shares an IRQ with another device? >> qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized >> >> options kvm allow_unsafe_assigned_interrupts=1 is set. >> >> There are shared IRQ's - but that's the same in 3.1.10. >> >> This VM is started with libvirt (virsh) the old fashioned way with the >> old qemu-kvm tool. Doesn't this work any more? > > Does this help: https://lkml.org/lkml/2012/6/1/261 Yes - this fixed the problem! Thanks, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 16:55 ` Andreas Hartmann 2012-06-05 18:43 ` Alex Williamson @ 2012-06-06 8:12 ` Andreas Hartmann 2012-06-06 8:46 ` Andreas Hartmann 2012-06-06 16:39 ` Alex Williamson 1 sibling, 2 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-06 8:12 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, kvm On Tue, 05 Jun 2012 18:55:42 +0200 Andreas Hartmann <andihartmann@01019freenet.de> wrote: > Alex Williamson wrote: > [...] > > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > > with allow_unsafe_interrupts=1 will solve it. > > Yes! Works now. Success!!!!! > > Works means: Device is seen in VM. I couldn't test it up to now, because > I don't have any driver in the VM for this device. Meanwhile, I enabled the drivers in the VM for the device I passed through. Unfortunately, it doesn't work :-(. I'm getting this entry in messages at the moment, the module rt2800pci is used by hostapd: Jun 6 09:25:02 host kernel: [ 201.895812] irq 21: nobody cared (try booting with the "irqpoll" option) Jun 6 09:25:02 host kernel: [ 201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6 Jun 6 09:25:02 host kernel: [ 201.895822] Call Trace: Jun 6 09:25:02 host kernel: [ 201.895836] <IRQ> [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0 Jun 6 09:25:02 host kernel: [ 201.895842] [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220 Jun 6 09:25:02 host kernel: [ 201.895849] [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270 Jun 6 09:25:02 host kernel: [ 201.895855] [<ffffffff810d14c9>] handle_irq_event+0x49/0x70 Jun 6 09:25:02 host kernel: [ 201.895860] [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130 Jun 6 09:25:02 host kernel: [ 201.895865] [<ffffffff81004460>] handle_irq+0x20/0x30 Jun 6 09:25:02 host kernel: [ 201.895869] [<ffffffff81004098>] do_IRQ+0x58/0xe0 Jun 6 09:25:02 host kernel: [ 201.895876] [<ffffffff815f112a>] common_interrupt+0x6a/0x6a Jun 6 09:25:02 host kernel: [ 201.895907] <EOI> [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor] Jun 6 09:25:02 host kernel: [ 201.895915] [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20 Jun 6 09:25:02 host kernel: [ 201.895929] [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor] Jun 6 09:25:02 host kernel: [ 201.895939] [<ffffffff814915f9>] cpuidle_enter+0x19/0x20 Jun 6 09:25:02 host kernel: [ 201.895943] [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0 Jun 6 09:25:02 host kernel: [ 201.895949] [<ffffffff8100bd45>] cpu_idle+0x85/0xd0 Jun 6 09:25:02 host kernel: [ 201.895955] [<ffffffff815e63d5>] start_secondary+0x8a/0x8c Jun 6 09:25:02 host kernel: [ 201.895958] handlers: Jun 6 09:25:02 host kernel: [ 201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci] Jun 6 09:25:02 host kernel: [ 201.895969] Disabling IRQ #21 I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared interrupt! Did I miss an option during kernel configuration? Just to remember: The passed through device is a WLAN-device and it's driven by rt2800pci: 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI I unbound all the devices in the same group: echo "1002 4385" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.0 > /sys/bus/pci/devices/0000:00:14.0/driver/unbind echo 0000:00:14.0 > /sys/bus/pci/drivers/pci-stub/bind echo "1002 439c " > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.1 > /sys/bus/pci/devices/0000:00:14.1/driver/unbind echo 0000:00:14.1 > /sys/bus/pci/drivers/pci-stub/bind echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.2 > /sys/bus/pci/devices/0000:00:14.2/driver/unbind echo 0000:00:14.2 > /sys/bus/pci/drivers/pci-stub/bind echo "1002 439d" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.3 > /sys/bus/pci/devices/0000:00:14.3/driver/unbind echo 0000:00:14.3 > /sys/bus/pci/drivers/pci-stub/bind echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.4 > /sys/bus/pci/devices/0000:00:14.4/driver/unbind echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/bind echo "1002 4399" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:00:14.5 > /sys/bus/pci/devices/0000:00:14.5/driver/unbind echo 0000:00:14.5 > /sys/bus/pci/drivers/pci-stub/bind modprobe vfio-pci echo "1814 0601" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:06:07.0 > /sys/bus/pci/devices/0000:06:07.0/driver/unbind echo 0000:06:07.0 > /sys/bus/pci/drivers/vfio-pci/bind The corresponding entries in messages are: Jun 6 09:23:01 host kernel: [ 81.777453] pci-stub 0000:00:14.0: claimed by stub Jun 6 09:23:01 host kernel: [ 81.777788] pci-stub 0000:00:14.0: claimed by stub Jun 6 09:23:02 host kernel: [ 81.782465] pci-stub 0000:00:14.1: claimed by stub Jun 6 09:23:02 host kernel: [ 81.801572] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600] Jun 6 09:23:02 host kernel: [ 81.950388] pci-stub 0000:00:14.2: claimed by stub Jun 6 09:23:02 host kernel: [ 81.950505] pci-stub 0000:00:14.3: claimed by stub Jun 6 09:23:02 host kernel: [ 81.950723] pci-stub 0000:00:14.3: claimed by stub Jun 6 09:23:02 host kernel: [ 81.950828] pci-stub 0000:00:14.4: claimed by stub Jun 6 09:23:02 host kernel: [ 81.951031] pci-stub 0000:00:14.4: claimed by stub Jun 6 09:23:02 host kernel: [ 81.951280] ohci_hcd 0000:00:14.5: remove, state 4 Jun 6 09:23:02 host kernel: [ 81.951293] usb usb6: USB disconnect, device number 1 Jun 6 09:23:02 host kernel: [ 81.951840] ohci_hcd 0000:00:14.5: USB bus 6 deregistered Jun 6 09:23:02 host kernel: [ 81.951993] pci-stub 0000:00:14.5: claimed by stub Jun 6 09:23:02 host kernel: [ 81.957543] VFIO - User Level meta-driver version: 0.3 Jun 6 09:23:02 host kernel: [ 81.960441] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 bound to driver vfio-pci Jun 6 09:23:02 host kernel: [ 81.960617] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 unbinding from driver vfio-pci Jun 6 09:23:02 host kernel: [ 81.961002] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 bound to driver vfio-pci The result with lspci -vs14 is: 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) Flags: 66MHz, medium devsel Kernel driver in use: pci-stub 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) Subsystem: Giga-byte Technology Device 5002 Flags: 66MHz, medium devsel, IRQ 17 I/O ports at 01f0 [size=8] I/O ports at 03f4 [size=1] I/O ports at 0170 [size=8] I/O ports at 0374 [size=1] I/O ports at fa00 [size=16] Kernel driver in use: pci-stub 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) Subsystem: Giga-byte Technology Device a132 Flags: slow devsel, IRQ 16 Memory at fdff4000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Kernel driver in use: pci-stub 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller Flags: bus master, 66MHz, medium devsel, latency 0 Kernel driver in use: pci-stub 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) Flags: bus master, VGA palette snoop, 66MHz, medium devsel, latency 64 Bus: primary=00, secondary=06, subordinate=06, sec-latency=64 I/O behind bridge: 00009000-00009fff Memory behind bridge: fd800000-fd8fffff Prefetchable memory behind bridge: fd700000-fd7fffff Kernel driver in use: pci-stub 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) Subsystem: Giga-byte Technology Device 5004 Flags: 66MHz, medium devsel, IRQ 18 Memory at fdffa000 (32-bit, non-prefetchable) [size=4K] Kernel driver in use: pci-stub lspci -vs7 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI Subsystem: Linksys Device 0067 Flags: bus master, slow devsel, latency 32, IRQ 5 Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Kernel driver in use: vfio-pci On starting the VM, the following entries came up in messages: Jun 6 09:23:51 host kernel: [ 131.397055] kvm: 8175: cpu0 unhandled rdmsr: 0xc0010001 The VM is started as root with this command: /usr/local/bin/qemu-system-x86_64 \ -enable-kvm \ -m 1024 \ -rtc base=utc \ -drive file=/my.hd,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -netdev tap,ifname=tap5,script=no,id=hostnet0 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=12:34:56:78:90:12,bus=pci.0,addr=0x3 \ -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 qemu-system-x86_64 prints out the following on startup: Warning, device 0000:06:07.0 does not support reset (-> only the first time the VM is started) sometimes: qemu-system-x86_64: vfio_dma_map(0x7fb10a848170, 0x00000000000cc000, 0x4000, 0x7fb0bfacc000) = -16 (Device or resource busy) qemu-system-x86_64: vfio_dma_map(0x7fb10a848170, 0x00000000000cc000, 0x4000, 0x7fb0bfacc000) = -16 (Device or resource busy) Do you have any idea what's going wrong here? Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 8:12 ` Andreas Hartmann @ 2012-06-06 8:46 ` Andreas Hartmann 2012-06-06 9:35 ` Andreas Hartmann 2012-06-06 16:39 ` Alex Williamson 1 sibling, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-06-06 8:46 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, kvm On Wed, 6 Jun 2012 10:12:27 +0200 Andreas Hartmann <andihartmann@01019freenet.de> wrote: > On Tue, 05 Jun 2012 18:55:42 +0200 > Andreas Hartmann <andihartmann@01019freenet.de> wrote: > > > Alex Williamson wrote: > > [...] > > > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > > > with allow_unsafe_interrupts=1 will solve it. > > > > Yes! Works now. Success!!!!! > > > > Works means: Device is seen in VM. I couldn't test it up to now, because > > I don't have any driver in the VM for this device. > > Meanwhile, I enabled the drivers in the VM for the device I passed > through. Unfortunately, it doesn't work :-(. I'm getting this entry in > messages at the moment, the module rt2800pci is used by hostapd: > > Jun 6 09:25:02 host kernel: [ 201.895812] irq 21: nobody cared (try booting with the "irqpoll" option) > Jun 6 09:25:02 host kernel: [ 201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6 > Jun 6 09:25:02 host kernel: [ 201.895822] Call Trace: > Jun 6 09:25:02 host kernel: [ 201.895836] <IRQ> [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0 > Jun 6 09:25:02 host kernel: [ 201.895842] [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220 > Jun 6 09:25:02 host kernel: [ 201.895849] [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270 > Jun 6 09:25:02 host kernel: [ 201.895855] [<ffffffff810d14c9>] handle_irq_event+0x49/0x70 > Jun 6 09:25:02 host kernel: [ 201.895860] [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130 > Jun 6 09:25:02 host kernel: [ 201.895865] [<ffffffff81004460>] handle_irq+0x20/0x30 > Jun 6 09:25:02 host kernel: [ 201.895869] [<ffffffff81004098>] do_IRQ+0x58/0xe0 > Jun 6 09:25:02 host kernel: [ 201.895876] [<ffffffff815f112a>] common_interrupt+0x6a/0x6a > Jun 6 09:25:02 host kernel: [ 201.895907] <EOI> [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor] > Jun 6 09:25:02 host kernel: [ 201.895915] [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20 > Jun 6 09:25:02 host kernel: [ 201.895929] [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor] > Jun 6 09:25:02 host kernel: [ 201.895939] [<ffffffff814915f9>] cpuidle_enter+0x19/0x20 > Jun 6 09:25:02 host kernel: [ 201.895943] [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0 > Jun 6 09:25:02 host kernel: [ 201.895949] [<ffffffff8100bd45>] cpu_idle+0x85/0xd0 > Jun 6 09:25:02 host kernel: [ 201.895955] [<ffffffff815e63d5>] start_secondary+0x8a/0x8c > Jun 6 09:25:02 host kernel: [ 201.895958] handlers: > Jun 6 09:25:02 host kernel: [ 201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci] > Jun 6 09:25:02 host kernel: [ 201.895969] Disabling IRQ #21 > > I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared > interrupt! Did I miss an option during kernel configuration? No device at all here listens for IRQ 21. The WLAN-device is at IRQ 5 (no shared IRQ)! [...] > lspci -vs7 > 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI > Subsystem: Linksys Device 0067 > Flags: bus master, slow devsel, latency 32, IRQ 5 > Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K] > Capabilities: [40] Power Management version 3 > Kernel driver in use: vfio-pci What's going on here? Where does interrupt 21 come from? It should be 5! Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 8:46 ` Andreas Hartmann @ 2012-06-06 9:35 ` Andreas Hartmann 0 siblings, 0 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-06 9:35 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, kvm Andreas Hartmann wrote: > On Wed, 6 Jun 2012 10:12:27 +0200 > Andreas Hartmann <andihartmann@01019freenet.de> wrote: > >> On Tue, 05 Jun 2012 18:55:42 +0200 >> Andreas Hartmann <andihartmann@01019freenet.de> wrote: >> >>> Alex Williamson wrote: >>> [...] >>>> Yep, I think the previous suggestion about reloading vfio_iommu_type1 >>>> with allow_unsafe_interrupts=1 will solve it. >>> >>> Yes! Works now. Success!!!!! >>> >>> Works means: Device is seen in VM. I couldn't test it up to now, because >>> I don't have any driver in the VM for this device. >> >> Meanwhile, I enabled the drivers in the VM for the device I passed >> through. Unfortunately, it doesn't work :-(. I'm getting this entry in >> messages at the moment, the module rt2800pci is used by hostapd: >> >> Jun 6 09:25:02 host kernel: [ 201.895812] irq 21: nobody cared (try booting with the "irqpoll" option) >> Jun 6 09:25:02 host kernel: [ 201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6 >> Jun 6 09:25:02 host kernel: [ 201.895822] Call Trace: >> Jun 6 09:25:02 host kernel: [ 201.895836] <IRQ> [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0 >> Jun 6 09:25:02 host kernel: [ 201.895842] [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220 >> Jun 6 09:25:02 host kernel: [ 201.895849] [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270 >> Jun 6 09:25:02 host kernel: [ 201.895855] [<ffffffff810d14c9>] handle_irq_event+0x49/0x70 >> Jun 6 09:25:02 host kernel: [ 201.895860] [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130 >> Jun 6 09:25:02 host kernel: [ 201.895865] [<ffffffff81004460>] handle_irq+0x20/0x30 >> Jun 6 09:25:02 host kernel: [ 201.895869] [<ffffffff81004098>] do_IRQ+0x58/0xe0 >> Jun 6 09:25:02 host kernel: [ 201.895876] [<ffffffff815f112a>] common_interrupt+0x6a/0x6a >> Jun 6 09:25:02 host kernel: [ 201.895907] <EOI> [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor] >> Jun 6 09:25:02 host kernel: [ 201.895915] [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20 >> Jun 6 09:25:02 host kernel: [ 201.895929] [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor] >> Jun 6 09:25:02 host kernel: [ 201.895939] [<ffffffff814915f9>] cpuidle_enter+0x19/0x20 >> Jun 6 09:25:02 host kernel: [ 201.895943] [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0 >> Jun 6 09:25:02 host kernel: [ 201.895949] [<ffffffff8100bd45>] cpu_idle+0x85/0xd0 >> Jun 6 09:25:02 host kernel: [ 201.895955] [<ffffffff815e63d5>] start_secondary+0x8a/0x8c >> Jun 6 09:25:02 host kernel: [ 201.895958] handlers: >> Jun 6 09:25:02 host kernel: [ 201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci] >> Jun 6 09:25:02 host kernel: [ 201.895969] Disabling IRQ #21 >> >> I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared >> interrupt! Did I miss an option during kernel configuration? > > No device at all here listens for IRQ 21. The WLAN-device is at IRQ 5 > (no shared IRQ)! To be more precise: after the bind procedure, the device is at IRQ 5. After starting of /usr/local/bin/qemu-system-x86_64, it's at 21. > > [...] > >> lspci -vs7 >> 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI >> Subsystem: Linksys Device 0067 >> Flags: bus master, slow devsel, latency 32, IRQ 5 >> Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K] >> Capabilities: [40] Power Management version 3 >> Kernel driver in use: vfio-pci > > What's going on here? Where does interrupt 21 come from? It should be 5! -> this is before starting of /usr/local/bin/qemu-system-x86_64. That's after starting of the VM: 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI Subsystem: Linksys Device 0067 Flags: bus master, slow devsel, latency 32, IRQ 21 Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Kernel driver in use: vfio-pci Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 8:12 ` Andreas Hartmann 2012-06-06 8:46 ` Andreas Hartmann @ 2012-06-06 16:39 ` Alex Williamson 2012-06-06 19:17 ` Andreas Hartmann 1 sibling, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-06 16:39 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg.Roedel, kvm On Wed, 2012-06-06 at 10:12 +0200, Andreas Hartmann wrote: > On Tue, 05 Jun 2012 18:55:42 +0200 > Andreas Hartmann <andihartmann@01019freenet.de> wrote: > > > Alex Williamson wrote: > > [...] > > > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > > > with allow_unsafe_interrupts=1 will solve it. > > > > Yes! Works now. Success!!!!! > > > > Works means: Device is seen in VM. I couldn't test it up to now, because > > I don't have any driver in the VM for this device. > > Meanwhile, I enabled the drivers in the VM for the device I passed > through. Unfortunately, it doesn't work :-(. I'm getting this entry in > messages at the moment, the module rt2800pci is used by hostapd: > > Jun 6 09:25:02 host kernel: [ 201.895812] irq 21: nobody cared (try booting with the "irqpoll" option) > Jun 6 09:25:02 host kernel: [ 201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6 > Jun 6 09:25:02 host kernel: [ 201.895822] Call Trace: > Jun 6 09:25:02 host kernel: [ 201.895836] <IRQ> [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0 > Jun 6 09:25:02 host kernel: [ 201.895842] [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220 > Jun 6 09:25:02 host kernel: [ 201.895849] [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270 > Jun 6 09:25:02 host kernel: [ 201.895855] [<ffffffff810d14c9>] handle_irq_event+0x49/0x70 > Jun 6 09:25:02 host kernel: [ 201.895860] [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130 > Jun 6 09:25:02 host kernel: [ 201.895865] [<ffffffff81004460>] handle_irq+0x20/0x30 > Jun 6 09:25:02 host kernel: [ 201.895869] [<ffffffff81004098>] do_IRQ+0x58/0xe0 > Jun 6 09:25:02 host kernel: [ 201.895876] [<ffffffff815f112a>] common_interrupt+0x6a/0x6a > Jun 6 09:25:02 host kernel: [ 201.895907] <EOI> [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor] > Jun 6 09:25:02 host kernel: [ 201.895915] [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20 > Jun 6 09:25:02 host kernel: [ 201.895929] [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor] > Jun 6 09:25:02 host kernel: [ 201.895939] [<ffffffff814915f9>] cpuidle_enter+0x19/0x20 > Jun 6 09:25:02 host kernel: [ 201.895943] [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0 > Jun 6 09:25:02 host kernel: [ 201.895949] [<ffffffff8100bd45>] cpu_idle+0x85/0xd0 > Jun 6 09:25:02 host kernel: [ 201.895955] [<ffffffff815e63d5>] start_secondary+0x8a/0x8c > Jun 6 09:25:02 host kernel: [ 201.895958] handlers: > Jun 6 09:25:02 host kernel: [ 201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci] > Jun 6 09:25:02 host kernel: [ 201.895969] Disabling IRQ #21 If there's nothing else on irq 21, this might indicate that we're using the wrong mechanism to disable interrupts from the device. Please try the debug patch below and report what you get for the printk in dmesg and whether or not the problem goes away. This may be a candidate for a device that needs to be blacklisted from reporting that it supports pci 2.3 interrupt disabling. Please also report 'sudo lspci -vvvxxx -s 06:07.0'. Thanks, Alex diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 1e5315c..1aa45b5 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -49,7 +49,8 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev) if (ret) goto out; - vdev->pci_2_3 = pci_intx_mask_supported(pdev); + printk("%s(%s) supports intx mask: %d\n", __func__, dev_name(&vdev->pdev->dev), pci_intx_mask_supported(pdev)); + //vdev->pci_2_3 = pci_intx_mask_supported(pdev); pci_read_config_word(pdev, PCI_COMMAND, &cmd); if (vdev->pci_2_3 && (cmd & PCI_COMMAND_INTX_DISABLE)) { ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 16:39 ` Alex Williamson @ 2012-06-06 19:17 ` Andreas Hartmann 0 siblings, 0 replies; 30+ messages in thread From: Andreas Hartmann @ 2012-06-06 19:17 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, kvm On Wed, 06 Jun 2012 10:39:30 -0600 Alex Williamson <alex.williamson@redhat.com> wrote: > On Wed, 2012-06-06 at 10:12 +0200, Andreas Hartmann wrote: > > On Tue, 05 Jun 2012 18:55:42 +0200 > > Andreas Hartmann <andihartmann@01019freenet.de> wrote: > > > > > Alex Williamson wrote: > > > [...] > > > > Yep, I think the previous suggestion about reloading vfio_iommu_type1 > > > > with allow_unsafe_interrupts=1 will solve it. > > > > > > Yes! Works now. Success!!!!! > > > > > > Works means: Device is seen in VM. I couldn't test it up to now, because > > > I don't have any driver in the VM for this device. > > > > Meanwhile, I enabled the drivers in the VM for the device I passed > > through. Unfortunately, it doesn't work :-(. I'm getting this entry in > > messages at the moment, the module rt2800pci is used by hostapd: > > > > Jun 6 09:25:02 host kernel: [ 201.895812] irq 21: nobody cared (try booting with the "irqpoll" option) > > Jun 6 09:25:02 host kernel: [ 201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6 > > Jun 6 09:25:02 host kernel: [ 201.895822] Call Trace: > > Jun 6 09:25:02 host kernel: [ 201.895836] <IRQ> [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0 > > Jun 6 09:25:02 host kernel: [ 201.895842] [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220 > > Jun 6 09:25:02 host kernel: [ 201.895849] [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270 > > Jun 6 09:25:02 host kernel: [ 201.895855] [<ffffffff810d14c9>] handle_irq_event+0x49/0x70 > > Jun 6 09:25:02 host kernel: [ 201.895860] [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130 > > Jun 6 09:25:02 host kernel: [ 201.895865] [<ffffffff81004460>] handle_irq+0x20/0x30 > > Jun 6 09:25:02 host kernel: [ 201.895869] [<ffffffff81004098>] do_IRQ+0x58/0xe0 > > Jun 6 09:25:02 host kernel: [ 201.895876] [<ffffffff815f112a>] common_interrupt+0x6a/0x6a > > Jun 6 09:25:02 host kernel: [ 201.895907] <EOI> [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor] > > Jun 6 09:25:02 host kernel: [ 201.895915] [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20 > > Jun 6 09:25:02 host kernel: [ 201.895929] [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor] > > Jun 6 09:25:02 host kernel: [ 201.895939] [<ffffffff814915f9>] cpuidle_enter+0x19/0x20 > > Jun 6 09:25:02 host kernel: [ 201.895943] [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0 > > Jun 6 09:25:02 host kernel: [ 201.895949] [<ffffffff8100bd45>] cpu_idle+0x85/0xd0 > > Jun 6 09:25:02 host kernel: [ 201.895955] [<ffffffff815e63d5>] start_secondary+0x8a/0x8c > > Jun 6 09:25:02 host kernel: [ 201.895958] handlers: > > Jun 6 09:25:02 host kernel: [ 201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci] > > Jun 6 09:25:02 host kernel: [ 201.895969] Disabling IRQ #21 > > If there's nothing else on irq 21, this might indicate that we're using > the wrong mechanism to disable interrupts from the device. Please try > the debug patch below and report what you get for the printk in dmesg > and whether or not the problem goes away. Jun 6 21:05:43 host kernel: [ 186.133235] vfio_pci_enable(0000:06:07.0) supports intx mask: 1 The device works fine with this patch :-) !! > This may be a candidate for a > device that needs to be blacklisted from reporting that it supports pci > 2.3 interrupt disabling. Please also report 'sudo lspci -vvvxxx -s > 06:07.0'. 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI Subsystem: Linksys Device 0067 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 32 (500ns min, 1000ns max), Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 21 Region 0: Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: vfio-pci 00: 14 18 01 06 17 00 10 04 00 00 80 02 10 20 00 00 10: 00 00 8e fd 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 01 80 00 00 37 17 67 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 05 01 02 04 40: 01 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > Thanks, > > Alex I've to say thank you! If you have the final patch, I would like to test it again! Kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 14:27 ` Alex Williamson 2012-06-05 15:17 ` Andreas Hartmann @ 2012-06-06 10:11 ` Joerg Roedel 2012-06-25 5:55 ` Andreas Hartmann 2012-07-11 14:26 ` Andreas Hartmann 1 sibling, 2 replies; 30+ messages in thread From: Joerg Roedel @ 2012-06-06 10:11 UTC (permalink / raw) To: Alex Williamson; +Cc: Andreas Hartmann, Chris Sanders, kvm On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote: > Joerg, the question is whether the multifunction device above allows > peer-to-peer between functions that could bypass the iommu. If not, we > can make it the first entry in device specific acs enabled function I > proposed: > > https://lkml.org/lkml/2012/5/30/438 > > and it would greatly simplify assigning PCI devices on these systems > with VFIO. Thanks, Hm, good question. I will ask around and let you know what I find out. Regards, Joerg -- AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 10:11 ` Joerg Roedel @ 2012-06-25 5:55 ` Andreas Hartmann 2012-06-25 11:22 ` Joerg Roedel 2012-07-11 14:26 ` Andreas Hartmann 1 sibling, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-06-25 5:55 UTC (permalink / raw) To: Joerg Roedel; +Cc: Alex Williamson, Chris Sanders, kvm Hello Joerg, Joerg Roedel wrote: > On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote: >> Joerg, the question is whether the multifunction device above allows >> peer-to-peer between functions that could bypass the iommu. If not, we >> can make it the first entry in device specific acs enabled function I >> proposed: >> >> https://lkml.org/lkml/2012/5/30/438 >> >> and it would greatly simplify assigning PCI devices on these systems >> with VFIO. Thanks, > > Hm, good question. I will ask around and let you know what I find out. May I please ask, if you meanwhile could get any information about potential peer-to-peer communication between the functions of the following multifunction device: 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) 00:14.0 0c05: 1002:4385 (rev 42) 00:14.1 0101: 1002:439c (rev 40) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-25 5:55 ` Andreas Hartmann @ 2012-06-25 11:22 ` Joerg Roedel 0 siblings, 0 replies; 30+ messages in thread From: Joerg Roedel @ 2012-06-25 11:22 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Chris Sanders, kvm On Mon, Jun 25, 2012 at 07:55:55AM +0200, Andreas Hartmann wrote: > Hello Joerg, > > Joerg Roedel wrote: > > On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote: > >> Joerg, the question is whether the multifunction device above allows > >> peer-to-peer between functions that could bypass the iommu. If not, we > >> can make it the first entry in device specific acs enabled function I > >> proposed: > >> > >> https://lkml.org/lkml/2012/5/30/438 > >> > >> and it would greatly simplify assigning PCI devices on these systems > >> with VFIO. Thanks, > > > > Hm, good question. I will ask around and let you know what I find out. > > May I please ask, if you meanwhile could get any information about > potential peer-to-peer communication between the functions of the > following multifunction device: > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > 00:14.0 0c05: 1002:4385 (rev 42) > 00:14.1 0101: 1002:439c (rev 40) > 00:14.2 0403: 1002:4383 (rev 40) > 00:14.3 0601: 1002:439d (rev 40) > 00:14.4 0604: 1002:4384 (rev 40) > 00:14.5 0c03: 1002:4399 I asked the 'supposed-to-be-resonsible-person' but didn't get a resonse so far. I will ping them again. Joerg -- AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 10:11 ` Joerg Roedel 2012-06-25 5:55 ` Andreas Hartmann @ 2012-07-11 14:26 ` Andreas Hartmann 2012-07-11 16:58 ` Joerg Roedel 1 sibling, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-07-11 14:26 UTC (permalink / raw) To: Joerg Roedel; +Cc: Alex Williamson, Chris Sanders, kvm Hello Joerg, Joerg Roedel wrote: > On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote: >> Joerg, the question is whether the multifunction device above allows >> peer-to-peer between functions that could bypass the iommu. If not, we >> can make it the first entry in device specific acs enabled function I >> proposed: >> >> https://lkml.org/lkml/2012/5/30/438 >> >> and it would greatly simplify assigning PCI devices on these systems >> with VFIO. Thanks, > > Hm, good question. I will ask around and let you know what I find out. ping :-) May I please ask, if you meanwhile could get any information about potential peer-to-peer communication between the functions of the following multifunction device: 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) 00:14.0 0c05: 1002:4385 (rev 42) 00:14.1 0101: 1002:439c (rev 40) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 Thanks, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-07-11 14:26 ` Andreas Hartmann @ 2012-07-11 16:58 ` Joerg Roedel 2012-07-11 19:32 ` Andreas Hartmann 0 siblings, 1 reply; 30+ messages in thread From: Joerg Roedel @ 2012-07-11 16:58 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Chris Sanders, kvm Hi Andreas, On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote: > May I please ask, if you meanwhile could get any information about > potential peer-to-peer communication between the functions of the > following multifunction device: Good news: I actually found the right person to ask and the answer is, that peer-to-peer communication between these devices is not possible. So it is safe to pass them through to guests individually. Regards, Joerg -- AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-07-11 16:58 ` Joerg Roedel @ 2012-07-11 19:32 ` Andreas Hartmann 2012-07-11 20:01 ` Alex Williamson 0 siblings, 1 reply; 30+ messages in thread From: Andreas Hartmann @ 2012-07-11 19:32 UTC (permalink / raw) To: Joerg Roedel, Alex Williamson; +Cc: Chris Sanders, kvm Hello Joerg, Joerg Roedel wrote: > Hi Andreas, > > On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote: >> May I please ask, if you meanwhile could get any information about >> potential peer-to-peer communication between the functions of the >> following multifunction device: > > Good news: I actually found the right person to ask and the answer is, > that peer-to-peer communication between these devices is not > possible. So it is safe to pass them through to guests > individually. This is really good news! Thank you very much for your support! This makes this chipset really useful for users who want to passthrough PCI devices to a VM without loosing other functions used on the host. Your information confirms the result of the individual tests I already did. @Alex: Could you please send this patch upstream now: http://news.gmane.org/find-root.php?group=gmane.linux.kernel&article=1310198 Thank you, kind regards, Andreas ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-07-11 19:32 ` Andreas Hartmann @ 2012-07-11 20:01 ` Alex Williamson 0 siblings, 0 replies; 30+ messages in thread From: Alex Williamson @ 2012-07-11 20:01 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Joerg Roedel, Chris Sanders, kvm On Wed, 2012-07-11 at 21:32 +0200, Andreas Hartmann wrote: > Hello Joerg, > > Joerg Roedel wrote: > > Hi Andreas, > > > > On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote: > >> May I please ask, if you meanwhile could get any information about > >> potential peer-to-peer communication between the functions of the > >> following multifunction device: > > > > Good news: I actually found the right person to ask and the answer is, > > that peer-to-peer communication between these devices is not > > possible. So it is safe to pass them through to guests > > individually. Thanks Joerg! > This is really good news! Thank you very much for your support! This > makes this chipset really useful for users who want to passthrough PCI > devices to a VM without loosing other functions used on the host. > > Your information confirms the result of the individual tests I already did. > > > @Alex: > Could you please send this patch upstream now: > > http://news.gmane.org/find-root.php?group=gmane.linux.kernel&article=1310198 Sure. Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-05 10:39 ` Andreas Hartmann 2012-06-05 14:27 ` Alex Williamson @ 2012-06-06 1:32 ` sheng qiu 2012-06-06 3:07 ` Chris Sanders 1 sibling, 1 reply; 30+ messages in thread From: sheng qiu @ 2012-06-06 1:32 UTC (permalink / raw) To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, Chris Sanders, kvm test On Tue, Jun 5, 2012 at 5:39 AM, Andreas Hartmann <andihartmann@01019freenet.de> wrote: > Hello Alex, > > thanks for your efforts! > > Maybe, you already know that I'm suffering the same problem :-( with a > GA-990XA-UD3 (990X chipset). > > > On Mon, 04 Jun 2012 21:44:05 -0600 > Alex Williamson <alex.williamson@redhat.com> wrote: > [...] >> I have a setup with an AMD 990FX system and a spare PVR-350 card that I >> installed to reproduce. The sad answer is that it's nearly impossible >> to assign PCI devices on these systems due to the aliasing of devices >> below the PCIe-to-PCI bridge (PCIe devices are much, much easier to >> assign). If you boot with amd_iommu_dump, you'll see some output like >> this: >> >> AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 >> AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > here: > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > This means according to your description, the following devices have to > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 > > > These are the following additional devices: > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > > Among them is the sound device and the USB device - no good idea to > disable them on a desktop. > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > I'm getting errors after rebooting the system (the filesystems haven't > been cleanly unmounted during shutdown). > > Anyway, I would have tested it, but I'm getting a compile error while > compiling qemu. It complains about missing pci/header.h while > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > should it come from? > > [...] > >> The downside is that VFIO is strict about >> multifunction devices supporting ACS to prevent peer-to-peer between >> domains, so will require all of the 14.x devices to be bound to pci-stub >> as well. On my system, this includes an smbus controller, audio device, >> lpc controller, and usb device. If AMD could confirm this device >> doesn't allow peer-to-peer between functions, we could relax this >> requirement a bit. > > Please Joerg, could you take a look at this problem? > > > Thanks, > kind regards, > Andreas > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Sheng Qiu Texas A & M University Room 332B Wisenbaker email: herbert1984106@gmail.com College Station, TX 77843-3259 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 1:32 ` sheng qiu @ 2012-06-06 3:07 ` Chris Sanders 2012-06-06 3:25 ` Alex Williamson 0 siblings, 1 reply; 30+ messages in thread From: Chris Sanders @ 2012-06-06 3:07 UTC (permalink / raw) To: Alex Williamson; +Cc: kvm Alex, This solution appears to be working for me as well on a GA-970-D3. I've been able to pass the Hauppauge 350 through to a Windows XP guest which has installed drivers and loaded it into my recording software (SageTV). I've got a few more things I want to clean up on my qemu start command but I'm very close to where I need to be now. Once I get a few more configurations finished I'll record some shows for a final test of functionality. Thank you for your time in assisting me, the vfio drivers are working exactly as expected so far. Chris ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 3:07 ` Chris Sanders @ 2012-06-06 3:25 ` Alex Williamson 2012-06-06 3:31 ` Chris Sanders 0 siblings, 1 reply; 30+ messages in thread From: Alex Williamson @ 2012-06-06 3:25 UTC (permalink / raw) To: Chris Sanders; +Cc: kvm On Tue, 2012-06-05 at 22:07 -0500, Chris Sanders wrote: > Alex, > > This solution appears to be working for me as well on a GA-970-D3. > > I've been able to pass the Hauppauge 350 through to a Windows XP guest > which has installed drivers and loaded it into my recording software > (SageTV). I've got a few more things I want to clean up on my qemu > start command but I'm very close to where I need to be now. > > Once I get a few more configurations finished I'll record some shows > for a final test of functionality. > > Thank you for your time in assisting me, the vfio drivers are working > exactly as expected so far. Great, glad to hear it. You can expect performance to get better once we get this upstream. Interrupts are currently getting bounced through Qemu which adds overhead. Eventually we'll accelerate them through KVM. I'm still able to play static directly from the device using mplayer, so it doesn't seem to get in the way. Stay tuned for vfio upstream. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 3:25 ` Alex Williamson @ 2012-06-06 3:31 ` Chris Sanders 2012-06-06 5:27 ` Alex Williamson 0 siblings, 1 reply; 30+ messages in thread From: Chris Sanders @ 2012-06-06 3:31 UTC (permalink / raw) To: Alex Williamson; +Cc: kvm > Great, glad to hear it. You can expect performance to get better once > we get this upstream. Interrupts are currently getting bounced through > Qemu which adds overhead. Eventually we'll accelerate them through KVM. > I'm still able to play static directly from the device using mplayer, so > it doesn't seem to get in the way. Stay tuned for vfio upstream. > Thanks, > > Alex > What would be the best way to stay up to date on the status of vfio? Subscribe to this mailing list, or something else? Chris ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: AMD KVM Pci Passthrough reports device busy 2012-06-06 3:31 ` Chris Sanders @ 2012-06-06 5:27 ` Alex Williamson 0 siblings, 0 replies; 30+ messages in thread From: Alex Williamson @ 2012-06-06 5:27 UTC (permalink / raw) To: Chris Sanders; +Cc: kvm On Tue, 2012-06-05 at 22:31 -0500, Chris Sanders wrote: > > Great, glad to hear it. You can expect performance to get better once > > we get this upstream. Interrupts are currently getting bounced through > > Qemu which adds overhead. Eventually we'll accelerate them through KVM. > > I'm still able to play static directly from the device using mplayer, so > > it doesn't seem to get in the way. Stay tuned for vfio upstream. > > Thanks, > > > > Alex > > > > What would be the best way to stay up to date on the status of vfio? > Subscribe to this mailing list, or something else? Yes, probably subscribe here and just filter vfio in the subject. Thanks, Alex ^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2012-07-11 20:02 UTC | newest] Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-06-04 21:11 AMD KVM Pci Passthrough reports device busy Chris Sanders 2012-06-05 3:44 ` Alex Williamson 2012-06-05 10:39 ` Andreas Hartmann 2012-06-05 14:27 ` Alex Williamson 2012-06-05 15:17 ` Andreas Hartmann 2012-06-05 15:48 ` Alex Williamson 2012-06-05 15:58 ` Andreas Hartmann 2012-06-05 16:19 ` Alex Williamson 2012-06-05 16:55 ` Andreas Hartmann 2012-06-05 18:43 ` Alex Williamson 2012-06-05 20:37 ` Andreas Hartmann 2012-06-05 21:09 ` Alex Williamson 2012-06-05 22:02 ` Andreas Hartmann 2012-06-06 8:12 ` Andreas Hartmann 2012-06-06 8:46 ` Andreas Hartmann 2012-06-06 9:35 ` Andreas Hartmann 2012-06-06 16:39 ` Alex Williamson 2012-06-06 19:17 ` Andreas Hartmann 2012-06-06 10:11 ` Joerg Roedel 2012-06-25 5:55 ` Andreas Hartmann 2012-06-25 11:22 ` Joerg Roedel 2012-07-11 14:26 ` Andreas Hartmann 2012-07-11 16:58 ` Joerg Roedel 2012-07-11 19:32 ` Andreas Hartmann 2012-07-11 20:01 ` Alex Williamson 2012-06-06 1:32 ` sheng qiu 2012-06-06 3:07 ` Chris Sanders 2012-06-06 3:25 ` Alex Williamson 2012-06-06 3:31 ` Chris Sanders 2012-06-06 5:27 ` Alex Williamson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.