All of lore.kernel.org
 help / color / mirror / Atom feed
* AMD KVM Pci Passthrough reports device busy
@ 2012-06-04 21:11 Chris Sanders
  2012-06-05  3:44 ` Alex Williamson
  0 siblings, 1 reply; 30+ messages in thread
From: Chris Sanders @ 2012-06-04 21:11 UTC (permalink / raw)
  To: kvm

Hello, I've been working for several days trying to get Pci
Passthrough to work.  So far the #kvm IRC channel has helped me with a
few suggestions, though that hasn't yet solved the problem.  I'm
running CentOS 6.2 and was suggested I try compiling 3.2.18 kernel
form kernel.org.  This has changed a few of the messages but the guest
still fails to start.

Grepping for AMD-VI produces:
# dmesg | grep AMD-Vi
AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Lazy IO/TLB flushing enabled

After boot I'm running the following script
echo "unbind pci-pci bridge"
echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id
echo "unbind pci device"
echo "0000:03:07.0" > /sys/bus/pci/drivers/ivtv/unbind
echo "4444 0803" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:03:07.0 > /sys/bus/pci/devices/0000\:03\:07.0/driver/unbind
echo 0000:03:07.0 > /sys/bus/pci/drivers/pci-stub/bind

This is lspci -n showing my device behind the Pci-Pci bridge
-[0000:00]-+-00.0
           +-00.2
           +-02.0-[01]--+-00.0
           |            \-00.1
           +-09.0-[02]----00.0
           +-11.0
           +-12.0
           +-12.2
           +-13.0
           +-13.2
           +-14.0
           +-14.1
           +-14.3
           +-14.4-[03]----07.0
           +-14.5
           +-15.0-[04]--
           +-16.0
           +-16.2
           +-18.0
           +-18.1
           +-18.2
           +-18.3
           +-18.4
           \-18.5

My kvm command and error are:
# /usr/libexec/qemu-kvm -m 3048 -net none -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/dev/vg_hdd/lv_sagetv,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device pci-assign,host=03:07.0
Failed to assign device "(null)" : Device or resource busy
qemu-kvm: -device pci-assign,host=03:07.0: Device 'pci-assign' could
not be initialized

dmesg after attempting to start shows:
fuse init (API version 7.17)
ivtv 0000:03:07.0: BAR 0: can't reserve [mem 0xf8000000-0xfbffffff pref]
kvm_vm_ioctl_assign_device: Could not get access to device regions
tda9887 5-0043: destroying instance
tuner-simple 5-0061: destroying instance
ivtv 0000:03:07.0: PCI INT A disabled
ivtv: Removed Hauppauge WinTV PVR-350
pci-stub 0000:03:07.0: claimed by stub
pci-stub 0000:03:07.0: claimed by stub
pci-stub 0000:03:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
pci-stub 0000:03:07.0: restoring config space at offset 0x1 (was
0x2100000, writing 0x2100002)
assign device 0:3:7.0 failed
pci-stub 0000:03:07.0: PCI INT A disabled

This is on the
3.2.18 kernel
amd_iommu=1
clear_emulator_capabilities = 0
relaxed_acs_check = 1

Any help would be greatly appreciated.  I've just recently bought this
hardware specifically for this purpose and if the motherboard has a
bad IOMMU implementation I'd like to figure it out in time to exchange
it.  This is an GA-970A-D3

Thanks,
Chris
P.S. I'm not subscribed to the list so if this list requires is be
sure to cc me on responses (sanders.chris@gmail.com)

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-04 21:11 AMD KVM Pci Passthrough reports device busy Chris Sanders
@ 2012-06-05  3:44 ` Alex Williamson
  2012-06-05 10:39   ` Andreas Hartmann
  0 siblings, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-05  3:44 UTC (permalink / raw)
  To: Chris Sanders; +Cc: kvm

On Mon, 2012-06-04 at 16:11 -0500, Chris Sanders wrote:
> Hello, I've been working for several days trying to get Pci
> Passthrough to work.  So far the #kvm IRC channel has helped me with a
> few suggestions, though that hasn't yet solved the problem.  I'm
> running CentOS 6.2 and was suggested I try compiling 3.2.18 kernel
> form kernel.org.  This has changed a few of the messages but the guest
> still fails to start.
> 
> Grepping for AMD-VI produces:
> # dmesg | grep AMD-Vi
> AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> AMD-Vi: Lazy IO/TLB flushing enabled
> 
> After boot I'm running the following script
> echo "unbind pci-pci bridge"
> echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "unbind pci device"
> echo "0000:03:07.0" > /sys/bus/pci/drivers/ivtv/unbind
> echo "4444 0803" > /sys/bus/pci/drivers/pci-stub/new_id
> echo 0000:03:07.0 > /sys/bus/pci/devices/0000\:03\:07.0/driver/unbind
> echo 0000:03:07.0 > /sys/bus/pci/drivers/pci-stub/bind
> 
> This is lspci -n showing my device behind the Pci-Pci bridge
> -[0000:00]-+-00.0
>            +-00.2
>            +-02.0-[01]--+-00.0
>            |            \-00.1
>            +-09.0-[02]----00.0
>            +-11.0
>            +-12.0
>            +-12.2
>            +-13.0
>            +-13.2
>            +-14.0
>            +-14.1
>            +-14.3
>            +-14.4-[03]----07.0
>            +-14.5
>            +-15.0-[04]--
>            +-16.0
>            +-16.2
>            +-18.0
>            +-18.1
>            +-18.2
>            +-18.3
>            +-18.4
>            \-18.5
> 
> My kvm command and error are:
> # /usr/libexec/qemu-kvm -m 3048 -net none -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> file=/dev/vg_hdd/lv_sagetv,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> -device pci-assign,host=03:07.0
> Failed to assign device "(null)" : Device or resource busy
> qemu-kvm: -device pci-assign,host=03:07.0: Device 'pci-assign' could
> not be initialized

I have a setup with an AMD 990FX system and a spare PVR-350 card that I
installed to reproduce.  The sad answer is that it's nearly impossible
to assign PCI devices on these systems due to the aliasing of devices
below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
assign).  If you boot with amd_iommu_dump, you'll see some output like
this:

AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7

This says the devices on bus 5 (my bus 5 is equivalent to your bus 3)
are all aliased to device 14.4:

00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])

What that means is that the IOMMU can't distinguish devices behind the
PCI-to-PCI bridge so all devices are grouped as an alias to device 14.4.
You would hopefully not care about this, you don't have any other
devices anyway.  Unfortunately amd_iommu pre-allocates IOMMU domains for
every device, so it's already allocated a domain for device 14.4 and
adds device 03:07.0 into it.  Unbinding 03:07.0 from the ivtv driver
detaches that devices from the domain, but when we go to assign it to a
guest we create a new domain.  Assigning 03:07.0 into that new domain
fails because the device is an alias for 00:14.4, which still has a
different domain.  One way to get around this would be to also assign
the bridge to the guest, but we don't support and actually reject
assigning bridges :(

This works a bit better on Intel VT-d systems because domains are
dynamically allocated.  Thus for streaming DMA, the domain is only
created when the driver attempts to setup a DMA transaction.  When the
driver is unbound, the domain is destroyed thus allowing us to setup a
new domain for device assignment.

If you don't mind running non-upstream code, VFIO is a re-write of
device assignment for Qemu that is aware of such alias problems and
actually works in this case.  The downside is that VFIO is strict about
multifunction devices supporting ACS to prevent peer-to-peer between
domains, so will require all of the 14.x devices to be bound to pci-stub
as well.  On my system, this includes an smbus controller, audio device,
lpc controller, and usb device.  If AMD could confirm this device
doesn't allow peer-to-peer between functions, we could relax this
requirement a bit.  VFIO kernel and qemu can be found here:

git://github.com/awilliam/linux-vfio.git (iommu-group-vfio-next-20120529)
git://github.com/awilliam/qemu-vfio.git (iommu-group-vfio)

See Documentation/vfio.txt for description.  The major difference is the
setup of drivers.  At a minimum, the device you want to assign to the
guest needs to be bound to the vfio-pci driver, much in the same way you
bind devices to pci-stub.  All other devices in the group should be
bound to pci-stub.  You can find the other devices in the group by
following the iommu_group link in sysfs, ex:

/sys/bus/pci/devices/0000:03:07.0/iommu_group/devices

Once you have that, simply replace pci-assign with vfio-pci on the qemu
command line.  I'm actively trying to get this code upstream and hoping
we can do it asap, so barring it getting rejected, this would hopefully
eventually make mainline.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05  3:44 ` Alex Williamson
@ 2012-06-05 10:39   ` Andreas Hartmann
  2012-06-05 14:27     ` Alex Williamson
  2012-06-06  1:32     ` sheng qiu
  0 siblings, 2 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 10:39 UTC (permalink / raw)
  To: Alex Williamson, Joerg.Roedel; +Cc: Chris Sanders, kvm

Hello Alex,

thanks for your efforts!

Maybe, you already know that I'm suffering the same problem :-( with a
GA-990XA-UD3 (990X chipset).


On Mon, 04 Jun 2012 21:44:05 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:
[...]
> I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> installed to reproduce.  The sad answer is that it's nearly impossible
> to assign PCI devices on these systems due to the aliasing of devices
> below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> assign).  If you boot with amd_iommu_dump, you'll see some output like
> this:
> 
> AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7

here:
AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7


This means according to your description, the following devices have to
be unbound here, too (0000:06:07.0 is the device I want to passthrough):

0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0


These are the following additional devices:

00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])


Among them is the sound device and the USB device - no good idea to
disable them on a desktop.

Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
I'm getting errors after rebooting the system (the filesystems haven't
been cleanly unmounted during shutdown).

Anyway, I would have tested it, but I'm getting a compile error while
compiling qemu. It complains about missing pci/header.h while
processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
should it come from?

[...]

> The downside is that VFIO is strict about
> multifunction devices supporting ACS to prevent peer-to-peer between
> domains, so will require all of the 14.x devices to be bound to pci-stub
> as well.  On my system, this includes an smbus controller, audio device,
> lpc controller, and usb device.  If AMD could confirm this device
> doesn't allow peer-to-peer between functions, we could relax this
> requirement a bit.

Please Joerg, could you take a look at this problem?


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 10:39   ` Andreas Hartmann
@ 2012-06-05 14:27     ` Alex Williamson
  2012-06-05 15:17       ` Andreas Hartmann
  2012-06-06 10:11       ` Joerg Roedel
  2012-06-06  1:32     ` sheng qiu
  1 sibling, 2 replies; 30+ messages in thread
From: Alex Williamson @ 2012-06-05 14:27 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm

On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote:
> Hello Alex,
> 
> thanks for your efforts!
> 
> Maybe, you already know that I'm suffering the same problem :-( with a
> GA-990XA-UD3 (990X chipset).

My system is a GA-990FXA-UD3, so just a slightly different version of
the chipset.

> On Mon, 04 Jun 2012 21:44:05 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> [...]
> > I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> > installed to reproduce.  The sad answer is that it's nearly impossible
> > to assign PCI devices on these systems due to the aliasing of devices
> > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> > assign).  If you boot with amd_iommu_dump, you'll see some output like
> > this:
> > 
> > AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> > AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7
> 
> here:
> AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> 
> 
> This means according to your description, the following devices have to
> be unbound here, too (0000:06:07.0 is the device I want to passthrough):
> 
> 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0

Yep.

> These are the following additional devices:
> 
> 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])

My system shows this for lspci -n -s 14.

00:14.0 0c05: 1002:4385 (rev 42)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399

Can you confirm yours matches and fill in 14.1 so we have the
vendor:device IDs for these.

> Among them is the sound device and the USB device - no good idea to
> disable them on a desktop.

You can always blacklist drivers if the device is unique or unbind
devices in an rc.local type script.  I'm hoping that Joerg or someone
else from AMD can tell us this devices does not allow internal
peer-to-peer between functions.  Then we can add it to the device
specific ACS checks and you wouldn't need to worry about displacing all
the other 14.x functions.

> Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> I'm getting errors after rebooting the system (the filesystems haven't
> been cleanly unmounted during shutdown).

Did you have a disk off that controller?  My system doesn't expose
function 1, but is the same otherwise.

> Anyway, I would have tested it, but I'm getting a compile error while
> compiling qemu. It complains about missing pci/header.h while
> processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> should it come from?

It is from pciutils-devel for me.  I'll see what I'm getting out of
there and try to remove it.

> > The downside is that VFIO is strict about
> > multifunction devices supporting ACS to prevent peer-to-peer between
> > domains, so will require all of the 14.x devices to be bound to pci-stub
> > as well.  On my system, this includes an smbus controller, audio device,
> > lpc controller, and usb device.  If AMD could confirm this device
> > doesn't allow peer-to-peer between functions, we could relax this
> > requirement a bit.
> 
> Please Joerg, could you take a look at this problem?

Joerg, the question is whether the multifunction device above allows
peer-to-peer between functions that could bypass the iommu.  If not, we
can make it the first entry in device specific acs enabled function I
proposed:

https://lkml.org/lkml/2012/5/30/438

and it would greatly simplify assigning PCI devices on these systems
with VFIO.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 14:27     ` Alex Williamson
@ 2012-06-05 15:17       ` Andreas Hartmann
  2012-06-05 15:48         ` Alex Williamson
  2012-06-05 15:58         ` Andreas Hartmann
  2012-06-06 10:11       ` Joerg Roedel
  1 sibling, 2 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 15:17 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm

On Tue, 05 Jun 2012 08:27:05 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote:
> > Hello Alex,
> > 
> > thanks for your efforts!
> > 
> > Maybe, you already know that I'm suffering the same problem :-( with a
> > GA-990XA-UD3 (990X chipset).
> 
> My system is a GA-990FXA-UD3, so just a slightly different version of
> the chipset.
> 
> > On Mon, 04 Jun 2012 21:44:05 -0600
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> > [...]
> > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> > > installed to reproduce.  The sad answer is that it's nearly impossible
> > > to assign PCI devices on these systems due to the aliasing of devices
> > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> > > assign).  If you boot with amd_iommu_dump, you'll see some output like
> > > this:
> > > 
> > > AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> > > AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7
> > 
> > here:
> > AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> > AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> > 
> > 
> > This means according to your description, the following devices have to
> > be unbound here, too (0000:06:07.0 is the device I want to passthrough):
> > 
> > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0
> 
> Yep.
> 
> > These are the following additional devices:
> > 
> > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
> 
> My system shows this for lspci -n -s 14.
> 
> 00:14.0 0c05: 1002:4385 (rev 42)
> 00:14.2 0403: 1002:4383 (rev 40)
> 00:14.3 0601: 1002:439d (rev 40)
> 00:14.4 0604: 1002:4384 (rev 40)
> 00:14.5 0c03: 1002:4399

My system:
00:14.0 0c05: 1002:4385 (rev 42)
00:14.1 0101: 1002:439c (rev 40)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399

You miss 00.14.1!? But the others are equal.


> Can you confirm yours matches and fill in 14.1 so we have the
> vendor:device IDs for these.

Yes.

> > Among them is the sound device and the USB device - no good idea to
> > disable them on a desktop.
> 
> You can always blacklist drivers if the device is unique or unbind
> devices in an rc.local type script.  I'm hoping that Joerg or someone
> else from AMD can tell us this devices does not allow internal
> peer-to-peer between functions.  Then we can add it to the device
> specific ACS checks and you wouldn't need to worry about displacing all
> the other 14.x functions.

This would be cool.

> > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> > I'm getting errors after rebooting the system (the filesystems haven't
> > been cleanly unmounted during shutdown).
> 
> Did you have a disk off that controller?  My system doesn't expose
> function 1, but is the same otherwise.

Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the
reason is another?

> > Anyway, I would have tested it, but I'm getting a compile error while
> > compiling qemu. It complains about missing pci/header.h while
> > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> > should it come from?
> 
> It is from pciutils-devel for me.  I'll see what I'm getting out of
> there and try to remove it.

Meanwhile I found it :-). Thanks for your hint!

> > > The downside is that VFIO is strict about
> > > multifunction devices supporting ACS to prevent peer-to-peer between
> > > domains, so will require all of the 14.x devices to be bound to pci-stub
> > > as well.  On my system, this includes an smbus controller, audio device,
> > > lpc controller, and usb device.  If AMD could confirm this device
> > > doesn't allow peer-to-peer between functions, we could relax this
> > > requirement a bit.
> > 
> > Please Joerg, could you take a look at this problem?
> 
> Joerg, the question is whether the multifunction device above allows
> peer-to-peer between functions that could bypass the iommu.  If not, we
> can make it the first entry in device specific acs enabled function I
> proposed:
> 
> https://lkml.org/lkml/2012/5/30/438
> 
> and it would greatly simplify assigning PCI devices on these systems
> with VFIO.  Thanks,

I tried to run qemu-system-x86_64 but got this error on startup:

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9

qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
**
ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)


I started qemu-system-x86_64 with this option among others 

-device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5

after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.


Could you please tell me, why the operation isn't permitted? I started
qemu-system-x86_64 as root.



Thanks,
kind regards,
Andreas Hartmann

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 15:17       ` Andreas Hartmann
@ 2012-06-05 15:48         ` Alex Williamson
  2012-06-05 15:58         ` Andreas Hartmann
  1 sibling, 0 replies; 30+ messages in thread
From: Alex Williamson @ 2012-06-05 15:48 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm

On Tue, 2012-06-05 at 17:17 +0200, Andreas Hartmann wrote:
> On Tue, 05 Jun 2012 08:27:05 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> 
> > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote:
> > > Hello Alex,
> > > 
> > > thanks for your efforts!
> > > 
> > > Maybe, you already know that I'm suffering the same problem :-( with a
> > > GA-990XA-UD3 (990X chipset).
> > 
> > My system is a GA-990FXA-UD3, so just a slightly different version of
> > the chipset.
> > 
> > > On Mon, 04 Jun 2012 21:44:05 -0600
> > > Alex Williamson <alex.williamson@redhat.com> wrote:
> > > [...]
> > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I
> > > > installed to reproduce.  The sad answer is that it's nearly impossible
> > > > to assign PCI devices on these systems due to the aliasing of devices
> > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
> > > > assign).  If you boot with amd_iommu_dump, you'll see some output like
> > > > this:
> > > > 
> > > > AMD-Vi:   DEV_ALIAS_RANGE		 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> > > > AMD-Vi:   DEV_RANGE_END		 devid: 05:1f.7
> > > 
> > > here:
> > > AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> > > AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> > > 
> > > 
> > > This means according to your description, the following devices have to
> > > be unbound here, too (0000:06:07.0 is the device I want to passthrough):
> > > 
> > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0
> > 
> > Yep.
> > 
> > > These are the following additional devices:
> > > 
> > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
> > 
> > My system shows this for lspci -n -s 14.
> > 
> > 00:14.0 0c05: 1002:4385 (rev 42)
> > 00:14.2 0403: 1002:4383 (rev 40)
> > 00:14.3 0601: 1002:439d (rev 40)
> > 00:14.4 0604: 1002:4384 (rev 40)
> > 00:14.5 0c03: 1002:4399
> 
> My system:
> 00:14.0 0c05: 1002:4385 (rev 42)
> 00:14.1 0101: 1002:439c (rev 40)
> 00:14.2 0403: 1002:4383 (rev 40)
> 00:14.3 0601: 1002:439d (rev 40)
> 00:14.4 0604: 1002:4384 (rev 40)
> 00:14.5 0c03: 1002:4399
> 
> You miss 00.14.1!? But the others are equal.

Yeah, my system doesn't expose the IDE, *shrug*

> > Can you confirm yours matches and fill in 14.1 so we have the
> > vendor:device IDs for these.
> 
> Yes.

Thanks

> > > Among them is the sound device and the USB device - no good idea to
> > > disable them on a desktop.
> > 
> > You can always blacklist drivers if the device is unique or unbind
> > devices in an rc.local type script.  I'm hoping that Joerg or someone
> > else from AMD can tell us this devices does not allow internal
> > peer-to-peer between functions.  Then we can add it to the device
> > specific ACS checks and you wouldn't need to worry about displacing all
> > the other 14.x functions.
> 
> This would be cool.
> 
> > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> > > I'm getting errors after rebooting the system (the filesystems haven't
> > > been cleanly unmounted during shutdown).
> > 
> > Did you have a disk off that controller?  My system doesn't expose
> > function 1, but is the same otherwise.
> 
> Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the
> reason is another?
> 
> > > Anyway, I would have tested it, but I'm getting a compile error while
> > > compiling qemu. It complains about missing pci/header.h while
> > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> > > should it come from?
> > 
> > It is from pciutils-devel for me.  I'll see what I'm getting out of
> > there and try to remove it.
> 
> Meanwhile I found it :-). Thanks for your hint!

Great.

> > > > The downside is that VFIO is strict about
> > > > multifunction devices supporting ACS to prevent peer-to-peer between
> > > > domains, so will require all of the 14.x devices to be bound to pci-stub
> > > > as well.  On my system, this includes an smbus controller, audio device,
> > > > lpc controller, and usb device.  If AMD could confirm this device
> > > > doesn't allow peer-to-peer between functions, we could relax this
> > > > requirement a bit.
> > > 
> > > Please Joerg, could you take a look at this problem?
> > 
> > Joerg, the question is whether the multifunction device above allows
> > peer-to-peer between functions that could bypass the iommu.  If not, we
> > can make it the first entry in device specific acs enabled function I
> > proposed:
> > 
> > https://lkml.org/lkml/2012/5/30/438
> > 
> > and it would greatly simplify assigning PCI devices on these systems
> > with VFIO.  Thanks,
> 
> I tried to run qemu-system-x86_64 but got this error on startup:
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
> **
> ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)
> 
> 
> I started qemu-system-x86_64 with this option among others 
> 
> -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5
> 
> after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.
> 
> 
> Could you please tell me, why the operation isn't permitted? I started
> qemu-system-x86_64 as root.

Check dmesg, it's probably complaining about no interrupt remapping
support.  If so:

# modprobe -r vfio_iommu_type1
# modprobe vfio_iommu_type1 allow_unsafe_interrupts=1

Then try again.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 15:17       ` Andreas Hartmann
  2012-06-05 15:48         ` Alex Williamson
@ 2012-06-05 15:58         ` Andreas Hartmann
  2012-06-05 16:19           ` Alex Williamson
  1 sibling, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 15:58 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, Chris Sanders, kvm

Andreas Hartmann wrote:
[...]
> I tried to run qemu-system-x86_64 but got this error on startup:
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9
> 
> qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
> **
> ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)
> 
> 
> I started qemu-system-x86_64 with this option among others 
> 
> -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5
> 
> after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.
> 
> 
> Could you please tell me, why the operation isn't permitted? I started
> qemu-system-x86_64 as root.

I straced the call with strace and got the following error:

...
8048  open("/usr/local/share/qemu/pxe-virtio.rom", O_RDONLY) = 14
8048  lseek(14, 0, SEEK_END)            = 60416
8048  lseek(14, 0, SEEK_SET)            = 0
8048  read(14, "U\252v\351\217\0z\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\177\0
\0<\0\365\274\266\16"..., 60416) = 60416
8048  close(14)                         = 0
8048  stat("/sys/bus/pci/devices/0000:06:07.0/", {st_mode=S_IFDIR|0755,
st_size=0, ...}) = 0
8048  readlink("/sys/bus/pci/devices/0000:06:07.0/iommu_group",
"../../../../kernel/iommu_groups/9"..., 4096) = 33
8048  open("/dev/vfio/9", O_RDWR)       = 14
8048  ioctl(14, 0x3b67, 0x7fff237d5ac0) = 0
8048  open("/dev/vfio/vfio", O_RDWR)    = 15
8048  ioctl(15, 0x3b64, 0xf)            = 0
8048  ioctl(15, 0x3b65, 0x1)            = 1
8048  ioctl(14, 0x3b68, 0x7fff237d5ad8) = 0
8048  ioctl(15, 0x3b66, 0x1)            = -1 EPERM (Operation not permitted)
...

/dev/vfio # ls -l
total 0
crw-rw-rw- 1 root root 250, 1 Jun  5 17:42 9
crw-rw-rw- 1 root root 250, 0 Jun  5 17:41 vfio



Kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 15:58         ` Andreas Hartmann
@ 2012-06-05 16:19           ` Alex Williamson
  2012-06-05 16:55             ` Andreas Hartmann
  0 siblings, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-05 16:19 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm

On Tue, 2012-06-05 at 17:58 +0200, Andreas Hartmann wrote:
> Andreas Hartmann wrote:
> [...]
> > I tried to run qemu-system-x86_64 but got this error on startup:
> > 
> > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
> > 
> > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9
> > 
> > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9
> > **
> > ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0)
> > 
> > 
> > I started qemu-system-x86_64 with this option among others 
> > 
> > -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5
> > 
> > after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci.
> > 
> > 
> > Could you please tell me, why the operation isn't permitted? I started
> > qemu-system-x86_64 as root.
> 
> I straced the call with strace and got the following error:
> 
> ...
> 8048  open("/usr/local/share/qemu/pxe-virtio.rom", O_RDONLY) = 14
> 8048  lseek(14, 0, SEEK_END)            = 60416
> 8048  lseek(14, 0, SEEK_SET)            = 0
> 8048  read(14, "U\252v\351\217\0z\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\177\0
> \0<\0\365\274\266\16"..., 60416) = 60416
> 8048  close(14)                         = 0
> 8048  stat("/sys/bus/pci/devices/0000:06:07.0/", {st_mode=S_IFDIR|0755,
> st_size=0, ...}) = 0
> 8048  readlink("/sys/bus/pci/devices/0000:06:07.0/iommu_group",
> "../../../../kernel/iommu_groups/9"..., 4096) = 33
> 8048  open("/dev/vfio/9", O_RDWR)       = 14
> 8048  ioctl(14, 0x3b67, 0x7fff237d5ac0) = 0
> 8048  open("/dev/vfio/vfio", O_RDWR)    = 15
> 8048  ioctl(15, 0x3b64, 0xf)            = 0
> 8048  ioctl(15, 0x3b65, 0x1)            = 1
> 8048  ioctl(14, 0x3b68, 0x7fff237d5ad8) = 0
> 8048  ioctl(15, 0x3b66, 0x1)            = -1 EPERM (Operation not permitted)
> ...

Yep, I think the previous suggestion about reloading vfio_iommu_type1
with allow_unsafe_interrupts=1 will solve it.  It has nothing to do with
file permissions, you're getting EPERM at the point where we set the
iommu type, which enables access to devices.  By default we want an
iommu which protects against malicious MSI attacks, which requires
interrupt remapping on x86.  Joerg has been working to add this for AMD
since the hardware supports it, but for now, the above options lets us
bypass the check.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 16:19           ` Alex Williamson
@ 2012-06-05 16:55             ` Andreas Hartmann
  2012-06-05 18:43               ` Alex Williamson
  2012-06-06  8:12               ` Andreas Hartmann
  0 siblings, 2 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 16:55 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm

Alex Williamson wrote:
[...]
> Yep, I think the previous suggestion about reloading vfio_iommu_type1
> with allow_unsafe_interrupts=1 will solve it.

Yes! Works now. Success!!!!!

Works means: Device is seen in VM. I couldn't test it up to now, because
I don't have any driver in the VM for this device.

> It has nothing to do with
> file permissions, you're getting EPERM at the point where we set the
> iommu type, which enables access to devices.  By default we want an
> iommu which protects against malicious MSI attacks, which requires
> interrupt remapping on x86.  Joerg has been working to add this for AMD
> since the hardware supports it, but for now, the above options lets us
> bypass the check.  Thanks,

Now, I have to check my problem with the filesystem first, which isn't
cleanly unmounted. I suspect a few missing options for kernel compile.
Next will be fglrx :-) for 3.4

Hmmm, is there a chance to get your extension isolated to get it
compiled for 3.1, too?


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 16:55             ` Andreas Hartmann
@ 2012-06-05 18:43               ` Alex Williamson
  2012-06-05 20:37                 ` Andreas Hartmann
  2012-06-06  8:12               ` Andreas Hartmann
  1 sibling, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-05 18:43 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm

On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote:
> Alex Williamson wrote:
> [...]
> > Yep, I think the previous suggestion about reloading vfio_iommu_type1
> > with allow_unsafe_interrupts=1 will solve it.
> 
> Yes! Works now. Success!!!!!
> 
> Works means: Device is seen in VM. I couldn't test it up to now, because
> I don't have any driver in the VM for this device.

I tested it as far as doing 'cat /dev/video0 > /tmp/test'.  The file
grows, the device gets interrupts and `file` thinks it's a video (no
input connected to the card).

> > It has nothing to do with
> > file permissions, you're getting EPERM at the point where we set the
> > iommu type, which enables access to devices.  By default we want an
> > iommu which protects against malicious MSI attacks, which requires
> > interrupt remapping on x86.  Joerg has been working to add this for AMD
> > since the hardware supports it, but for now, the above options lets us
> > bypass the check.  Thanks,
> 
> Now, I have to check my problem with the filesystem first, which isn't
> cleanly unmounted. I suspect a few missing options for kernel compile.
> Next will be fglrx :-) for 3.4
> 
> Hmmm, is there a chance to get your extension isolated to get it
> compiled for 3.1, too?

The patches are all layered on top of that branch, so you should be able
to easily 'git show' to get patches or cherry-pick them across trees.  I
don't think there should be too much trouble backporting them.  Thanks,

Alex



^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 18:43               ` Alex Williamson
@ 2012-06-05 20:37                 ` Andreas Hartmann
  2012-06-05 21:09                   ` Alex Williamson
  0 siblings, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 20:37 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm

Alex Williamson wrote:
> On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote:
>> Alex Williamson wrote:
>> [...]
>>> Yep, I think the previous suggestion about reloading vfio_iommu_type1
>>> with allow_unsafe_interrupts=1 will solve it.
>>
>> Yes! Works now. Success!!!!!
>>
>> Works means: Device is seen in VM. I couldn't test it up to now, because
>> I don't have any driver in the VM for this device.

Well, I've got another problem now with 3.4: I can't pass through my
PCIe device any more, which works fine without any problem with 3.1.10.
I'm getting the following error in 3.4:

Failed to assign irq for "hostdev0": Input/output error
Perhaps you are assigning a device that shares an IRQ with another device?
qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized

options kvm allow_unsafe_assigned_interrupts=1 is set.

There are share IRQ's - but that's the same in 3.1.10.

This VM is started with libvirt (virsh) the old fashioned way with the
old qemu-kvm tool. Doesn't this work any more?

BTW: I think libvirt / virsh isn't aware yet of the new vfio way, isn't
it?


Thanks,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 20:37                 ` Andreas Hartmann
@ 2012-06-05 21:09                   ` Alex Williamson
  2012-06-05 22:02                     ` Andreas Hartmann
  0 siblings, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-05 21:09 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, Chris Sanders, kvm

On Tue, 2012-06-05 at 22:37 +0200, Andreas Hartmann wrote:
> Alex Williamson wrote:
> > On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote:
> >> Alex Williamson wrote:
> >> [...]
> >>> Yep, I think the previous suggestion about reloading vfio_iommu_type1
> >>> with allow_unsafe_interrupts=1 will solve it.
> >>
> >> Yes! Works now. Success!!!!!
> >>
> >> Works means: Device is seen in VM. I couldn't test it up to now, because
> >> I don't have any driver in the VM for this device.
> 
> Well, I've got another problem now with 3.4: I can't pass through my
> PCIe device any more, which works fine without any problem with 3.1.10.
> I'm getting the following error in 3.4:
> 
> Failed to assign irq for "hostdev0": Input/output error
> Perhaps you are assigning a device that shares an IRQ with another device?
> qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized
> 
> options kvm allow_unsafe_assigned_interrupts=1 is set.
> 
> There are share IRQ's - but that's the same in 3.1.10.
> 
> This VM is started with libvirt (virsh) the old fashioned way with the
> old qemu-kvm tool. Doesn't this work any more?

Does this help: https://lkml.org/lkml/2012/6/1/261

Otherwise, dmesg and device info please.

> BTW: I think libvirt / virsh isn't aware yet of the new vfio way, isn't
> it?

Correct, need to get it into the kernel and qemu first.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 21:09                   ` Alex Williamson
@ 2012-06-05 22:02                     ` Andreas Hartmann
  0 siblings, 0 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-05 22:02 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, Chris Sanders, kvm

Alex Williamson wrote:
> On Tue, 2012-06-05 at 22:37 +0200, Andreas Hartmann wrote:
>> Alex Williamson wrote:
>>> On Tue, 2012-06-05 at 18:55 +0200, Andreas Hartmann wrote:
>>>> Alex Williamson wrote:
>>>> [...]
>>>>> Yep, I think the previous suggestion about reloading vfio_iommu_type1
>>>>> with allow_unsafe_interrupts=1 will solve it.
>>>>
>>>> Yes! Works now. Success!!!!!
>>>>
>>>> Works means: Device is seen in VM. I couldn't test it up to now, because
>>>> I don't have any driver in the VM for this device.
>>
>> Well, I've got another problem now with 3.4: I can't pass through my
>> PCIe device any more, which works fine without any problem with 3.1.10.
>> I'm getting the following error in 3.4:
>>
>> Failed to assign irq for "hostdev0": Input/output error
>> Perhaps you are assigning a device that shares an IRQ with another device?
>> qemu-kvm: -device pci-assign,host=04:00.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized
>>
>> options kvm allow_unsafe_assigned_interrupts=1 is set.
>>
>> There are shared IRQ's - but that's the same in 3.1.10.
>>
>> This VM is started with libvirt (virsh) the old fashioned way with the
>> old qemu-kvm tool. Doesn't this work any more?
> 
> Does this help: https://lkml.org/lkml/2012/6/1/261

Yes - this fixed the problem!


Thanks,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 10:39   ` Andreas Hartmann
  2012-06-05 14:27     ` Alex Williamson
@ 2012-06-06  1:32     ` sheng qiu
  2012-06-06  3:07       ` Chris Sanders
  1 sibling, 1 reply; 30+ messages in thread
From: sheng qiu @ 2012-06-06  1:32 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, Chris Sanders, kvm

test

On Tue, Jun 5, 2012 at 5:39 AM, Andreas Hartmann
<andihartmann@01019freenet.de> wrote:
> Hello Alex,
>
> thanks for your efforts!
>
> Maybe, you already know that I'm suffering the same problem :-( with a
> GA-990XA-UD3 (990X chipset).
>
>
> On Mon, 04 Jun 2012 21:44:05 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> [...]
>> I have a setup with an AMD 990FX system and a spare PVR-350 card that I
>> installed to reproduce.  The sad answer is that it's nearly impossible
>> to assign PCI devices on these systems due to the aliasing of devices
>> below the PCIe-to-PCI bridge (PCIe devices are much, much easier to
>> assign).  If you boot with amd_iommu_dump, you'll see some output like
>> this:
>>
>> AMD-Vi:   DEV_ALIAS_RANGE              devid: 05:00.0 flags: 00 devid_to: 00:14.4
>> AMD-Vi:   DEV_RANGE_END                devid: 05:1f.7
>
> here:
> AMD-Vi:   DEV_ALIAS_RANGE         devid: 06:00.0 flags: 00 devid_to: 00:14.4
> AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
>
>
> This means according to your description, the following devices have to
> be unbound here, too (0000:06:07.0 is the device I want to passthrough):
>
> 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
> 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
> 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
> 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
> 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
> 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
> 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0
>
>
> These are the following additional devices:
>
> 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
>
>
> Among them is the sound device and the USB device - no good idea to
> disable them on a desktop.
>
> Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but
> I'm getting errors after rebooting the system (the filesystems haven't
> been cleanly unmounted during shutdown).
>
> Anyway, I would have tested it, but I'm getting a compile error while
> compiling qemu. It complains about missing pci/header.h while
> processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where
> should it come from?
>
> [...]
>
>> The downside is that VFIO is strict about
>> multifunction devices supporting ACS to prevent peer-to-peer between
>> domains, so will require all of the 14.x devices to be bound to pci-stub
>> as well.  On my system, this includes an smbus controller, audio device,
>> lpc controller, and usb device.  If AMD could confirm this device
>> doesn't allow peer-to-peer between functions, we could relax this
>> requirement a bit.
>
> Please Joerg, could you take a look at this problem?
>
>
> Thanks,
> kind regards,
> Andreas
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  1:32     ` sheng qiu
@ 2012-06-06  3:07       ` Chris Sanders
  2012-06-06  3:25         ` Alex Williamson
  0 siblings, 1 reply; 30+ messages in thread
From: Chris Sanders @ 2012-06-06  3:07 UTC (permalink / raw)
  To: Alex Williamson; +Cc: kvm

Alex,

This solution appears to be working for me as well on a GA-970-D3.

I've been able to pass the Hauppauge 350 through to a Windows XP guest
which has installed drivers and loaded it into my recording software
(SageTV).  I've got a few more things I want to clean up on my qemu
start command but I'm very close to where I need to be now.

Once I get a few more configurations finished I'll record some shows
for a final test of functionality.

Thank you for your time in assisting me, the vfio drivers are working
exactly as expected so far.

Chris

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  3:07       ` Chris Sanders
@ 2012-06-06  3:25         ` Alex Williamson
  2012-06-06  3:31           ` Chris Sanders
  0 siblings, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-06  3:25 UTC (permalink / raw)
  To: Chris Sanders; +Cc: kvm

On Tue, 2012-06-05 at 22:07 -0500, Chris Sanders wrote:
> Alex,
> 
> This solution appears to be working for me as well on a GA-970-D3.
> 
> I've been able to pass the Hauppauge 350 through to a Windows XP guest
> which has installed drivers and loaded it into my recording software
> (SageTV).  I've got a few more things I want to clean up on my qemu
> start command but I'm very close to where I need to be now.
> 
> Once I get a few more configurations finished I'll record some shows
> for a final test of functionality.
> 
> Thank you for your time in assisting me, the vfio drivers are working
> exactly as expected so far.

Great, glad to hear it.  You can expect performance to get better once
we get this upstream.  Interrupts are currently getting bounced through
Qemu which adds overhead.  Eventually we'll accelerate them through KVM.
I'm still able to play static directly from the device using mplayer, so
it doesn't seem to get in the way.  Stay tuned for vfio upstream.
Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  3:25         ` Alex Williamson
@ 2012-06-06  3:31           ` Chris Sanders
  2012-06-06  5:27             ` Alex Williamson
  0 siblings, 1 reply; 30+ messages in thread
From: Chris Sanders @ 2012-06-06  3:31 UTC (permalink / raw)
  To: Alex Williamson; +Cc: kvm

> Great, glad to hear it.  You can expect performance to get better once
> we get this upstream.  Interrupts are currently getting bounced through
> Qemu which adds overhead.  Eventually we'll accelerate them through KVM.
> I'm still able to play static directly from the device using mplayer, so
> it doesn't seem to get in the way.  Stay tuned for vfio upstream.
> Thanks,
>
> Alex
>

What would be the best way to stay up to date on the status of vfio?
Subscribe to this mailing list, or something else?

Chris

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  3:31           ` Chris Sanders
@ 2012-06-06  5:27             ` Alex Williamson
  0 siblings, 0 replies; 30+ messages in thread
From: Alex Williamson @ 2012-06-06  5:27 UTC (permalink / raw)
  To: Chris Sanders; +Cc: kvm

On Tue, 2012-06-05 at 22:31 -0500, Chris Sanders wrote:
> > Great, glad to hear it.  You can expect performance to get better once
> > we get this upstream.  Interrupts are currently getting bounced through
> > Qemu which adds overhead.  Eventually we'll accelerate them through KVM.
> > I'm still able to play static directly from the device using mplayer, so
> > it doesn't seem to get in the way.  Stay tuned for vfio upstream.
> > Thanks,
> >
> > Alex
> >
> 
> What would be the best way to stay up to date on the status of vfio?
> Subscribe to this mailing list, or something else?

Yes, probably subscribe here and just filter vfio in the subject.
Thanks,

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 16:55             ` Andreas Hartmann
  2012-06-05 18:43               ` Alex Williamson
@ 2012-06-06  8:12               ` Andreas Hartmann
  2012-06-06  8:46                 ` Andreas Hartmann
  2012-06-06 16:39                 ` Alex Williamson
  1 sibling, 2 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-06  8:12 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, kvm

On Tue, 05 Jun 2012 18:55:42 +0200
Andreas Hartmann <andihartmann@01019freenet.de> wrote:

> Alex Williamson wrote:
> [...]
> > Yep, I think the previous suggestion about reloading vfio_iommu_type1
> > with allow_unsafe_interrupts=1 will solve it.
> 
> Yes! Works now. Success!!!!!
> 
> Works means: Device is seen in VM. I couldn't test it up to now, because
> I don't have any driver in the VM for this device.

Meanwhile, I enabled the drivers in the VM for the device I passed
through. Unfortunately, it doesn't work :-(. I'm getting this entry in
messages at the moment, the module rt2800pci is used by hostapd:

Jun  6 09:25:02 host kernel: [  201.895812] irq 21: nobody cared (try booting with the "irqpoll" option)
Jun  6 09:25:02 host kernel: [  201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6
Jun  6 09:25:02 host kernel: [  201.895822] Call Trace:
Jun  6 09:25:02 host kernel: [  201.895836]  <IRQ>  [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0
Jun  6 09:25:02 host kernel: [  201.895842]  [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220
Jun  6 09:25:02 host kernel: [  201.895849]  [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270
Jun  6 09:25:02 host kernel: [  201.895855]  [<ffffffff810d14c9>] handle_irq_event+0x49/0x70
Jun  6 09:25:02 host kernel: [  201.895860]  [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130
Jun  6 09:25:02 host kernel: [  201.895865]  [<ffffffff81004460>] handle_irq+0x20/0x30
Jun  6 09:25:02 host kernel: [  201.895869]  [<ffffffff81004098>] do_IRQ+0x58/0xe0
Jun  6 09:25:02 host kernel: [  201.895876]  [<ffffffff815f112a>] common_interrupt+0x6a/0x6a
Jun  6 09:25:02 host kernel: [  201.895907]  <EOI>  [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor]
Jun  6 09:25:02 host kernel: [  201.895915]  [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
Jun  6 09:25:02 host kernel: [  201.895929]  [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor]
Jun  6 09:25:02 host kernel: [  201.895939]  [<ffffffff814915f9>] cpuidle_enter+0x19/0x20
Jun  6 09:25:02 host kernel: [  201.895943]  [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0
Jun  6 09:25:02 host kernel: [  201.895949]  [<ffffffff8100bd45>] cpu_idle+0x85/0xd0
Jun  6 09:25:02 host kernel: [  201.895955]  [<ffffffff815e63d5>] start_secondary+0x8a/0x8c
Jun  6 09:25:02 host kernel: [  201.895958] handlers:
Jun  6 09:25:02 host kernel: [  201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci]
Jun  6 09:25:02 host kernel: [  201.895969] Disabling IRQ #21

I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared
interrupt! Did I miss an option during kernel configuration?


Just to remember:
The passed through device is a WLAN-device and it's driven by rt2800pci:
06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI

I unbound all the devices in the same group:

echo "1002 4385" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.0 > /sys/bus/pci/devices/0000:00:14.0/driver/unbind
echo 0000:00:14.0 > /sys/bus/pci/drivers/pci-stub/bind

echo "1002 439c " > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.1 > /sys/bus/pci/devices/0000:00:14.1/driver/unbind
echo 0000:00:14.1 > /sys/bus/pci/drivers/pci-stub/bind

echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.2 > /sys/bus/pci/devices/0000:00:14.2/driver/unbind
echo 0000:00:14.2 > /sys/bus/pci/drivers/pci-stub/bind

echo "1002 439d" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.3 > /sys/bus/pci/devices/0000:00:14.3/driver/unbind
echo 0000:00:14.3 > /sys/bus/pci/drivers/pci-stub/bind

echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.4 > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/bind

echo "1002 4399" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.5 > /sys/bus/pci/devices/0000:00:14.5/driver/unbind
echo 0000:00:14.5 > /sys/bus/pci/drivers/pci-stub/bind

modprobe vfio-pci
echo "1814 0601" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:06:07.0 > /sys/bus/pci/devices/0000:06:07.0/driver/unbind
echo 0000:06:07.0 > /sys/bus/pci/drivers/vfio-pci/bind


The corresponding entries in messages are:

Jun  6 09:23:01 host kernel: [   81.777453] pci-stub 0000:00:14.0: claimed by stub
Jun  6 09:23:01 host kernel: [   81.777788] pci-stub 0000:00:14.0: claimed by stub
Jun  6 09:23:02 host kernel: [   81.782465] pci-stub 0000:00:14.1: claimed by stub
Jun  6 09:23:02 host kernel: [   81.801572] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.0 domain=0x0000 address=0x000000fdf9103300 flags=0x0600]
Jun  6 09:23:02 host kernel: [   81.950388] pci-stub 0000:00:14.2: claimed by stub
Jun  6 09:23:02 host kernel: [   81.950505] pci-stub 0000:00:14.3: claimed by stub
Jun  6 09:23:02 host kernel: [   81.950723] pci-stub 0000:00:14.3: claimed by stub
Jun  6 09:23:02 host kernel: [   81.950828] pci-stub 0000:00:14.4: claimed by stub
Jun  6 09:23:02 host kernel: [   81.951031] pci-stub 0000:00:14.4: claimed by stub
Jun  6 09:23:02 host kernel: [   81.951280] ohci_hcd 0000:00:14.5: remove, state 4
Jun  6 09:23:02 host kernel: [   81.951293] usb usb6: USB disconnect, device number 1
Jun  6 09:23:02 host kernel: [   81.951840] ohci_hcd 0000:00:14.5: USB bus 6 deregistered
Jun  6 09:23:02 host kernel: [   81.951993] pci-stub 0000:00:14.5: claimed by stub
Jun  6 09:23:02 host kernel: [   81.957543] VFIO - User Level meta-driver version: 0.3
Jun  6 09:23:02 host kernel: [   81.960441] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 bound to driver vfio-pci
Jun  6 09:23:02 host kernel: [   81.960617] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 unbinding from driver vfio-pci
Jun  6 09:23:02 host kernel: [   81.961002] vfio_iommu_group_notifier: Device 0000:06:07.0, group 9 bound to driver vfio-pci

The result with lspci -vs14 is:

00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
        Flags: 66MHz, medium devsel
        Kernel driver in use: pci-stub

00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
        Subsystem: Giga-byte Technology Device 5002
        Flags: 66MHz, medium devsel, IRQ 17
        I/O ports at 01f0 [size=8]
        I/O ports at 03f4 [size=1]
        I/O ports at 0170 [size=8]
        I/O ports at 0374 [size=1]
        I/O ports at fa00 [size=16]
        Kernel driver in use: pci-stub

00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
        Subsystem: Giga-byte Technology Device a132
        Flags: slow devsel, IRQ 16
        Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
        Kernel driver in use: pci-stub

00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller
        Flags: bus master, 66MHz, medium devsel, latency 0
        Kernel driver in use: pci-stub

00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
        Flags: bus master, VGA palette snoop, 66MHz, medium devsel, latency 64
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=64
        I/O behind bridge: 00009000-00009fff
        Memory behind bridge: fd800000-fd8fffff
        Prefetchable memory behind bridge: fd700000-fd7fffff
        Kernel driver in use: pci-stub

00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
        Subsystem: Giga-byte Technology Device 5004
        Flags: 66MHz, medium devsel, IRQ 18
        Memory at fdffa000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: pci-stub


lspci -vs7
06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI
        Subsystem: Linksys Device 0067
        Flags: bus master, slow devsel, latency 32, IRQ 5
        Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
        Kernel driver in use: vfio-pci


On starting the VM, the following entries came up in messages:

Jun  6 09:23:51 host kernel: [  131.397055] kvm: 8175: cpu0 unhandled rdmsr: 0xc0010001

The VM is started as root with this command:

  /usr/local/bin/qemu-system-x86_64 \
  -enable-kvm \
  -m 1024 \
  -rtc base=utc \
  -drive file=/my.hd,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
  -netdev tap,ifname=tap5,script=no,id=hostnet0 \
  -device virtio-net-pci,netdev=hostnet0,id=net0,mac=12:34:56:78:90:12,bus=pci.0,addr=0x3 \
  -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5

qemu-system-x86_64 prints out the following on startup:
Warning, device 0000:06:07.0 does not support reset (-> only the first time the VM is started)

sometimes:
qemu-system-x86_64: vfio_dma_map(0x7fb10a848170, 0x00000000000cc000, 0x4000, 0x7fb0bfacc000) = -16 (Device or resource busy)

qemu-system-x86_64: vfio_dma_map(0x7fb10a848170, 0x00000000000cc000, 0x4000, 0x7fb0bfacc000) = -16 (Device or resource busy)



Do you have any idea what's going wrong here?


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  8:12               ` Andreas Hartmann
@ 2012-06-06  8:46                 ` Andreas Hartmann
  2012-06-06  9:35                   ` Andreas Hartmann
  2012-06-06 16:39                 ` Alex Williamson
  1 sibling, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-06  8:46 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Joerg.Roedel, kvm

On Wed, 6 Jun 2012 10:12:27 +0200
Andreas Hartmann <andihartmann@01019freenet.de> wrote:

> On Tue, 05 Jun 2012 18:55:42 +0200
> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> 
> > Alex Williamson wrote:
> > [...]
> > > Yep, I think the previous suggestion about reloading vfio_iommu_type1
> > > with allow_unsafe_interrupts=1 will solve it.
> > 
> > Yes! Works now. Success!!!!!
> > 
> > Works means: Device is seen in VM. I couldn't test it up to now, because
> > I don't have any driver in the VM for this device.
> 
> Meanwhile, I enabled the drivers in the VM for the device I passed
> through. Unfortunately, it doesn't work :-(. I'm getting this entry in
> messages at the moment, the module rt2800pci is used by hostapd:
> 
> Jun  6 09:25:02 host kernel: [  201.895812] irq 21: nobody cared (try booting with the "irqpoll" option)
> Jun  6 09:25:02 host kernel: [  201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6
> Jun  6 09:25:02 host kernel: [  201.895822] Call Trace:
> Jun  6 09:25:02 host kernel: [  201.895836]  <IRQ>  [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0
> Jun  6 09:25:02 host kernel: [  201.895842]  [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220
> Jun  6 09:25:02 host kernel: [  201.895849]  [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270
> Jun  6 09:25:02 host kernel: [  201.895855]  [<ffffffff810d14c9>] handle_irq_event+0x49/0x70
> Jun  6 09:25:02 host kernel: [  201.895860]  [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130
> Jun  6 09:25:02 host kernel: [  201.895865]  [<ffffffff81004460>] handle_irq+0x20/0x30
> Jun  6 09:25:02 host kernel: [  201.895869]  [<ffffffff81004098>] do_IRQ+0x58/0xe0
> Jun  6 09:25:02 host kernel: [  201.895876]  [<ffffffff815f112a>] common_interrupt+0x6a/0x6a
> Jun  6 09:25:02 host kernel: [  201.895907]  <EOI>  [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor]
> Jun  6 09:25:02 host kernel: [  201.895915]  [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
> Jun  6 09:25:02 host kernel: [  201.895929]  [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor]
> Jun  6 09:25:02 host kernel: [  201.895939]  [<ffffffff814915f9>] cpuidle_enter+0x19/0x20
> Jun  6 09:25:02 host kernel: [  201.895943]  [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0
> Jun  6 09:25:02 host kernel: [  201.895949]  [<ffffffff8100bd45>] cpu_idle+0x85/0xd0
> Jun  6 09:25:02 host kernel: [  201.895955]  [<ffffffff815e63d5>] start_secondary+0x8a/0x8c
> Jun  6 09:25:02 host kernel: [  201.895958] handlers:
> Jun  6 09:25:02 host kernel: [  201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci]
> Jun  6 09:25:02 host kernel: [  201.895969] Disabling IRQ #21
> 
> I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared
> interrupt! Did I miss an option during kernel configuration?

No device at all here listens for IRQ 21. The WLAN-device is at IRQ 5
(no shared IRQ)!

[...]

> lspci -vs7
> 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI
>         Subsystem: Linksys Device 0067
>         Flags: bus master, slow devsel, latency 32, IRQ 5
>         Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
>         Capabilities: [40] Power Management version 3
>         Kernel driver in use: vfio-pci

What's going on here? Where does interrupt 21 come from? It should be 5!


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  8:46                 ` Andreas Hartmann
@ 2012-06-06  9:35                   ` Andreas Hartmann
  0 siblings, 0 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-06  9:35 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, kvm

Andreas Hartmann wrote:
> On Wed, 6 Jun 2012 10:12:27 +0200
> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> 
>> On Tue, 05 Jun 2012 18:55:42 +0200
>> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
>>
>>> Alex Williamson wrote:
>>> [...]
>>>> Yep, I think the previous suggestion about reloading vfio_iommu_type1
>>>> with allow_unsafe_interrupts=1 will solve it.
>>>
>>> Yes! Works now. Success!!!!!
>>>
>>> Works means: Device is seen in VM. I couldn't test it up to now, because
>>> I don't have any driver in the VM for this device.
>>
>> Meanwhile, I enabled the drivers in the VM for the device I passed
>> through. Unfortunately, it doesn't work :-(. I'm getting this entry in
>> messages at the moment, the module rt2800pci is used by hostapd:
>>
>> Jun  6 09:25:02 host kernel: [  201.895812] irq 21: nobody cared (try booting with the "irqpoll" option)
>> Jun  6 09:25:02 host kernel: [  201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6
>> Jun  6 09:25:02 host kernel: [  201.895822] Call Trace:
>> Jun  6 09:25:02 host kernel: [  201.895836]  <IRQ>  [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0
>> Jun  6 09:25:02 host kernel: [  201.895842]  [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220
>> Jun  6 09:25:02 host kernel: [  201.895849]  [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270
>> Jun  6 09:25:02 host kernel: [  201.895855]  [<ffffffff810d14c9>] handle_irq_event+0x49/0x70
>> Jun  6 09:25:02 host kernel: [  201.895860]  [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130
>> Jun  6 09:25:02 host kernel: [  201.895865]  [<ffffffff81004460>] handle_irq+0x20/0x30
>> Jun  6 09:25:02 host kernel: [  201.895869]  [<ffffffff81004098>] do_IRQ+0x58/0xe0
>> Jun  6 09:25:02 host kernel: [  201.895876]  [<ffffffff815f112a>] common_interrupt+0x6a/0x6a
>> Jun  6 09:25:02 host kernel: [  201.895907]  <EOI>  [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor]
>> Jun  6 09:25:02 host kernel: [  201.895915]  [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
>> Jun  6 09:25:02 host kernel: [  201.895929]  [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor]
>> Jun  6 09:25:02 host kernel: [  201.895939]  [<ffffffff814915f9>] cpuidle_enter+0x19/0x20
>> Jun  6 09:25:02 host kernel: [  201.895943]  [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0
>> Jun  6 09:25:02 host kernel: [  201.895949]  [<ffffffff8100bd45>] cpu_idle+0x85/0xd0
>> Jun  6 09:25:02 host kernel: [  201.895955]  [<ffffffff815e63d5>] start_secondary+0x8a/0x8c
>> Jun  6 09:25:02 host kernel: [  201.895958] handlers:
>> Jun  6 09:25:02 host kernel: [  201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci]
>> Jun  6 09:25:02 host kernel: [  201.895969] Disabling IRQ #21
>>
>> I tried with irqpoll, but this didn't help. BTW: IRQ 21 isn't a shared
>> interrupt! Did I miss an option during kernel configuration?
> 
> No device at all here listens for IRQ 21. The WLAN-device is at IRQ 5
> (no shared IRQ)!

To be more precise:

after the bind procedure, the device is at IRQ 5. After starting of
/usr/local/bin/qemu-system-x86_64, it's at 21.

> 
> [...]
> 
>> lspci -vs7
>> 06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI
>>         Subsystem: Linksys Device 0067
>>         Flags: bus master, slow devsel, latency 32, IRQ 5
>>         Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
>>         Capabilities: [40] Power Management version 3
>>         Kernel driver in use: vfio-pci
> 
> What's going on here? Where does interrupt 21 come from? It should be 5!

-> this is before starting of /usr/local/bin/qemu-system-x86_64.

That's after starting of the VM:

06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI
        Subsystem: Linksys Device 0067
        Flags: bus master, slow devsel, latency 32, IRQ 21
        Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
        Kernel driver in use: vfio-pci




Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-05 14:27     ` Alex Williamson
  2012-06-05 15:17       ` Andreas Hartmann
@ 2012-06-06 10:11       ` Joerg Roedel
  2012-06-25  5:55         ` Andreas Hartmann
  2012-07-11 14:26         ` Andreas Hartmann
  1 sibling, 2 replies; 30+ messages in thread
From: Joerg Roedel @ 2012-06-06 10:11 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Chris Sanders, kvm

On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote:
> Joerg, the question is whether the multifunction device above allows
> peer-to-peer between functions that could bypass the iommu.  If not, we
> can make it the first entry in device specific acs enabled function I
> proposed:
> 
> https://lkml.org/lkml/2012/5/30/438
> 
> and it would greatly simplify assigning PCI devices on these systems
> with VFIO.  Thanks,

Hm, good question. I will ask around and let you know what I find out.

Regards,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06  8:12               ` Andreas Hartmann
  2012-06-06  8:46                 ` Andreas Hartmann
@ 2012-06-06 16:39                 ` Alex Williamson
  2012-06-06 19:17                   ` Andreas Hartmann
  1 sibling, 1 reply; 30+ messages in thread
From: Alex Williamson @ 2012-06-06 16:39 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg.Roedel, kvm

On Wed, 2012-06-06 at 10:12 +0200, Andreas Hartmann wrote:
> On Tue, 05 Jun 2012 18:55:42 +0200
> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> 
> > Alex Williamson wrote:
> > [...]
> > > Yep, I think the previous suggestion about reloading vfio_iommu_type1
> > > with allow_unsafe_interrupts=1 will solve it.
> > 
> > Yes! Works now. Success!!!!!
> > 
> > Works means: Device is seen in VM. I couldn't test it up to now, because
> > I don't have any driver in the VM for this device.
> 
> Meanwhile, I enabled the drivers in the VM for the device I passed
> through. Unfortunately, it doesn't work :-(. I'm getting this entry in
> messages at the moment, the module rt2800pci is used by hostapd:
> 
> Jun  6 09:25:02 host kernel: [  201.895812] irq 21: nobody cared (try booting with the "irqpoll" option)
> Jun  6 09:25:02 host kernel: [  201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6
> Jun  6 09:25:02 host kernel: [  201.895822] Call Trace:
> Jun  6 09:25:02 host kernel: [  201.895836]  <IRQ>  [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0
> Jun  6 09:25:02 host kernel: [  201.895842]  [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220
> Jun  6 09:25:02 host kernel: [  201.895849]  [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270
> Jun  6 09:25:02 host kernel: [  201.895855]  [<ffffffff810d14c9>] handle_irq_event+0x49/0x70
> Jun  6 09:25:02 host kernel: [  201.895860]  [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130
> Jun  6 09:25:02 host kernel: [  201.895865]  [<ffffffff81004460>] handle_irq+0x20/0x30
> Jun  6 09:25:02 host kernel: [  201.895869]  [<ffffffff81004098>] do_IRQ+0x58/0xe0
> Jun  6 09:25:02 host kernel: [  201.895876]  [<ffffffff815f112a>] common_interrupt+0x6a/0x6a
> Jun  6 09:25:02 host kernel: [  201.895907]  <EOI>  [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor]
> Jun  6 09:25:02 host kernel: [  201.895915]  [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
> Jun  6 09:25:02 host kernel: [  201.895929]  [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor]
> Jun  6 09:25:02 host kernel: [  201.895939]  [<ffffffff814915f9>] cpuidle_enter+0x19/0x20
> Jun  6 09:25:02 host kernel: [  201.895943]  [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0
> Jun  6 09:25:02 host kernel: [  201.895949]  [<ffffffff8100bd45>] cpu_idle+0x85/0xd0
> Jun  6 09:25:02 host kernel: [  201.895955]  [<ffffffff815e63d5>] start_secondary+0x8a/0x8c
> Jun  6 09:25:02 host kernel: [  201.895958] handlers:
> Jun  6 09:25:02 host kernel: [  201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci]
> Jun  6 09:25:02 host kernel: [  201.895969] Disabling IRQ #21

If there's nothing else on irq 21, this might indicate that we're using
the wrong mechanism to disable interrupts from the device.  Please try
the debug patch below and report what you get for the printk in dmesg
and whether or not the problem goes away.  This may be a candidate for a
device that needs to be blacklisted from reporting that it supports pci
2.3 interrupt disabling.  Please also report 'sudo lspci -vvvxxx -s
06:07.0'.  Thanks,

Alex

diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 1e5315c..1aa45b5 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -49,7 +49,8 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
 	if (ret)
 		goto out;
 
-	vdev->pci_2_3 = pci_intx_mask_supported(pdev);
+	printk("%s(%s) supports intx mask: %d\n", __func__, dev_name(&vdev->pdev->dev), pci_intx_mask_supported(pdev));
+	//vdev->pci_2_3 = pci_intx_mask_supported(pdev);
 
 	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
 	if (vdev->pci_2_3 && (cmd & PCI_COMMAND_INTX_DISABLE)) {



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06 16:39                 ` Alex Williamson
@ 2012-06-06 19:17                   ` Andreas Hartmann
  0 siblings, 0 replies; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-06 19:17 UTC (permalink / raw)
  To: Alex Williamson; +Cc: Andreas Hartmann, Joerg.Roedel, kvm

On Wed, 06 Jun 2012 10:39:30 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Wed, 2012-06-06 at 10:12 +0200, Andreas Hartmann wrote:
> > On Tue, 05 Jun 2012 18:55:42 +0200
> > Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> > 
> > > Alex Williamson wrote:
> > > [...]
> > > > Yep, I think the previous suggestion about reloading vfio_iommu_type1
> > > > with allow_unsafe_interrupts=1 will solve it.
> > > 
> > > Yes! Works now. Success!!!!!
> > > 
> > > Works means: Device is seen in VM. I couldn't test it up to now, because
> > > I don't have any driver in the VM for this device.
> > 
> > Meanwhile, I enabled the drivers in the VM for the device I passed
> > through. Unfortunately, it doesn't work :-(. I'm getting this entry in
> > messages at the moment, the module rt2800pci is used by hostapd:
> > 
> > Jun  6 09:25:02 host kernel: [  201.895812] irq 21: nobody cared (try booting with the "irqpoll" option)
> > Jun  6 09:25:02 host kernel: [  201.895819] Pid: 0, comm: swapper/1 Not tainted 3.4.0-next-20120529-16.1-desktop #6
> > Jun  6 09:25:02 host kernel: [  201.895822] Call Trace:
> > Jun  6 09:25:02 host kernel: [  201.895836]  <IRQ>  [<ffffffff810d37a8>] __report_bad_irq+0x38/0xe0
> > Jun  6 09:25:02 host kernel: [  201.895842]  [<ffffffff810d3a6d>] note_interrupt+0x16d/0x220
> > Jun  6 09:25:02 host kernel: [  201.895849]  [<ffffffff810d12d6>] handle_irq_event_percpu+0xc6/0x270
> > Jun  6 09:25:02 host kernel: [  201.895855]  [<ffffffff810d14c9>] handle_irq_event+0x49/0x70
> > Jun  6 09:25:02 host kernel: [  201.895860]  [<ffffffff810d45d2>] handle_fasteoi_irq+0x82/0x130
> > Jun  6 09:25:02 host kernel: [  201.895865]  [<ffffffff81004460>] handle_irq+0x20/0x30
> > Jun  6 09:25:02 host kernel: [  201.895869]  [<ffffffff81004098>] do_IRQ+0x58/0xe0
> > Jun  6 09:25:02 host kernel: [  201.895876]  [<ffffffff815f112a>] common_interrupt+0x6a/0x6a
> > Jun  6 09:25:02 host kernel: [  201.895907]  <EOI>  [<ffffffffa0029077>] ? arch_local_irq_enable+0x8/0xd [processor]
> > Jun  6 09:25:02 host kernel: [  201.895915]  [<ffffffff8107a37a>] ? sched_clock_idle_wakeup_event+0x1a/0x20
> > Jun  6 09:25:02 host kernel: [  201.895929]  [<ffffffffa002a046>] acpi_idle_enter_simple+0xd0/0x111 [processor]
> > Jun  6 09:25:02 host kernel: [  201.895939]  [<ffffffff814915f9>] cpuidle_enter+0x19/0x20
> > Jun  6 09:25:02 host kernel: [  201.895943]  [<ffffffff81491d81>] cpuidle_idle_call+0xc1/0x1e0
> > Jun  6 09:25:02 host kernel: [  201.895949]  [<ffffffff8100bd45>] cpu_idle+0x85/0xd0
> > Jun  6 09:25:02 host kernel: [  201.895955]  [<ffffffff815e63d5>] start_secondary+0x8a/0x8c
> > Jun  6 09:25:02 host kernel: [  201.895958] handlers:
> > Jun  6 09:25:02 host kernel: [  201.895967] [<ffffffffa0488230>] vfio_intx_handler [vfio_pci] threaded [<ffffffffa04884e0>] vfio_intx_thread [vfio_pci]
> > Jun  6 09:25:02 host kernel: [  201.895969] Disabling IRQ #21
> 
> If there's nothing else on irq 21, this might indicate that we're using
> the wrong mechanism to disable interrupts from the device.  Please try
> the debug patch below and report what you get for the printk in dmesg
> and whether or not the problem goes away.

Jun  6 21:05:43 host kernel: [  186.133235] vfio_pci_enable(0000:06:07.0) supports intx mask: 1

The device works fine with this patch :-) !!

> This may be a candidate for a
> device that needs to be blacklisted from reporting that it supports pci
> 2.3 interrupt disabling.  Please also report 'sudo lspci -vvvxxx -s
> 06:07.0'.

06:07.0 Network controller: Ralink corp. RT2800 802.11n PCI
        Subsystem: Linksys Device 0067
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32 (500ns min, 1000ns max), Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 21
        Region 0: Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: vfio-pci
00: 14 18 01 06 17 00 10 04 00 00 80 02 10 20 00 00
10: 00 00 8e fd 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 01 80 00 00 37 17 67 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 05 01 02 04
40: 01 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


> Thanks,
> 
> Alex

I've to say thank you! 
If you have the final patch, I would like to test it again!


Kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06 10:11       ` Joerg Roedel
@ 2012-06-25  5:55         ` Andreas Hartmann
  2012-06-25 11:22           ` Joerg Roedel
  2012-07-11 14:26         ` Andreas Hartmann
  1 sibling, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-06-25  5:55 UTC (permalink / raw)
  To: Joerg Roedel; +Cc: Alex Williamson, Chris Sanders, kvm

Hello Joerg,

Joerg Roedel wrote:
> On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote:
>> Joerg, the question is whether the multifunction device above allows
>> peer-to-peer between functions that could bypass the iommu.  If not, we
>> can make it the first entry in device specific acs enabled function I
>> proposed:
>>
>> https://lkml.org/lkml/2012/5/30/438
>>
>> and it would greatly simplify assigning PCI devices on these systems
>> with VFIO.  Thanks,
> 
> Hm, good question. I will ask around and let you know what I find out.

May I please ask, if you meanwhile could get any information about
potential peer-to-peer communication between the functions of the
following multifunction device:

00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])

00:14.0 0c05: 1002:4385 (rev 42)
00:14.1 0101: 1002:439c (rev 40)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-25  5:55         ` Andreas Hartmann
@ 2012-06-25 11:22           ` Joerg Roedel
  0 siblings, 0 replies; 30+ messages in thread
From: Joerg Roedel @ 2012-06-25 11:22 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Chris Sanders, kvm

On Mon, Jun 25, 2012 at 07:55:55AM +0200, Andreas Hartmann wrote:
> Hello Joerg,
> 
> Joerg Roedel wrote:
> > On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote:
> >> Joerg, the question is whether the multifunction device above allows
> >> peer-to-peer between functions that could bypass the iommu.  If not, we
> >> can make it the first entry in device specific acs enabled function I
> >> proposed:
> >>
> >> https://lkml.org/lkml/2012/5/30/438
> >>
> >> and it would greatly simplify assigning PCI devices on these systems
> >> with VFIO.  Thanks,
> > 
> > Hm, good question. I will ask around and let you know what I find out.
> 
> May I please ask, if you meanwhile could get any information about
> potential peer-to-peer communication between the functions of the
> following multifunction device:
> 
> 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
> 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
> 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])
> 
> 00:14.0 0c05: 1002:4385 (rev 42)
> 00:14.1 0101: 1002:439c (rev 40)
> 00:14.2 0403: 1002:4383 (rev 40)
> 00:14.3 0601: 1002:439d (rev 40)
> 00:14.4 0604: 1002:4384 (rev 40)
> 00:14.5 0c03: 1002:4399

I asked the 'supposed-to-be-resonsible-person' but didn't get a resonse
so far. I will ping them again.


	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-06-06 10:11       ` Joerg Roedel
  2012-06-25  5:55         ` Andreas Hartmann
@ 2012-07-11 14:26         ` Andreas Hartmann
  2012-07-11 16:58           ` Joerg Roedel
  1 sibling, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-07-11 14:26 UTC (permalink / raw)
  To: Joerg Roedel; +Cc: Alex Williamson, Chris Sanders, kvm

Hello Joerg,

Joerg Roedel wrote:
> On Tue, Jun 05, 2012 at 08:27:05AM -0600, Alex Williamson wrote:
>> Joerg, the question is whether the multifunction device above allows
>> peer-to-peer between functions that could bypass the iommu.  If not, we
>> can make it the first entry in device specific acs enabled function I
>> proposed:
>>
>> https://lkml.org/lkml/2012/5/30/438
>>
>> and it would greatly simplify assigning PCI devices on these systems
>> with VFIO.  Thanks,
> 
> Hm, good question. I will ask around and let you know what I find out.

ping :-)

May I please ask, if you meanwhile could get any information about
potential peer-to-peer communication between the functions of the
following multifunction device:

00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP])
00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI])

00:14.0 0c05: 1002:4385 (rev 42)
00:14.1 0101: 1002:439c (rev 40)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399


Thanks,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-07-11 14:26         ` Andreas Hartmann
@ 2012-07-11 16:58           ` Joerg Roedel
  2012-07-11 19:32             ` Andreas Hartmann
  0 siblings, 1 reply; 30+ messages in thread
From: Joerg Roedel @ 2012-07-11 16:58 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Alex Williamson, Chris Sanders, kvm

Hi Andreas,

On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote:
> May I please ask, if you meanwhile could get any information about
> potential peer-to-peer communication between the functions of the
> following multifunction device:

Good news: I actually found the right person to ask and the answer is,
     that peer-to-peer communication between these devices is not
     possible. So it is safe to pass them through to guests
     individually.

Regards,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-07-11 16:58           ` Joerg Roedel
@ 2012-07-11 19:32             ` Andreas Hartmann
  2012-07-11 20:01               ` Alex Williamson
  0 siblings, 1 reply; 30+ messages in thread
From: Andreas Hartmann @ 2012-07-11 19:32 UTC (permalink / raw)
  To: Joerg Roedel, Alex Williamson; +Cc: Chris Sanders, kvm

Hello Joerg,

Joerg Roedel wrote:
> Hi Andreas,
> 
> On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote:
>> May I please ask, if you meanwhile could get any information about
>> potential peer-to-peer communication between the functions of the
>> following multifunction device:
> 
> Good news: I actually found the right person to ask and the answer is,
>      that peer-to-peer communication between these devices is not
>      possible. So it is safe to pass them through to guests
>      individually.

This is really good news! Thank you very much for your support! This
makes this chipset really useful for users who want to passthrough PCI
devices to a VM without loosing other functions used on the host.

Your information confirms the result of the individual tests I already did.


@Alex:
Could you please send this patch upstream now:

http://news.gmane.org/find-root.php?group=gmane.linux.kernel&article=1310198


Thank you,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: AMD KVM Pci Passthrough reports device busy
  2012-07-11 19:32             ` Andreas Hartmann
@ 2012-07-11 20:01               ` Alex Williamson
  0 siblings, 0 replies; 30+ messages in thread
From: Alex Williamson @ 2012-07-11 20:01 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: Joerg Roedel, Chris Sanders, kvm

On Wed, 2012-07-11 at 21:32 +0200, Andreas Hartmann wrote:
> Hello Joerg,
> 
> Joerg Roedel wrote:
> > Hi Andreas,
> > 
> > On Wed, Jul 11, 2012 at 04:26:30PM +0200, Andreas Hartmann wrote:
> >> May I please ask, if you meanwhile could get any information about
> >> potential peer-to-peer communication between the functions of the
> >> following multifunction device:
> > 
> > Good news: I actually found the right person to ask and the answer is,
> >      that peer-to-peer communication between these devices is not
> >      possible. So it is safe to pass them through to guests
> >      individually.

Thanks Joerg!

> This is really good news! Thank you very much for your support! This
> makes this chipset really useful for users who want to passthrough PCI
> devices to a VM without loosing other functions used on the host.
> 
> Your information confirms the result of the individual tests I already did.
> 
> 
> @Alex:
> Could you please send this patch upstream now:
> 
> http://news.gmane.org/find-root.php?group=gmane.linux.kernel&article=1310198

Sure.

Alex


^ permalink raw reply	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2012-07-11 20:02 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-04 21:11 AMD KVM Pci Passthrough reports device busy Chris Sanders
2012-06-05  3:44 ` Alex Williamson
2012-06-05 10:39   ` Andreas Hartmann
2012-06-05 14:27     ` Alex Williamson
2012-06-05 15:17       ` Andreas Hartmann
2012-06-05 15:48         ` Alex Williamson
2012-06-05 15:58         ` Andreas Hartmann
2012-06-05 16:19           ` Alex Williamson
2012-06-05 16:55             ` Andreas Hartmann
2012-06-05 18:43               ` Alex Williamson
2012-06-05 20:37                 ` Andreas Hartmann
2012-06-05 21:09                   ` Alex Williamson
2012-06-05 22:02                     ` Andreas Hartmann
2012-06-06  8:12               ` Andreas Hartmann
2012-06-06  8:46                 ` Andreas Hartmann
2012-06-06  9:35                   ` Andreas Hartmann
2012-06-06 16:39                 ` Alex Williamson
2012-06-06 19:17                   ` Andreas Hartmann
2012-06-06 10:11       ` Joerg Roedel
2012-06-25  5:55         ` Andreas Hartmann
2012-06-25 11:22           ` Joerg Roedel
2012-07-11 14:26         ` Andreas Hartmann
2012-07-11 16:58           ` Joerg Roedel
2012-07-11 19:32             ` Andreas Hartmann
2012-07-11 20:01               ` Alex Williamson
2012-06-06  1:32     ` sheng qiu
2012-06-06  3:07       ` Chris Sanders
2012-06-06  3:25         ` Alex Williamson
2012-06-06  3:31           ` Chris Sanders
2012-06-06  5:27             ` Alex Williamson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.