From mboxrd@z Thu Jan 1 00:00:00 1970 From: James Neave Subject: Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2 Date: Wed, 23 Feb 2011 19:44:16 +0000 Message-ID: References: <20110207132641.GD2665@redhat.com> <1297705728.14733.50.camel@x201> <1298322078.5764.45.camel@x201> <20110222015119.GY9869@sequoia.sous-sol.org> <20110223001103.GZ9869@sequoia.sous-sol.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Alex Williamson , kvm@vger.kernel.org, "Roedel, Joerg" To: Chris Wright Return-path: Received: from mail-qy0-f181.google.com ([209.85.216.181]:60618 "EHLO mail-qy0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751927Ab1BWToj convert rfc822-to-8bit (ORCPT ); Wed, 23 Feb 2011 14:44:39 -0500 Received: by qyg14 with SMTP id 14so3203902qyg.19 for ; Wed, 23 Feb 2011 11:44:38 -0800 (PST) In-Reply-To: <20110223001103.GZ9869@sequoia.sous-sol.org> Sender: kvm-owner@vger.kernel.org List-ID: On Wed, Feb 23, 2011 at 12:11 AM, Chris Wright wr= ote: > * James Neave (roboj1m@gmail.com) wrote: >> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright = wrote: >> > * James Neave (roboj1m@gmail.com) wrote: >> >> Does anybody know the debug kernel switches for iommu? >> > >> > Two helpful kernel commandline options are: >> > >> > amd_iommu_dump debug (and drop "quiet") >> > >> > The problem is when you attach the device (function) you're gettin= g >> > stuck up in conflicts with the existing domain for that function. >> > >> > My guess is that all the functions are behind a PCI to PCI bridge,= so the alias >> > lookup is finding a conflict. >> >> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an >> earlier email: > > Sorry, I missed that in your original mail, thanks for reposting. > >> cat /proc/interruts >> http://pastebin.com/LQdB3hms >> >> lspci -vvv >> http://pastebin.com/GJDkC8B4 > > =C2=A000:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridg= e (rev 40) > >> lspci -t -v >> http://pastebin.com/Ftx8Hfjt > > Yup, that's what I expected: > > =C2=A0+-14.4-[08]--+-06.0 =C2=A0VIA Technologies, Inc. VT82xxxxx UHCI= USB 1.1 Controller > =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+-06.1 =C2=A0VIA Tec= hnologies, Inc. VT82xxxxx UHCI USB 1.1 Controller > =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+-06.2 =C2=A0VIA Tec= hnologies, Inc. USB 2.0 > =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0\-0e.0 =C2=A0Texas I= nstruments TSB43AB23 IEEE-1394a-2000 Controller > > I'd now expect to see (if you boot with amd_iommu_dump) some IVRS > details showing an alias range entry basically showing 08:* pointing > back to 00:14.4. =C2=A0This means that from the point of view of the = IOMMU the > devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if the= y > are 00:14.4. > > When you assign a device to a guest, the guest VM gets an IOMMU domai= n > (a context to manage IOMMU page table mappings) and the device is put > into that guest's IOMMU domain. =C2=A0However, if the device is behin= d a > PCI-PCI bridge it will appear as an alias for the bridge itself. =C2=A0= The > bridge is a PCI device with an IOMMU domain. =C2=A0When trying to ass= ign a > device to a guest there's some sanity checking to verify that the dev= ice > (or its alias) aren't already under some IOMMU domain other than the > guest VM's IOMMU domain. > > I suspect this is what you are hitting. =C2=A0You could test this the= ory by > adding 2 more devices to your guest -- the firewire device (08:0e.0) > and the PCI-PCI bridge itself (00:14.4). > > thanks, > -chris > Hi, OK, here we go again! Right, I've diabled apparmor (I think) with this: sudo invoke-rc.d apparmor stop sudo update-rc.d -f apparmor remove After a reboot I'm back to getting the error about pci-stub claiming th= e device. Apparmor being off made no difference to that (except there are no apparmor messages in dmesg) Then I try adding 08:0e.0 and 00:14.4 to the VM and I get this error me= ssage: libvirtError: this function is not supported by the connection driver: Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset available There is nothing written to test.log when you try to start the VM with 00:14.4 attached. At this point libvirt goes screwy and I have to restart it before I can remove 00:14.4 from the VM. However, once I've done THAT, starting the VM gets the different error message in test.log and a new dmesg: 2011-02-23 19:21:13.483: starting up LC_ALL=3DC PATH=3D/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sb= in:/bin QEMU_AUDIO_DRV=3Dnone /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -sm= p 3,sockets=3D3,cores=3D1,threads=3D1 -name test -uuid 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/test.monitor,serve= r,nowait -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dreadline -rtc base=3Dutc= -boot order=3Dcd,menu=3Doff -drive file=3D/var/lib/libvirt/images/test.img,if=3Dnone,id=3Ddrive-virtio-dis= k0,boot=3Don,format=3Draw -device virtio-blk-pci,bus=3Dpci.0,addr=3D0x4,drive=3Ddrive-virtio-disk= 0,id=3Dvirtio-disk0 -drive if=3Dnone,media=3Dcdrom,id=3Ddrive-ide0-1-0,readonly=3Don,format= =3Draw -device ide-drive,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0= -1-0 -netdev tap,fd=3D57,id=3Dhostnet0 -device virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:7d:32:7c,bus=3D= pci.0,addr=3D0x3 -chardev pty,id=3Dcharserial0 -device isa-serial,chardev=3Dcharserial0,id=3Dserial0 -usb -vnc 127.0.0.1:0 -vg= a cirrus -device pci-assign,host=3D08:06.0,id=3Dhostdev0,configfd=3D58,bu= s=3Dpci.0,addr=3D0x6 -device pci-assign,host=3D08:06.1,id=3Dhostdev1,configfd=3D59,bus=3Dpci= =2E0,addr=3D0x7 -device pci-assign,host=3D08:06.2,id=3Dhostdev2,configfd=3D60,bus=3Dpci= =2E0,addr=3D0x8 -device pci-assign,host=3D08:0e.0,id=3Dhostdev3,configfd=3D61,bus=3Dpci= =2E0,addr=3D0xa -device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x5 char device redirected to /dev/pts/1 kvm: -device virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:= 7d:32:7c,bus=3Dpci.0,addr=3D0x3: pci_add_option_rom: failed to find romfile "pxe-virtio.bin" Using raw in/out ioport access (sysfs - Input/output error) =46ailed to assign irq for "hostdev0": Operation not permitted Perhaps you are assigning a device that shares an IRQ with another devi= ce? kvm: -device pci-assign,host=3D08:06.0,id=3Dhostdev0,configfd=3D58,bus=3D= pci.0,addr=3D0x6: Device 'pci-assign' could not be initialized 2011-02-23 19:21:13.958: shutting down dmesg: http://pastebin.com/70D26xp4 This bit is different: [ 201.625221] uhci_hcd 0000:08:06.0: remove, state 4 [ 201.625237] usb usb4: USB disconnect, address 1 [ 201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered [ 201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled [ 201.626028] pci-stub 0000:08:06.0: claimed by stub [ 201.631922] uhci_hcd 0000:08:06.1: remove, state 4 [ 201.631937] usb usb9: USB disconnect, address 1 [ 201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered [ 201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled [ 201.632419] pci-stub 0000:08:06.1: claimed by stub [ 201.638160] ehci_hcd 0000:08:06.2: remove, state 1 [ 201.638172] usb usb10: USB disconnect, address 1 [ 201.638178] usb 10-1: USB disconnect, address 2 [ 201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully deinitialized and disconnected. [ 201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered [ 201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled [ 201.725042] pci-stub 0000:08:06.2: claimed by stub [ 201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled [ 201.731838] firewire_ohci: Removed fw-ohci device. [ 201.732536] pci-stub 0000:08:0e.0: claimed by stub [ 202.303880] device vnet0 entered promiscuous mode [ 202.305184] virbr0: topology change detected, propagating [ 202.305193] virbr0: port 1(vnet0) entering forwarding state [ 202.305199] virbr0: port 1(vnet0) entering forwarding state [ 202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) = -> IRQ 20 [ 202.470076] pci-stub 0000:08:06.0: restoring config space at offset 0x1 (was 0x2100000, writing 0x2100001) [ 202.697270] assign device 0:8:6.0 [ 202.697325] deassign device 0:8:6.0 [ 202.730080] pci-stub 0000:08:06.0: restoring config space at offset 0x1 (was 0x2100000, writing 0x2100001) [ 202.730107] pci-stub 0000:08:06.0: PCI INT A disabled This time the pci-stub claimed lines are not all bunched up and there is only one per device, rather than three per device. Also for the first time it says "assign device 0:8:6.0" rather than "assign device 0:8:6.0 failed" It them immediately deassigns the device and stops. test.log shows: =46ailed to assign irq for "hostdev0": Operation not permitted Perhaps you are assigning a device that shares an IRQ with another devi= ce? lspsci -vv for the relevant devices shows: http://pastebin.com/EUtUMj8x 00:14.4 now appears to be using pci-stub as it's driver, as well as 08:06.1, 2, 3 but not 0e.0 Anyway, that's all for now. I think I'll try 'amd_iommu_dump' next, does it write to dmesg? Many Thanks, James.