Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave <roboj1m@gmail.com>]

On Wed, Feb 23, 2011 at 8:09 PM, James Neave <roboj1m@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 7:44 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Wed, Feb 23, 2011 at 12:11 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>> * James Neave (roboj1m@gmail.com) wrote:
>>>> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>>> > * James Neave (roboj1m@gmail.com) wrote:
>>>> >> Does anybody know the debug kernel switches for iommu?
>>>> >
>>>> > Two helpful kernel commandline options are:
>>>> >
>>>> > amd_iommu_dump debug (and drop "quiet")
>>>> >
>>>> > The problem is when you attach the device (function) you're getting
>>>> > stuck up in conflicts with the existing domain for that function.
>>>> >
>>>> > My guess is that all the functions are behind a PCI to PCI bridge, so the alias
>>>> > lookup is finding a conflict.
>>>>
>>>> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
>>>> earlier email:
>>>
>>> Sorry, I missed that in your original mail, thanks for reposting.
>>>
>>>> cat /proc/interruts
>>>> http://pastebin.com/LQdB3hms
>>>>
>>>> lspci -vvv
>>>> http://pastebin.com/GJDkC8B4
>>>
>>>  00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>>>
>>>> lspci -t -v
>>>> http://pastebin.com/Ftx8Hfjt
>>>
>>> Yup, that's what I expected:
>>>
>>>  +-14.4-[08]--+-06.0  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>>  |            +-06.1  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>>  |            +-06.2  VIA Technologies, Inc. USB 2.0
>>>  |            \-0e.0  Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller
>>>
>>> I'd now expect to see (if you boot with amd_iommu_dump) some IVRS
>>> details showing an alias range entry basically showing 08:* pointing
>>> back to 00:14.4.  This means that from the point of view of the IOMMU the
>>> devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if they
>>> are 00:14.4.
>>>
>>> When you assign a device to a guest, the guest VM gets an IOMMU domain
>>> (a context to manage IOMMU page table mappings) and the device is put
>>> into that guest's IOMMU domain.  However, if the device is behind a
>>> PCI-PCI bridge it will appear as an alias for the bridge itself.  The
>>> bridge is a PCI device with an IOMMU domain.  When trying to assign a
>>> device to a guest there's some sanity checking to verify that the device
>>> (or its alias) aren't already under some IOMMU domain other than the
>>> guest VM's IOMMU domain.
>>>
>>> I suspect this is what you are hitting.  You could test this theory by
>>> adding 2 more devices to your guest -- the firewire device (08:0e.0)
>>> and the PCI-PCI bridge itself (00:14.4).
>>>
>>> thanks,
>>> -chris
>>>
>>
>> Hi,
>>
>> OK, here we go again!
>>
>> Right, I've diabled apparmor (I think) with this:
>>
>> sudo invoke-rc.d apparmor stop
>> sudo update-rc.d -f apparmor remove
>>
>> After a reboot I'm back to getting the error about pci-stub claiming the device.
>> Apparmor being off made no difference to that (except there are no
>> apparmor messages in dmesg)
>>
>> Then I try adding 08:0e.0 and 00:14.4 to the VM and I get this error message:
>>
>> libvirtError: this function is not supported by the connection driver:
>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>> available
>>
>> There is nothing written to test.log when you try to start the VM with
>> 00:14.4 attached.
>>
>> At this point libvirt goes screwy and I have to restart it before I
>> can remove 00:14.4 from the VM.
>>
>> However, once I've done THAT, starting the VM gets the different error
>> message in test.log and a new dmesg:
>>
>> 2011-02-23 19:21:13.483: starting up
>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
>> 3,sockets=3,cores=1,threads=1 -name test -uuid
>> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
>> order=cd,menu=off -drive
>> file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
>> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> -netdev tap,fd=57,id=hostnet0 -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga
>> cirrus -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6
>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7
>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>> -device pci-assign,host=08:0e.0,id=hostdev3,configfd=61,bus=pci.0,addr=0xa
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> char device redirected to /dev/pts/1
>> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
>> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
>> Using raw in/out ioport access (sysfs - Input/output error)
>> Failed to assign irq for "hostdev0": Operation not permitted
>> Perhaps you are assigning a device that shares an IRQ with another device?
>> kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6:
>> Device 'pci-assign' could not be initialized
>> 2011-02-23 19:21:13.958: shutting down
>>
>> dmesg:
>> http://pastebin.com/70D26xp4
>>
>> This bit is different:
>>
>> [  201.625221] uhci_hcd 0000:08:06.0: remove, state 4
>> [  201.625237] usb usb4: USB disconnect, address 1
>> [  201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
>> [  201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled
>> [  201.626028] pci-stub 0000:08:06.0: claimed by stub
>> [  201.631922] uhci_hcd 0000:08:06.1: remove, state 4
>> [  201.631937] usb usb9: USB disconnect, address 1
>> [  201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered
>> [  201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled
>> [  201.632419] pci-stub 0000:08:06.1: claimed by stub
>> [  201.638160] ehci_hcd 0000:08:06.2: remove, state 1
>> [  201.638172] usb usb10: USB disconnect, address 1
>> [  201.638178] usb 10-1: USB disconnect, address 2
>> [  201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully
>> deinitialized and disconnected.
>> [  201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered
>> [  201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled
>> [  201.725042] pci-stub 0000:08:06.2: claimed by stub
>> [  201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled
>> [  201.731838] firewire_ohci: Removed fw-ohci device.
>> [  201.732536] pci-stub 0000:08:0e.0: claimed by stub
>> [  202.303880] device vnet0 entered promiscuous mode
>> [  202.305184] virbr0: topology change detected, propagating
>> [  202.305193] virbr0: port 1(vnet0) entering forwarding state
>> [  202.305199] virbr0: port 1(vnet0) entering forwarding state
>> [  202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
>> [  202.470076] pci-stub 0000:08:06.0: restoring config space at offset
>> 0x1 (was 0x2100000, writing 0x2100001)
>> [  202.697270] assign device 0:8:6.0
>> [  202.697325] deassign device 0:8:6.0
>> [  202.730080] pci-stub 0000:08:06.0: restoring config space at offset
>> 0x1 (was 0x2100000, writing 0x2100001)
>> [  202.730107] pci-stub 0000:08:06.0: PCI INT A disabled
>>
>> This time the pci-stub claimed lines are not all bunched up and there
>> is only one per device, rather than three per device.
>> Also for the first time it says "assign device 0:8:6.0" rather than
>> "assign device 0:8:6.0 failed"
>> It them immediately deassigns the device and stops.
>>
>> test.log shows:
>>
>> Failed to assign irq for "hostdev0": Operation not permitted
>> Perhaps you are assigning a device that shares an IRQ with another device?
>>
>> lspsci -vv for the relevant devices shows:
>> http://pastebin.com/EUtUMj8x
>>
>> 00:14.4 now appears to be using pci-stub as it's driver, as well as
>> 08:06.1, 2, 3 but not 0e.0
>>
>> Anyway, that's all for now.
>> I think I'll try 'amd_iommu_dump' next, does it write to dmesg?
>>
>> Many Thanks,
>>
>> James.
>>
>
> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
> http://pastebin.com/JxEwvqRA
>
> I've just figured out a sequence of "echo DEV > PATH" commands to call
> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
> the "failed to assign IRQ" error.
> I'm going to narrow down the required sequence and then post it.
>
> Regards,
>
> James.
>

Hi,

Just out of interest, what kind of mileage would I expect out of
buying a shiny new PCIe tuner?
Can I pass through PCIe? Would it work better because it wouldn't be
behind a bridge? WOULD it not be behind a bridge?
As much as I'd hate to solve a problem with the application of money... :(
(OT question, on mailing lists should I use Reply All or just reply
and change the To address to kvm.vger.kernel.org?)

Regards,

James.