Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave <roboj1m@gmail.com>]

On Mon, Feb 21, 2011 at 10:49 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 21, 2011 at 10:25 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>>> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>>>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>>>> <alex.williamson@redhat.com> wrote:
>>>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>>>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>>>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>>>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>>> >> >>>> Hi,
>>>> >> >>>>
>>>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>>> >> >>>> 0000:08:06.2"
>>>> >> >>>
>>>> >> >>> This is a rather misleading error message. It is *expected* that
>>>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>>>> >> >>> error messages QEMU is printing aren't much help either, but
>>>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>>>> >> >>> step
>>>> >> >>
>>>> >> >> James, as far as I remember, I had the same issue when I set up my
>>>> >> >> system. Looking at my current (working) boot-script, apparently I've
>>>> >> >> added a 4th line which removes the pci-stub again as a
>>>> >> >> workaround....and it works:
>>>> >> >>
>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>>> >> >>
>>>> >> >> Best regards
>>>> >> >> Kenni
>>>> >> >>
>>>> >> >
>>>> >> > Hi Kenni,
>>>> >> >
>>>> >> > Can I get a bit more information on "boot-script" please? Which file
>>>> >> > exaclty have you put this in? Did you write your own service script
>>>> >> > and put it in init.d?
>>>> >> >
>>>> >> > I've tried this:
>>>> >> >
>>>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>>>> >> >
>>>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>>>> >> >
>>>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>>>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>>>> >> >
>>>> >> > Obviously one looks up which driver is being used by the PCI id, but
>>>> >> > how do I look up which driver my PCI card is using?
>>>> >> >
>>>> >> > Thanks,
>>>> >> >
>>>> >> > James.
>>>> >> >
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> OK, adding the fourth line does nothing, changing the second line to
>>>> >> "driver" rather than "device" does nothing, including in combination
>>>> >> with fourth line there/not there.
>>>> >
>>>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>>>> > change anything about devices that are already bound to it.  driver vs
>>>> > device are just different ways to get to the same thing.
>>>> >
>>>> >> So I'm still stuck, can anybody else help?
>>>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>>>> >> against my kernel?
>>>> >
>>>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>>>> > start from a clean slate and see if it helps.  Here's what I would
>>>> > suggest:
>>>> >
>>>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>>>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>>>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>>>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>>>> >
>>>> > Note we have to knock out the firewire because it shares an interrupt
>>>> > with the ehci device you're trying to assign.  We want to remove the USB
>>>> > controller entirely from the host.  Your dmesg indicates the host is
>>>> > still seeing the device via the uhci ports, and isn't happy about it.
>>>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>>>> > from claiming the device, it's not required for device assignment.  Now,
>>>> > instead of only trying to assign the ehci, let's move the whole usb
>>>> > controller to the guest:
>>>> >
>>>> > -device pci-assign,host=08:06.0,addr=5.0 \
>>>> > -device pci-assign,host=08:06.1,addr=5.1 \
>>>> > -device pci-assign,host=08:06.2,addr=5.2
>>>> >
>>>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>>>> > If that works, then you can bind all those devices to pci-stub and it
>>>> > should still work.
>>>> >
>>>> > Alex
>>>>
>>>> Hi,
>>>>
>>>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>>>> compiling the latest qemu, took me a while to put it all back together
>>>> again in between work!
>>>>
>>>> I'm afraid I use virtmanager, although I guess using the "add pci
>>>> device" function is the same as -device pci-assign? It does seem to
>>>> add it to the command line that gets written out to the log files in
>>>> /var/log/libvirt/qemu.
>>>>
>>>> Nevertheless, I've tried that and still not luck, the log output is
>>>> similar, with one extra line:
>>>>
>>>> http://pastebin.com/MJ6aqjNq
>>>
>>> You actually ended up with:
>>>
>>> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
>>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
>>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>>>
>>> Which isn't quite what I was suggesting.  You probably have xml that
>>> looks like this:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>    </hostdev>
>>>    ...
>>>
>>> Try adding an address line, so you get something more like this:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>>>    </hostdev>
>>>
>>>
>>>> Using raw in/out ioport access (sysfs - Input/output error)
>>>
>>> This is just an informational line to let us know whether pci-sysfs
>>> supports read/write on the ioport resource files.  If it does, we use
>>> those rather than doing raw in/out for access to the device.  This does
>>> highlight another potential problem.  Your distro probably doesn't have
>>> all the patches in place for non-privileged device assignment, which
>>> could be why you're having strange issues.  Check
>>> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
>>> they're not already, try setting them to 'root', restart libvirtd and
>>> see if anything improves.
>>>
>>>> Here's the dmesg output:
>>>> http://pastebin.com/AE1euUN1
>>>
>>> If still issues after the above, it might be useful to pastbin the
>>> entire dmesg so we can make sure the iommu is really on.  Thanks,
>>>
>>> Alex
>>
>> Hi,
>>
>> No such luck I'm afraid.
>>
>> Here is the original XML:
>>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>> function='0x0'/>
>>    </hostdev>
>>
>> The only difference from your recommended change is the target (I take
>> it they are target addresses for the VM?) addresses run:
>> 0000:00:06.0
>> 0000:00:07.0
>> 0000:00:08.0
>>
>> Instead of:
>> 0000:00:06.0
>> 0000:00:06.1
>> 0000:00:06.2
>>
>> Regardless, I still changed test.xml to:
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x1'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x2'/>
>>    </hostdev>
>>
>> To no effect.
>>
>> In qemu.conf, user and group were commented out, I uncommented both
>> and they were both already set to root.
>>
>> After both a restart of libvirt-bin and the pc itself, no change.
>>
>> Finally, here is the very latest dmesg:
>> http://pastebin.com/9HE61K62
>>
>> Does anybody know the debug kernel switches for iommu?
>>
>> Many Thanks,
>>
>> James.
>>
>
> Hi,
>
> A ray of sunshine (ish)
>
> I can apparently pass through this device:
>
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
>        Subsystem: Giga-byte Technology Device a102
>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 32, Cache Line Size: 64 bytes
>        Interrupt: pin ? routed to IRQ 16
>        Region 0: Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
>        Capabilities: <access denied>
>        Kernel driver in use: HDA Intel
>        Kernel modules: snd-hda-intel
>
> From test.log:
>
> 2011-02-21 22:40:58.888: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
> 3,sockets=3,cores=1,threads=1 -name test -uuid
> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
> c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -drive file=/home/james/ubuntu_10.10_x86.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=59,id=hostnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:1 -vga
> cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> char device redirected to /dev/pts/1
> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
> 2011-02-21 22:41:48.090: shutting down
>
> Specifically: -device
> pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>
> Unfortunately I can't get it to boot an ISO through virtmanager! ><
> Well, not without deleting the VM and recreating it again.
> I shall try getting the latest virt-manager.
>
> Regards,
>
> James.
>

Or I could just try pressing the Apply button... ¬.¬;;

J.