Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave <roboj1m@gmail.com>]

On Fri, Feb 25, 2011 at 11:31 PM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> On Fri, Feb 25, 2011 at 11:02 PM, James Neave <roboj1m@gmail.com> wrote:
>> > On Fri, Feb 25, 2011 at 10:47 PM, James Neave <roboj1m@gmail.com> wrote:
>> >> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> >>> * James Neave (roboj1m@gmail.com) wrote:
>> >>>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
>> >>>> http://pastebin.com/JxEwvqRA
>> >>>
>> >>> Yeah, that's what I expected:
>> >>>
>> >>> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
>> >>> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
>> >>>
>> >>> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
>> >>> should all go into same iommu domain).
>> >>>
>> >>>> I've just figured out a sequence of "echo DEV > PATH" commands to call
>> >>>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
>> >>>> the "failed to assign IRQ" error.
>> >>>> I'm going to narrow down the required sequence and then post it.
>> >>>
>> >>> Kind of afraid to ask, but does it include:
>> >>>
>> >>> (assuming 1002 4384 is the pci to pci bridge)
>> >>> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
>> >>> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
>> >>>
>> >>> (this has the side effect of detaching the bridge from its domain)
>> >>>
>> >>> thanks,
>> >>> -chris
>> >>>
>> >>
>> >> Exact sequence is:
>> >>
>> >> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
>> >>
>> >> I take it this is a bad thing then?
>> >>
>> >>> I assume this means that 00:14.4 is still left claimed by pci-stub?
>> >>
>> >> Yes
>> >>
>> >>> How are you determining this?  The lspci paste above has pci-stub for all
>> >>> of them.  The easiest thing might be to start with manually disabling
>> >>> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
>> >>> Then giving the guest only 08:06.1.
>> >>
>> >> I determined it by being half asleep and not reading it properly... >.<
>> >> You're right, all 5 devices were using pci-stub
>> >>
>> >>>> libvirtError: this function is not supported by the connection driver:
>> >>>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>> >>>> available
>> >>
>> >>> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
>> >>> libvirt here).
>> >>
>> >> What does that libvirt error mean? I can't find a definition.
>> >> Am I limiting myself by using libvirt? Would not using it help and how
>> >> would I go about not using it?
>> >>
>> >>> Trouble now is that
>> >>> with shared IRQ we don't have a good way to handle that right now.
>> >>
>> >> Game over then?
>> >> I've tried assigning the USB devices before, I couldn't do it because
>> >> qemu doesn't support USB2 devices.
>> >> I don't really understand where this IRQ conflict is, the firewire and
>> >> the USB2 device share IRQ22 but I'm assigning them both to the VM?
>> >> Is that still a problem?
>> >> I don't suppose there's any way to change which IRQ they use in the
>> >> BIOS or with a command is there?
>> >>
>> >> I don't know if it means anything but this page:
>> >>
>> >> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
>> >>
>> >> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
>> >> My knowledge it very limited on this subject so I don't know if that's
>> >> meaningless looking at the output from another person's lspci.
>> >>
>> >> Anything left to try?
>> >>
>> >> Regardless, many thanks for your help,
>> >>
>> >> James.
>> >>
>> >
>> > On the off chance I tried disabling the firewire in the BIOS, which
>> > leaves only my tuner card using IRQ 20, 21 and 22.
>> > No difference, still complains about IRQs:
>> >
>> > Using raw in/out ioport access (sysfs - Input/output error)
>> > Failed to assign irq for "hostdev0": Operation not permitted
>> > Perhaps you are assigning a device that shares an IRQ with another device?
>> >
>> > It does say "Operation not permitted" and that only "perhaps" I am
>> > assigning a device that shares an IRQ.
>> > Perhaps IRQ conflict it not the problem? They really are sitting on
>> > their own. Another permissions problem perhaps?
>> >
>> > Regards,
>> >
>> > James.
>> >
>>
>> I'm reading something about this error message being related to
>> libvirt and CAP_SYS_RAWIO?
>
> Depending on how new your libvirt is, you can force it to stop dropping
> capabilities.  Look for the config item "clear_emulator_capabilities"
> in /etc/libvirt/qemu.conf.  Setting this to 0 would verify that's the
> problem (and not a real shared irq...i thought i saw sharing on
> /proc/interrupts though).
>
>>
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg34338.html
>> http://www.google.co.uk/#hl=en&xhr=t&q=libvirt+CAP_SYS_RAWIO&cp=21&pf=p&sclient=psy&aq=f&aqi=&aql=&oq=libvirt+CAP_SYS_RAWIO&pbx=1&fp=2d8e3f69fec095f4
>>
>> "When I patch libvirt to not drop the capabilities, everything works
>> as expected."
>
> Well, that's a good point.  We fixed that a while ago, but I'm not sure
> your kernel has that fix.
>
> 2.6.35.10-dmar (btw, random nitpick, dmar == intel dma remapping engine,
> aka vt-d not amd iommu ;)
>
> This was fixed in 2.6.36, commit:
>
> 48bb09e KVM: remove CAP_SYS_RAWIO requirement from kvm_vm_ioctl_assign_irq
>
> The last 2.6.35 stable release is 2.6.35.9 and does not have that fix.
> So unless your .10-dmar has it, you could patch it in.
>
> thanks,
> -chris
>

HOLY CRAP IT WORKS!!!! 8@

...almost...

OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
as it turns out, the rawio problem)
My VM came up, both the tuners were there and after the firmware
install I was able to tune and watch the slowest TV in the world over
VNC.

Thank god for that, i was really starting to believe that slashing out
a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
collosal waste of money.
"Sigh of relief"

Well, thank you all so much for helping me to get to this point!

And yes, I did say "almost works"

Looks like I've run straight into Chris' ref counting problem when
shutting the guest down.
Some sort of critical error barf was on the servers' screen when I
shut down the guest, appeared to be very similar to Chris' example, in
amd_iommu.c

I'd post it but the server locks up after it's been shown and needed
resetting. No idea how I would post that bit of dmesg as it gets reset
after each boot.

Is there a solution for this at the moment or will I have to wait for
it to be patched?

Many Thanks,

James.