PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

Hi,

I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
I'm getting the error "The driver 'pci-stub' is occupying your device
0000:08:06.2"

I'm running Ubuntu Server 10.10 64bit.
I've recompiled the kernel with the default options + DMAR, uname -r =
2.6.35.10-vi
I've installed a recent version of QEMU-KVM using the Ubuntu Server
Edgers ppa here:
https://launchpad.net/~ubuntu-server-edgers/+archive/server-edgers-qemu-kvm
qemu -version = QEMU emulator version 0.13.50 (qemu-kvm-devel),
Copyright (c) 2003-2008 Fabrice Bellard

The hardware is an 890FX based Gigabyte GA-890FXA-UD5 with the IOMMU
and VT enabled in the BIOS

I've followed this page as a guide on how to get pci passthrough working:
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Can somebody help me get this working please?

Many thanks,

James.

I've tried to include all the relevant information in some pastebin
dumps (to stop this email being a headache)

/var/log/libvirt/qemu/test.log:
http://pastebin.com/hCv7XPR3

dmesg:
http://pastebin.com/swtW5tfq

dmesg | grep AMD-VI
[    0.698863] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
[    0.710165] AMD-Vi: Lazy IO/TLB flushing enabled

cat /proc/interruts
http://pastebin.com/LQdB3hms

lspci -vvv
http://pastebin.com/GJDkC8B4

lspci -t -v
http://pastebin.com/Ftx8Hfjt

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Daniel P. Berrange]

On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
> Hi,
> 
> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
> I'm getting the error "The driver 'pci-stub' is occupying your device
> 0000:08:06.2"

This is a rather misleading error message. It is *expected* that
pci-stub will occupy the device. Unfortunately the rest of the
error messages QEMU is printing aren't much help either, but 
ultimately something is returning -EBUSY in the PCI device assign
step

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 7, 2011 at 1:26 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>> Hi,
>>
>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>> I'm getting the error "The driver 'pci-stub' is occupying your device
>> 0000:08:06.2"
>
> This is a rather misleading error message. It is *expected* that
> pci-stub will occupy the device. Unfortunately the rest of the
> error messages QEMU is printing aren't much help either, but
> ultimately something is returning -EBUSY in the PCI device assign
> step
>
> Regards,
> Daniel

That's what I figured, but sadly I'm just a "user". Paucity of google
results suggests that this problem is new and unknown/unfixed.
I had a quick stab at this with Xen last night, but it seems Ubuntu
and Xen are no longer friends.
The only thing left I can think to try is to check out the latest code
and try that.
Is it "normal" with qemu-kvm? Do I just get the code with git, faf
around with dependencies and then ./configure, make and make install?

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Kenni Lund]

2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>> Hi,
>>
>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>> I'm getting the error "The driver 'pci-stub' is occupying your device
>> 0000:08:06.2"
>
> This is a rather misleading error message. It is *expected* that
> pci-stub will occupy the device. Unfortunately the rest of the
> error messages QEMU is printing aren't much help either, but
> ultimately something is returning -EBUSY in the PCI device assign
> step

James, as far as I remember, I had the same issue when I set up my
system. Looking at my current (working) boot-script, apparently I've
added a 4th line which removes the pci-stub again as a
workaround....and it works:

echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id

Best regards
Kenni

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>> Hi,
>>>
>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>> 0000:08:06.2"
>>
>> This is a rather misleading error message. It is *expected* that
>> pci-stub will occupy the device. Unfortunately the rest of the
>> error messages QEMU is printing aren't much help either, but
>> ultimately something is returning -EBUSY in the PCI device assign
>> step
>
> James, as far as I remember, I had the same issue when I set up my
> system. Looking at my current (working) boot-script, apparently I've
> added a 4th line which removes the pci-stub again as a
> workaround....and it works:
>
> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>
> Best regards
> Kenni
>

Hi Kenni,

Can I get a bit more information on "boot-script" please? Which file
exaclty have you put this in? Did you write your own service script
and put it in init.d?

I've tried this:

echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

I'll try it again with the fourth line added, manually before I start the VM.

How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind

Obviously one looks up which driver is being used by the PCI id, but
how do I look up which driver my PCI card is using?

Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
> On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>>> Hi,
>>>>
>>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>>> 0000:08:06.2"
>>>
>>> This is a rather misleading error message. It is *expected* that
>>> pci-stub will occupy the device. Unfortunately the rest of the
>>> error messages QEMU is printing aren't much help either, but
>>> ultimately something is returning -EBUSY in the PCI device assign
>>> step
>>
>> James, as far as I remember, I had the same issue when I set up my
>> system. Looking at my current (working) boot-script, apparently I've
>> added a 4th line which removes the pci-stub again as a
>> workaround....and it works:
>>
>> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>
>> Best regards
>> Kenni
>>
>
> Hi Kenni,
>
> Can I get a bit more information on "boot-script" please? Which file
> exaclty have you put this in? Did you write your own service script
> and put it in init.d?
>
> I've tried this:
>
> echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
> echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
> echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>
> I'll try it again with the fourth line added, manually before I start the VM.
>
> How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
> and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>
> Obviously one looks up which driver is being used by the PCI id, but
> how do I look up which driver my PCI card is using?
>
> Thanks,
>
> James.
>

Hi,

OK, adding the fourth line does nothing, changing the second line to
"driver" rather than "device" does nothing, including in combination
with fourth line there/not there.

So I'm still stuck, can anybody else help?
Perhaps point me to a guide on how to compile the latest qemu-kvm
against my kernel?

Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Alex Williamson]

On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
> >>>> Hi,
> >>>>
> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
> >>>> 0000:08:06.2"
> >>>
> >>> This is a rather misleading error message. It is *expected* that
> >>> pci-stub will occupy the device. Unfortunately the rest of the
> >>> error messages QEMU is printing aren't much help either, but
> >>> ultimately something is returning -EBUSY in the PCI device assign
> >>> step
> >>
> >> James, as far as I remember, I had the same issue when I set up my
> >> system. Looking at my current (working) boot-script, apparently I've
> >> added a 4th line which removes the pci-stub again as a
> >> workaround....and it works:
> >>
> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
> >>
> >> Best regards
> >> Kenni
> >>
> >
> > Hi Kenni,
> >
> > Can I get a bit more information on "boot-script" please? Which file
> > exaclty have you put this in? Did you write your own service script
> > and put it in init.d?
> >
> > I've tried this:
> >
> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
> >
> > I'll try it again with the fourth line added, manually before I start the VM.
> >
> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
> >
> > Obviously one looks up which driver is being used by the PCI id, but
> > how do I look up which driver my PCI card is using?
> >
> > Thanks,
> >
> > James.
> >
> 
> Hi,
> 
> OK, adding the fourth line does nothing, changing the second line to
> "driver" rather than "device" does nothing, including in combination
> with fourth line there/not there.

Yep, the last line is just removing the id from pci-stub, it doesn't
change anything about devices that are already bound to it.  driver vs
device are just different ways to get to the same thing.

> So I'm still stuck, can anybody else help?
> Perhaps point me to a guide on how to compile the latest qemu-kvm
> against my kernel?

I don't know why you're getting -EBUSY for this device, but maybe we can
start from a clean slate and see if it helps.  Here's what I would
suggest:

echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind

Note we have to knock out the firewire because it shares an interrupt
with the ehci device you're trying to assign.  We want to remove the USB
controller entirely from the host.  Your dmesg indicates the host is
still seeing the device via the uhci ports, and isn't happy about it.
You can ignore pci-stub for the moment, it's just a way to keep drivers
from claiming the device, it's not required for device assignment.  Now,
instead of only trying to assign the ehci, let's move the whole usb
controller to the guest:

-device pci-assign,host=08:06.0,addr=5.0 \
-device pci-assign,host=08:06.1,addr=5.1 \
-device pci-assign,host=08:06.2,addr=5.2

(slot 5 on the guest is arbitrary, pick something else if you need to)
If that works, then you can bind all those devices to pci-stub and it
should still work.

Alex

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>> >>>> Hi,
>> >>>>
>> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>> >>>> 0000:08:06.2"
>> >>>
>> >>> This is a rather misleading error message. It is *expected* that
>> >>> pci-stub will occupy the device. Unfortunately the rest of the
>> >>> error messages QEMU is printing aren't much help either, but
>> >>> ultimately something is returning -EBUSY in the PCI device assign
>> >>> step
>> >>
>> >> James, as far as I remember, I had the same issue when I set up my
>> >> system. Looking at my current (working) boot-script, apparently I've
>> >> added a 4th line which removes the pci-stub again as a
>> >> workaround....and it works:
>> >>
>> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>> >>
>> >> Best regards
>> >> Kenni
>> >>
>> >
>> > Hi Kenni,
>> >
>> > Can I get a bit more information on "boot-script" please? Which file
>> > exaclty have you put this in? Did you write your own service script
>> > and put it in init.d?
>> >
>> > I've tried this:
>> >
>> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>> >
>> > I'll try it again with the fourth line added, manually before I start the VM.
>> >
>> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>> >
>> > Obviously one looks up which driver is being used by the PCI id, but
>> > how do I look up which driver my PCI card is using?
>> >
>> > Thanks,
>> >
>> > James.
>> >
>>
>> Hi,
>>
>> OK, adding the fourth line does nothing, changing the second line to
>> "driver" rather than "device" does nothing, including in combination
>> with fourth line there/not there.
>
> Yep, the last line is just removing the id from pci-stub, it doesn't
> change anything about devices that are already bound to it.  driver vs
> device are just different ways to get to the same thing.
>
>> So I'm still stuck, can anybody else help?
>> Perhaps point me to a guide on how to compile the latest qemu-kvm
>> against my kernel?
>
> I don't know why you're getting -EBUSY for this device, but maybe we can
> start from a clean slate and see if it helps.  Here's what I would
> suggest:
>
> echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
> echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
> echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
> echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>
> Note we have to knock out the firewire because it shares an interrupt
> with the ehci device you're trying to assign.  We want to remove the USB
> controller entirely from the host.  Your dmesg indicates the host is
> still seeing the device via the uhci ports, and isn't happy about it.
> You can ignore pci-stub for the moment, it's just a way to keep drivers
> from claiming the device, it's not required for device assignment.  Now,
> instead of only trying to assign the ehci, let's move the whole usb
> controller to the guest:
>
> -device pci-assign,host=08:06.0,addr=5.0 \
> -device pci-assign,host=08:06.1,addr=5.1 \
> -device pci-assign,host=08:06.2,addr=5.2
>
> (slot 5 on the guest is arbitrary, pick something else if you need to)
> If that works, then you can bind all those devices to pci-stub and it
> should still work.
>
> Alex

Hi,

Sorry about the slow reply, I hosed all of my PCs fiddling with
compiling the latest qemu, took me a while to put it all back together
again in between work!

I'm afraid I use virtmanager, although I guess using the "add pci
device" function is the same as -device pci-assign? It does seem to
add it to the command line that gets written out to the log files in
/var/log/libvirt/qemu.

Nevertheless, I've tried that and still not luck, the log output is
similar, with one extra line:

http://pastebin.com/MJ6aqjNq

Using raw in/out ioport access (sysfs - Input/output error)

Here's the dmesg output:
http://pastebin.com/AE1euUN1

I'm going to google that new line and consider trying the libvirt ppa
that accompanies the qemu-kvm ppa I'm using.

Many Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Alex Williamson]

On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
> >> >>>> Hi,
> >> >>>>
> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
> >> >>>> 0000:08:06.2"
> >> >>>
> >> >>> This is a rather misleading error message. It is *expected* that
> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
> >> >>> error messages QEMU is printing aren't much help either, but
> >> >>> ultimately something is returning -EBUSY in the PCI device assign
> >> >>> step
> >> >>
> >> >> James, as far as I remember, I had the same issue when I set up my
> >> >> system. Looking at my current (working) boot-script, apparently I've
> >> >> added a 4th line which removes the pci-stub again as a
> >> >> workaround....and it works:
> >> >>
> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
> >> >>
> >> >> Best regards
> >> >> Kenni
> >> >>
> >> >
> >> > Hi Kenni,
> >> >
> >> > Can I get a bit more information on "boot-script" please? Which file
> >> > exaclty have you put this in? Did you write your own service script
> >> > and put it in init.d?
> >> >
> >> > I've tried this:
> >> >
> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
> >> >
> >> > I'll try it again with the fourth line added, manually before I start the VM.
> >> >
> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
> >> >
> >> > Obviously one looks up which driver is being used by the PCI id, but
> >> > how do I look up which driver my PCI card is using?
> >> >
> >> > Thanks,
> >> >
> >> > James.
> >> >
> >>
> >> Hi,
> >>
> >> OK, adding the fourth line does nothing, changing the second line to
> >> "driver" rather than "device" does nothing, including in combination
> >> with fourth line there/not there.
> >
> > Yep, the last line is just removing the id from pci-stub, it doesn't
> > change anything about devices that are already bound to it.  driver vs
> > device are just different ways to get to the same thing.
> >
> >> So I'm still stuck, can anybody else help?
> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
> >> against my kernel?
> >
> > I don't know why you're getting -EBUSY for this device, but maybe we can
> > start from a clean slate and see if it helps.  Here's what I would
> > suggest:
> >
> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
> >
> > Note we have to knock out the firewire because it shares an interrupt
> > with the ehci device you're trying to assign.  We want to remove the USB
> > controller entirely from the host.  Your dmesg indicates the host is
> > still seeing the device via the uhci ports, and isn't happy about it.
> > You can ignore pci-stub for the moment, it's just a way to keep drivers
> > from claiming the device, it's not required for device assignment.  Now,
> > instead of only trying to assign the ehci, let's move the whole usb
> > controller to the guest:
> >
> > -device pci-assign,host=08:06.0,addr=5.0 \
> > -device pci-assign,host=08:06.1,addr=5.1 \
> > -device pci-assign,host=08:06.2,addr=5.2
> >
> > (slot 5 on the guest is arbitrary, pick something else if you need to)
> > If that works, then you can bind all those devices to pci-stub and it
> > should still work.
> >
> > Alex
> 
> Hi,
> 
> Sorry about the slow reply, I hosed all of my PCs fiddling with
> compiling the latest qemu, took me a while to put it all back together
> again in between work!
> 
> I'm afraid I use virtmanager, although I guess using the "add pci
> device" function is the same as -device pci-assign? It does seem to
> add it to the command line that gets written out to the log files in
> /var/log/libvirt/qemu.
> 
> Nevertheless, I've tried that and still not luck, the log output is
> similar, with one extra line:
> 
> http://pastebin.com/MJ6aqjNq

You actually ended up with:

-device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
-device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
-device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8

Which isn't quite what I was suggesting.  You probably have xml that
looks like this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
    </hostdev>
    ...

Try adding an address line, so you get something more like this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </hostdev>


> Using raw in/out ioport access (sysfs - Input/output error)

This is just an informational line to let us know whether pci-sysfs
supports read/write on the ioport resource files.  If it does, we use
those rather than doing raw in/out for access to the device.  This does
highlight another potential problem.  Your distro probably doesn't have
all the patches in place for non-privileged device assignment, which
could be why you're having strange issues.  Check
your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
they're not already, try setting them to 'root', restart libvirtd and
see if anything improves.

> Here's the dmesg output:
> http://pastebin.com/AE1euUN1

If still issues after the above, it might be useful to pastbin the
entire dmesg so we can make sure the iommu is really on.  Thanks,

Alex

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>> >> >>>> 0000:08:06.2"
>> >> >>>
>> >> >>> This is a rather misleading error message. It is *expected* that
>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>> >> >>> error messages QEMU is printing aren't much help either, but
>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>> >> >>> step
>> >> >>
>> >> >> James, as far as I remember, I had the same issue when I set up my
>> >> >> system. Looking at my current (working) boot-script, apparently I've
>> >> >> added a 4th line which removes the pci-stub again as a
>> >> >> workaround....and it works:
>> >> >>
>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>> >> >>
>> >> >> Best regards
>> >> >> Kenni
>> >> >>
>> >> >
>> >> > Hi Kenni,
>> >> >
>> >> > Can I get a bit more information on "boot-script" please? Which file
>> >> > exaclty have you put this in? Did you write your own service script
>> >> > and put it in init.d?
>> >> >
>> >> > I've tried this:
>> >> >
>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>> >> >
>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>> >> >
>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>> >> >
>> >> > Obviously one looks up which driver is being used by the PCI id, but
>> >> > how do I look up which driver my PCI card is using?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > James.
>> >> >
>> >>
>> >> Hi,
>> >>
>> >> OK, adding the fourth line does nothing, changing the second line to
>> >> "driver" rather than "device" does nothing, including in combination
>> >> with fourth line there/not there.
>> >
>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>> > change anything about devices that are already bound to it.  driver vs
>> > device are just different ways to get to the same thing.
>> >
>> >> So I'm still stuck, can anybody else help?
>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>> >> against my kernel?
>> >
>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>> > start from a clean slate and see if it helps.  Here's what I would
>> > suggest:
>> >
>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>> >
>> > Note we have to knock out the firewire because it shares an interrupt
>> > with the ehci device you're trying to assign.  We want to remove the USB
>> > controller entirely from the host.  Your dmesg indicates the host is
>> > still seeing the device via the uhci ports, and isn't happy about it.
>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>> > from claiming the device, it's not required for device assignment.  Now,
>> > instead of only trying to assign the ehci, let's move the whole usb
>> > controller to the guest:
>> >
>> > -device pci-assign,host=08:06.0,addr=5.0 \
>> > -device pci-assign,host=08:06.1,addr=5.1 \
>> > -device pci-assign,host=08:06.2,addr=5.2
>> >
>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>> > If that works, then you can bind all those devices to pci-stub and it
>> > should still work.
>> >
>> > Alex
>>
>> Hi,
>>
>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>> compiling the latest qemu, took me a while to put it all back together
>> again in between work!
>>
>> I'm afraid I use virtmanager, although I guess using the "add pci
>> device" function is the same as -device pci-assign? It does seem to
>> add it to the command line that gets written out to the log files in
>> /var/log/libvirt/qemu.
>>
>> Nevertheless, I've tried that and still not luck, the log output is
>> similar, with one extra line:
>>
>> http://pastebin.com/MJ6aqjNq
>
> You actually ended up with:
>
> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>
> Which isn't quite what I was suggesting.  You probably have xml that
> looks like this:
>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>    </hostdev>
>    ...
>
> Try adding an address line, so you get something more like this:
>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>    </hostdev>
>
>
>> Using raw in/out ioport access (sysfs - Input/output error)
>
> This is just an informational line to let us know whether pci-sysfs
> supports read/write on the ioport resource files.  If it does, we use
> those rather than doing raw in/out for access to the device.  This does
> highlight another potential problem.  Your distro probably doesn't have
> all the patches in place for non-privileged device assignment, which
> could be why you're having strange issues.  Check
> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
> they're not already, try setting them to 'root', restart libvirtd and
> see if anything improves.
>
>> Here's the dmesg output:
>> http://pastebin.com/AE1euUN1
>
> If still issues after the above, it might be useful to pastbin the
> entire dmesg so we can make sure the iommu is really on.  Thanks,
>
> Alex

Hi,

No such luck I'm afraid.

Here is the original XML:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </hostdev>

The only difference from your recommended change is the target (I take
it they are target addresses for the VM?) addresses run:
0000:00:06.0
0000:00:07.0
0000:00:08.0

Instead of:
0000:00:06.0
0000:00:06.1
0000:00:06.2

Regardless, I still changed test.xml to:
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
    </hostdev>

To no effect.

In qemu.conf, user and group were commented out, I uncommented both
and they were both already set to root.

After both a restart of libvirt-bin and the pc itself, no change.

Finally, here is the very latest dmesg:
http://pastebin.com/9HE61K62

Does anybody know the debug kernel switches for iommu?

Many Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 21, 2011 at 10:25 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
>> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>>> <alex.williamson@redhat.com> wrote:
>>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>> >> >>>> Hi,
>>> >> >>>>
>>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>> >> >>>> 0000:08:06.2"
>>> >> >>>
>>> >> >>> This is a rather misleading error message. It is *expected* that
>>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>>> >> >>> error messages QEMU is printing aren't much help either, but
>>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>>> >> >>> step
>>> >> >>
>>> >> >> James, as far as I remember, I had the same issue when I set up my
>>> >> >> system. Looking at my current (working) boot-script, apparently I've
>>> >> >> added a 4th line which removes the pci-stub again as a
>>> >> >> workaround....and it works:
>>> >> >>
>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>> >> >>
>>> >> >> Best regards
>>> >> >> Kenni
>>> >> >>
>>> >> >
>>> >> > Hi Kenni,
>>> >> >
>>> >> > Can I get a bit more information on "boot-script" please? Which file
>>> >> > exaclty have you put this in? Did you write your own service script
>>> >> > and put it in init.d?
>>> >> >
>>> >> > I've tried this:
>>> >> >
>>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>>> >> >
>>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>>> >> >
>>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>>> >> >
>>> >> > Obviously one looks up which driver is being used by the PCI id, but
>>> >> > how do I look up which driver my PCI card is using?
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > James.
>>> >> >
>>> >>
>>> >> Hi,
>>> >>
>>> >> OK, adding the fourth line does nothing, changing the second line to
>>> >> "driver" rather than "device" does nothing, including in combination
>>> >> with fourth line there/not there.
>>> >
>>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>>> > change anything about devices that are already bound to it.  driver vs
>>> > device are just different ways to get to the same thing.
>>> >
>>> >> So I'm still stuck, can anybody else help?
>>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>>> >> against my kernel?
>>> >
>>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>>> > start from a clean slate and see if it helps.  Here's what I would
>>> > suggest:
>>> >
>>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>>> >
>>> > Note we have to knock out the firewire because it shares an interrupt
>>> > with the ehci device you're trying to assign.  We want to remove the USB
>>> > controller entirely from the host.  Your dmesg indicates the host is
>>> > still seeing the device via the uhci ports, and isn't happy about it.
>>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>>> > from claiming the device, it's not required for device assignment.  Now,
>>> > instead of only trying to assign the ehci, let's move the whole usb
>>> > controller to the guest:
>>> >
>>> > -device pci-assign,host=08:06.0,addr=5.0 \
>>> > -device pci-assign,host=08:06.1,addr=5.1 \
>>> > -device pci-assign,host=08:06.2,addr=5.2
>>> >
>>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>>> > If that works, then you can bind all those devices to pci-stub and it
>>> > should still work.
>>> >
>>> > Alex
>>>
>>> Hi,
>>>
>>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>>> compiling the latest qemu, took me a while to put it all back together
>>> again in between work!
>>>
>>> I'm afraid I use virtmanager, although I guess using the "add pci
>>> device" function is the same as -device pci-assign? It does seem to
>>> add it to the command line that gets written out to the log files in
>>> /var/log/libvirt/qemu.
>>>
>>> Nevertheless, I've tried that and still not luck, the log output is
>>> similar, with one extra line:
>>>
>>> http://pastebin.com/MJ6aqjNq
>>
>> You actually ended up with:
>>
>> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>>
>> Which isn't quite what I was suggesting.  You probably have xml that
>> looks like this:
>>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>    </hostdev>
>>    ...
>>
>> Try adding an address line, so you get something more like this:
>>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>>    </hostdev>
>>
>>
>>> Using raw in/out ioport access (sysfs - Input/output error)
>>
>> This is just an informational line to let us know whether pci-sysfs
>> supports read/write on the ioport resource files.  If it does, we use
>> those rather than doing raw in/out for access to the device.  This does
>> highlight another potential problem.  Your distro probably doesn't have
>> all the patches in place for non-privileged device assignment, which
>> could be why you're having strange issues.  Check
>> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
>> they're not already, try setting them to 'root', restart libvirtd and
>> see if anything improves.
>>
>>> Here's the dmesg output:
>>> http://pastebin.com/AE1euUN1
>>
>> If still issues after the above, it might be useful to pastbin the
>> entire dmesg so we can make sure the iommu is really on.  Thanks,
>>
>> Alex
>
> Hi,
>
> No such luck I'm afraid.
>
> Here is the original XML:
>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
>    </hostdev>
>
> The only difference from your recommended change is the target (I take
> it they are target addresses for the VM?) addresses run:
> 0000:00:06.0
> 0000:00:07.0
> 0000:00:08.0
>
> Instead of:
> 0000:00:06.0
> 0000:00:06.1
> 0000:00:06.2
>
> Regardless, I still changed test.xml to:
> <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x1'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x2'/>
>    </hostdev>
>
> To no effect.
>
> In qemu.conf, user and group were commented out, I uncommented both
> and they were both already set to root.
>
> After both a restart of libvirt-bin and the pc itself, no change.
>
> Finally, here is the very latest dmesg:
> http://pastebin.com/9HE61K62
>
> Does anybody know the debug kernel switches for iommu?
>
> Many Thanks,
>
> James.
>

Hi,

A ray of sunshine (ish)

I can apparently pass through this device:

00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
        Subsystem: Giga-byte Technology Device a102
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 16
        Region 0: Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: HDA Intel
        Kernel modules: snd-hda-intel

From test.log:

2011-02-21 22:40:58.888: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
3,sockets=3,cores=1,threads=1 -name test -uuid
307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive file=/home/james/ubuntu_10.10_x86.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=59,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:1 -vga
cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/1
kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
2011-02-21 22:41:48.090: shutting down

Specifically: -device
pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6

Unfortunately I can't get it to boot an ISO through virtmanager! ><
Well, not without deleting the VM and recreating it again.
I shall try getting the latest virt-manager.

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 21, 2011 at 10:49 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 21, 2011 at 10:25 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>>> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>>>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>>>> <alex.williamson@redhat.com> wrote:
>>>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>>>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>>>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>>>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>>> >> >>>> Hi,
>>>> >> >>>>
>>>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>>> >> >>>> 0000:08:06.2"
>>>> >> >>>
>>>> >> >>> This is a rather misleading error message. It is *expected* that
>>>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>>>> >> >>> error messages QEMU is printing aren't much help either, but
>>>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>>>> >> >>> step
>>>> >> >>
>>>> >> >> James, as far as I remember, I had the same issue when I set up my
>>>> >> >> system. Looking at my current (working) boot-script, apparently I've
>>>> >> >> added a 4th line which removes the pci-stub again as a
>>>> >> >> workaround....and it works:
>>>> >> >>
>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>>> >> >>
>>>> >> >> Best regards
>>>> >> >> Kenni
>>>> >> >>
>>>> >> >
>>>> >> > Hi Kenni,
>>>> >> >
>>>> >> > Can I get a bit more information on "boot-script" please? Which file
>>>> >> > exaclty have you put this in? Did you write your own service script
>>>> >> > and put it in init.d?
>>>> >> >
>>>> >> > I've tried this:
>>>> >> >
>>>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>>>> >> >
>>>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>>>> >> >
>>>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>>>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>>>> >> >
>>>> >> > Obviously one looks up which driver is being used by the PCI id, but
>>>> >> > how do I look up which driver my PCI card is using?
>>>> >> >
>>>> >> > Thanks,
>>>> >> >
>>>> >> > James.
>>>> >> >
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> OK, adding the fourth line does nothing, changing the second line to
>>>> >> "driver" rather than "device" does nothing, including in combination
>>>> >> with fourth line there/not there.
>>>> >
>>>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>>>> > change anything about devices that are already bound to it.  driver vs
>>>> > device are just different ways to get to the same thing.
>>>> >
>>>> >> So I'm still stuck, can anybody else help?
>>>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>>>> >> against my kernel?
>>>> >
>>>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>>>> > start from a clean slate and see if it helps.  Here's what I would
>>>> > suggest:
>>>> >
>>>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>>>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>>>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>>>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>>>> >
>>>> > Note we have to knock out the firewire because it shares an interrupt
>>>> > with the ehci device you're trying to assign.  We want to remove the USB
>>>> > controller entirely from the host.  Your dmesg indicates the host is
>>>> > still seeing the device via the uhci ports, and isn't happy about it.
>>>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>>>> > from claiming the device, it's not required for device assignment.  Now,
>>>> > instead of only trying to assign the ehci, let's move the whole usb
>>>> > controller to the guest:
>>>> >
>>>> > -device pci-assign,host=08:06.0,addr=5.0 \
>>>> > -device pci-assign,host=08:06.1,addr=5.1 \
>>>> > -device pci-assign,host=08:06.2,addr=5.2
>>>> >
>>>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>>>> > If that works, then you can bind all those devices to pci-stub and it
>>>> > should still work.
>>>> >
>>>> > Alex
>>>>
>>>> Hi,
>>>>
>>>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>>>> compiling the latest qemu, took me a while to put it all back together
>>>> again in between work!
>>>>
>>>> I'm afraid I use virtmanager, although I guess using the "add pci
>>>> device" function is the same as -device pci-assign? It does seem to
>>>> add it to the command line that gets written out to the log files in
>>>> /var/log/libvirt/qemu.
>>>>
>>>> Nevertheless, I've tried that and still not luck, the log output is
>>>> similar, with one extra line:
>>>>
>>>> http://pastebin.com/MJ6aqjNq
>>>
>>> You actually ended up with:
>>>
>>> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
>>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
>>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>>>
>>> Which isn't quite what I was suggesting.  You probably have xml that
>>> looks like this:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>    </hostdev>
>>>    ...
>>>
>>> Try adding an address line, so you get something more like this:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>>>    </hostdev>
>>>
>>>
>>>> Using raw in/out ioport access (sysfs - Input/output error)
>>>
>>> This is just an informational line to let us know whether pci-sysfs
>>> supports read/write on the ioport resource files.  If it does, we use
>>> those rather than doing raw in/out for access to the device.  This does
>>> highlight another potential problem.  Your distro probably doesn't have
>>> all the patches in place for non-privileged device assignment, which
>>> could be why you're having strange issues.  Check
>>> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
>>> they're not already, try setting them to 'root', restart libvirtd and
>>> see if anything improves.
>>>
>>>> Here's the dmesg output:
>>>> http://pastebin.com/AE1euUN1
>>>
>>> If still issues after the above, it might be useful to pastbin the
>>> entire dmesg so we can make sure the iommu is really on.  Thanks,
>>>
>>> Alex
>>
>> Hi,
>>
>> No such luck I'm afraid.
>>
>> Here is the original XML:
>>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>> function='0x0'/>
>>    </hostdev>
>>
>> The only difference from your recommended change is the target (I take
>> it they are target addresses for the VM?) addresses run:
>> 0000:00:06.0
>> 0000:00:07.0
>> 0000:00:08.0
>>
>> Instead of:
>> 0000:00:06.0
>> 0000:00:06.1
>> 0000:00:06.2
>>
>> Regardless, I still changed test.xml to:
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x1'/>
>>    </hostdev>
>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>      <source>
>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>      </source>
>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x2'/>
>>    </hostdev>
>>
>> To no effect.
>>
>> In qemu.conf, user and group were commented out, I uncommented both
>> and they were both already set to root.
>>
>> After both a restart of libvirt-bin and the pc itself, no change.
>>
>> Finally, here is the very latest dmesg:
>> http://pastebin.com/9HE61K62
>>
>> Does anybody know the debug kernel switches for iommu?
>>
>> Many Thanks,
>>
>> James.
>>
>
> Hi,
>
> A ray of sunshine (ish)
>
> I can apparently pass through this device:
>
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
>        Subsystem: Giga-byte Technology Device a102
>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 32, Cache Line Size: 64 bytes
>        Interrupt: pin ? routed to IRQ 16
>        Region 0: Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
>        Capabilities: <access denied>
>        Kernel driver in use: HDA Intel
>        Kernel modules: snd-hda-intel
>
> From test.log:
>
> 2011-02-21 22:40:58.888: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
> 3,sockets=3,cores=1,threads=1 -name test -uuid
> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
> c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -drive file=/home/james/ubuntu_10.10_x86.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=59,id=hostnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:1 -vga
> cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> char device redirected to /dev/pts/1
> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
> 2011-02-21 22:41:48.090: shutting down
>
> Specifically: -device
> pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>
> Unfortunately I can't get it to boot an ISO through virtmanager! ><
> Well, not without deleting the VM and recreating it again.
> I shall try getting the latest virt-manager.
>
> Regards,
>
> James.
>

Or I could just try pressing the Apply button... ¬.¬;;

J.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 21, 2011 at 10:55 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 21, 2011 at 10:49 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Mon, Feb 21, 2011 at 10:25 PM, James Neave <roboj1m@gmail.com> wrote:
>>> On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
>>> <alex.williamson@redhat.com> wrote:
>>>> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>>>>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>>>>> <alex.williamson@redhat.com> wrote:
>>>>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>>>>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>>>>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>>>>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>>>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>>>> >> >>>> Hi,
>>>>> >> >>>>
>>>>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>>>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>>>> >> >>>> 0000:08:06.2"
>>>>> >> >>>
>>>>> >> >>> This is a rather misleading error message. It is *expected* that
>>>>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>>>>> >> >>> error messages QEMU is printing aren't much help either, but
>>>>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>>>>> >> >>> step
>>>>> >> >>
>>>>> >> >> James, as far as I remember, I had the same issue when I set up my
>>>>> >> >> system. Looking at my current (working) boot-script, apparently I've
>>>>> >> >> added a 4th line which removes the pci-stub again as a
>>>>> >> >> workaround....and it works:
>>>>> >> >>
>>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>>>> >> >>
>>>>> >> >> Best regards
>>>>> >> >> Kenni
>>>>> >> >>
>>>>> >> >
>>>>> >> > Hi Kenni,
>>>>> >> >
>>>>> >> > Can I get a bit more information on "boot-script" please? Which file
>>>>> >> > exaclty have you put this in? Did you write your own service script
>>>>> >> > and put it in init.d?
>>>>> >> >
>>>>> >> > I've tried this:
>>>>> >> >
>>>>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>>>>> >> >
>>>>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>>>>> >> >
>>>>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>>>>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>>>>> >> >
>>>>> >> > Obviously one looks up which driver is being used by the PCI id, but
>>>>> >> > how do I look up which driver my PCI card is using?
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> >
>>>>> >> > James.
>>>>> >> >
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> OK, adding the fourth line does nothing, changing the second line to
>>>>> >> "driver" rather than "device" does nothing, including in combination
>>>>> >> with fourth line there/not there.
>>>>> >
>>>>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>>>>> > change anything about devices that are already bound to it.  driver vs
>>>>> > device are just different ways to get to the same thing.
>>>>> >
>>>>> >> So I'm still stuck, can anybody else help?
>>>>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>>>>> >> against my kernel?
>>>>> >
>>>>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>>>>> > start from a clean slate and see if it helps.  Here's what I would
>>>>> > suggest:
>>>>> >
>>>>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>>>>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>>>>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>>>>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>>>>> >
>>>>> > Note we have to knock out the firewire because it shares an interrupt
>>>>> > with the ehci device you're trying to assign.  We want to remove the USB
>>>>> > controller entirely from the host.  Your dmesg indicates the host is
>>>>> > still seeing the device via the uhci ports, and isn't happy about it.
>>>>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>>>>> > from claiming the device, it's not required for device assignment.  Now,
>>>>> > instead of only trying to assign the ehci, let's move the whole usb
>>>>> > controller to the guest:
>>>>> >
>>>>> > -device pci-assign,host=08:06.0,addr=5.0 \
>>>>> > -device pci-assign,host=08:06.1,addr=5.1 \
>>>>> > -device pci-assign,host=08:06.2,addr=5.2
>>>>> >
>>>>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>>>>> > If that works, then you can bind all those devices to pci-stub and it
>>>>> > should still work.
>>>>> >
>>>>> > Alex
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>>>>> compiling the latest qemu, took me a while to put it all back together
>>>>> again in between work!
>>>>>
>>>>> I'm afraid I use virtmanager, although I guess using the "add pci
>>>>> device" function is the same as -device pci-assign? It does seem to
>>>>> add it to the command line that gets written out to the log files in
>>>>> /var/log/libvirt/qemu.
>>>>>
>>>>> Nevertheless, I've tried that and still not luck, the log output is
>>>>> similar, with one extra line:
>>>>>
>>>>> http://pastebin.com/MJ6aqjNq
>>>>
>>>> You actually ended up with:
>>>>
>>>> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
>>>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
>>>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>>>>
>>>> Which isn't quite what I was suggesting.  You probably have xml that
>>>> looks like this:
>>>>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>      <source>
>>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>>      </source>
>>>>    </hostdev>
>>>>    ...
>>>>
>>>> Try adding an address line, so you get something more like this:
>>>>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>      <source>
>>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>>      </source>
>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>>>>    </hostdev>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>      <source>
>>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>>>      </source>
>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>>>>    </hostdev>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>      <source>
>>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>>>      </source>
>>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>>>>    </hostdev>
>>>>
>>>>
>>>>> Using raw in/out ioport access (sysfs - Input/output error)
>>>>
>>>> This is just an informational line to let us know whether pci-sysfs
>>>> supports read/write on the ioport resource files.  If it does, we use
>>>> those rather than doing raw in/out for access to the device.  This does
>>>> highlight another potential problem.  Your distro probably doesn't have
>>>> all the patches in place for non-privileged device assignment, which
>>>> could be why you're having strange issues.  Check
>>>> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
>>>> they're not already, try setting them to 'root', restart libvirtd and
>>>> see if anything improves.
>>>>
>>>>> Here's the dmesg output:
>>>>> http://pastebin.com/AE1euUN1
>>>>
>>>> If still issues after the above, it might be useful to pastbin the
>>>> entire dmesg so we can make sure the iommu is really on.  Thanks,
>>>>
>>>> Alex
>>>
>>> Hi,
>>>
>>> No such luck I'm afraid.
>>>
>>> Here is the original XML:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>>> function='0x0'/>
>>>    </hostdev>
>>>
>>> The only difference from your recommended change is the target (I take
>>> it they are target addresses for the VM?) addresses run:
>>> 0000:00:06.0
>>> 0000:00:07.0
>>> 0000:00:08.0
>>>
>>> Instead of:
>>> 0000:00:06.0
>>> 0000:00:06.1
>>> 0000:00:06.2
>>>
>>> Regardless, I still changed test.xml to:
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x1'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>      <source>
>>>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>>>      </source>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x2'/>
>>>    </hostdev>
>>>
>>> To no effect.
>>>
>>> In qemu.conf, user and group were commented out, I uncommented both
>>> and they were both already set to root.
>>>
>>> After both a restart of libvirt-bin and the pc itself, no change.
>>>
>>> Finally, here is the very latest dmesg:
>>> http://pastebin.com/9HE61K62
>>>
>>> Does anybody know the debug kernel switches for iommu?
>>>
>>> Many Thanks,
>>>
>>> James.
>>>
>>
>> Hi,
>>
>> A ray of sunshine (ish)
>>
>> I can apparently pass through this device:
>>
>> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
>>        Subsystem: Giga-byte Technology Device a102
>>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 32, Cache Line Size: 64 bytes
>>        Interrupt: pin ? routed to IRQ 16
>>        Region 0: Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
>>        Capabilities: <access denied>
>>        Kernel driver in use: HDA Intel
>>        Kernel modules: snd-hda-intel
>>
>> From test.log:
>>
>> 2011-02-21 22:40:58.888: starting up
>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
>> 3,sockets=3,cores=1,threads=1 -name test -uuid
>> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
>> c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
>> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> -drive file=/home/james/ubuntu_10.10_x86.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> -netdev tap,fd=59,id=hostnet0 -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:1 -vga
>> cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> char device redirected to /dev/pts/1
>> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
>> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
>> 2011-02-21 22:41:48.090: shutting down
>>
>> Specifically: -device
>> pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>>
>> Unfortunately I can't get it to boot an ISO through virtmanager! ><
>> Well, not without deleting the VM and recreating it again.
>> I shall try getting the latest virt-manager.
>>
>> Regards,
>>
>> James.
>>
>
> Or I could just try pressing the Apply button... ¬.¬;;
>
> J.
>

OK, it doesn't work but here's the dmesg from the test vm with the
host's on board sound passed through:

http://pastebin.com/N2bdMQTa

Specifically the error is:

[   13.795117] hda-intel: unable to grab IRQ 0, disabling device
[   13.797976] HDA Intel: probe of 0000:00:06.0 failed with error -16

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> Finally, here is the very latest dmesg:
> http://pastebin.com/9HE61K62

OK, this is an AMD IOMMU box.

[    0.000000] ACPI: IVRS 00000000cfcf9830 000E0 (v01  AMD     RD890S 00202031 AMD  00000000)

It's discovered and enalbed properly:

[    0.698992] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
[    0.710287] AMD-Vi: Lazy IO/TLB flushing enabled

> Does anybody know the debug kernel switches for iommu?

Two helpful kernel commandline options are:

amd_iommu_dump debug (and drop "quiet")

The problem is when you attach the device (function) you're getting
stuck up in conflicts with the existing domain for that function.

My guess is that all the functions are behind a PCI to PCI bridge, so the alias
lookup is finding a conflict.

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> Finally, here is the very latest dmesg:
>> http://pastebin.com/9HE61K62
>
> OK, this is an AMD IOMMU box.
>
> [    0.000000] ACPI: IVRS 00000000cfcf9830 000E0 (v01  AMD     RD890S 00202031 AMD  00000000)
>
> It's discovered and enalbed properly:
>
> [    0.698992] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> [    0.710287] AMD-Vi: Lazy IO/TLB flushing enabled
>
>> Does anybody know the debug kernel switches for iommu?
>
> Two helpful kernel commandline options are:
>
> amd_iommu_dump debug (and drop "quiet")
>
> The problem is when you attach the device (function) you're getting
> stuck up in conflicts with the existing domain for that function.
>
> My guess is that all the functions are behind a PCI to PCI bridge, so the alias
> lookup is finding a conflict.
>
> thanks,
> -chris
>

Hi,

Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
earlier email:

cat /proc/interruts
http://pastebin.com/LQdB3hms

lspci -vvv
http://pastebin.com/GJDkC8B4

lspci -t -v
http://pastebin.com/Ftx8Hfjt

I'll be interested to see whether the error messages revert to this
-ebusy now that the machine has been powered off.
Last thinh last night I got this:

[ 5813.048427] assign device 0:8:6.0
[ 5813.048463] type=1400 audit(1298331555.057:42): apparmor="DENIED"
operation="capable" parent=1
profile="libvirt-307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc" pid=3236
comm="kvm" capability=17  capname="sys_rawio"
[ 5813.048505] deassign device 0:8:6.0

I added "capability sys_rawio" to the libvirtd apparmor profile,
restarted apparmor and libvirt-bin and it just complained about the
apparmor profile.
The only other thing I did was compile the latest virt-manager 0.8.6,
don't see how that would have made any difference.

Many Thank,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Roedel, Joerg]

On Tue, Feb 22, 2011 at 04:18:20AM -0500, James Neave wrote:
> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> I added "capability sys_rawio" to the libvirtd apparmor profile,
> restarted apparmor and libvirt-bin and it just complained about the
> apparmor profile.
> The only other thing I did was compile the latest virt-manager 0.8.6,
> don't see how that would have made any difference.

Can you disable apparmor completly for testing purposes and try again?

	Joerg


-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

2011/2/22 Roedel, Joerg <Joerg.Roedel@amd.com>:
> On Tue, Feb 22, 2011 at 04:18:20AM -0500, James Neave wrote:
>> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> I added "capability sys_rawio" to the libvirtd apparmor profile,
>> restarted apparmor and libvirt-bin and it just complained about the
>> apparmor profile.
>> The only other thing I did was compile the latest virt-manager 0.8.6,
>> don't see how that would have made any difference.
>
> Can you disable apparmor completly for testing purposes and try again?
>
>        Joerg
>
>
> --
> AMD Operating System Research Center
>
> Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
> General Managers: Alberto Bozzo, Andrew Bowd
> Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
>
>

Hi,

Never tried that before, although I guess this is how you do
it:https://help.ubuntu.com/community/AppArmor#Disable%20AppArmor%20framework

I'll try that next time I get some time home, which probably won't be
until Wednesday evening GMT.

That's assuming that after the powercycle the machine doesn't revert
to it's "pci-stub is occupying the device" error! (still not really
sure how I stopped that happening, it just stopped after I tried to
pass through the sound card)

Regardless, many thanks and I shall return later this week.

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> > * James Neave (roboj1m@gmail.com) wrote:
> >> Does anybody know the debug kernel switches for iommu?
> >
> > Two helpful kernel commandline options are:
> >
> > amd_iommu_dump debug (and drop "quiet")
> >
> > The problem is when you attach the device (function) you're getting
> > stuck up in conflicts with the existing domain for that function.
> >
> > My guess is that all the functions are behind a PCI to PCI bridge, so the alias
> > lookup is finding a conflict.
> 
> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
> earlier email:

Sorry, I missed that in your original mail, thanks for reposting.

> cat /proc/interruts
> http://pastebin.com/LQdB3hms
> 
> lspci -vvv
> http://pastebin.com/GJDkC8B4

 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)

> lspci -t -v
> http://pastebin.com/Ftx8Hfjt

Yup, that's what I expected:

 +-14.4-[08]--+-06.0  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
 |            +-06.1  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
 |            +-06.2  VIA Technologies, Inc. USB 2.0
 |            \-0e.0  Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller

I'd now expect to see (if you boot with amd_iommu_dump) some IVRS
details showing an alias range entry basically showing 08:* pointing
back to 00:14.4.  This means that from the point of view of the IOMMU the
devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if they
are 00:14.4.

When you assign a device to a guest, the guest VM gets an IOMMU domain
(a context to manage IOMMU page table mappings) and the device is put
into that guest's IOMMU domain.  However, if the device is behind a
PCI-PCI bridge it will appear as an alias for the bridge itself.  The
bridge is a PCI device with an IOMMU domain.  When trying to assign a
device to a guest there's some sanity checking to verify that the device
(or its alias) aren't already under some IOMMU domain other than the
guest VM's IOMMU domain.

I suspect this is what you are hitting.  You could test this theory by
adding 2 more devices to your guest -- the firewire device (08:0e.0)
and the PCI-PCI bridge itself (00:14.4).

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Wed, Feb 23, 2011 at 12:11 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> > * James Neave (roboj1m@gmail.com) wrote:
>> >> Does anybody know the debug kernel switches for iommu?
>> >
>> > Two helpful kernel commandline options are:
>> >
>> > amd_iommu_dump debug (and drop "quiet")
>> >
>> > The problem is when you attach the device (function) you're getting
>> > stuck up in conflicts with the existing domain for that function.
>> >
>> > My guess is that all the functions are behind a PCI to PCI bridge, so the alias
>> > lookup is finding a conflict.
>>
>> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
>> earlier email:
>
> Sorry, I missed that in your original mail, thanks for reposting.
>
>> cat /proc/interruts
>> http://pastebin.com/LQdB3hms
>>
>> lspci -vvv
>> http://pastebin.com/GJDkC8B4
>
>  00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>
>> lspci -t -v
>> http://pastebin.com/Ftx8Hfjt
>
> Yup, that's what I expected:
>
>  +-14.4-[08]--+-06.0  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>  |            +-06.1  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>  |            +-06.2  VIA Technologies, Inc. USB 2.0
>  |            \-0e.0  Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller
>
> I'd now expect to see (if you boot with amd_iommu_dump) some IVRS
> details showing an alias range entry basically showing 08:* pointing
> back to 00:14.4.  This means that from the point of view of the IOMMU the
> devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if they
> are 00:14.4.
>
> When you assign a device to a guest, the guest VM gets an IOMMU domain
> (a context to manage IOMMU page table mappings) and the device is put
> into that guest's IOMMU domain.  However, if the device is behind a
> PCI-PCI bridge it will appear as an alias for the bridge itself.  The
> bridge is a PCI device with an IOMMU domain.  When trying to assign a
> device to a guest there's some sanity checking to verify that the device
> (or its alias) aren't already under some IOMMU domain other than the
> guest VM's IOMMU domain.
>
> I suspect this is what you are hitting.  You could test this theory by
> adding 2 more devices to your guest -- the firewire device (08:0e.0)
> and the PCI-PCI bridge itself (00:14.4).
>
> thanks,
> -chris
>

Hi,

OK, here we go again!

Right, I've diabled apparmor (I think) with this:

sudo invoke-rc.d apparmor stop
sudo update-rc.d -f apparmor remove

After a reboot I'm back to getting the error about pci-stub claiming the device.
Apparmor being off made no difference to that (except there are no
apparmor messages in dmesg)

Then I try adding 08:0e.0 and 00:14.4 to the VM and I get this error message:

libvirtError: this function is not supported by the connection driver:
Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
available

There is nothing written to test.log when you try to start the VM with
00:14.4 attached.

At this point libvirt goes screwy and I have to restart it before I
can remove 00:14.4 from the VM.

However, once I've done THAT, starting the VM gets the different error
message in test.log and a new dmesg:

2011-02-23 19:21:13.483: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
3,sockets=3,cores=1,threads=1 -name test -uuid
307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
order=cd,menu=off -drive
file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=57,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga
cirrus -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6
-device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7
-device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
-device pci-assign,host=08:0e.0,id=hostdev3,configfd=61,bus=pci.0,addr=0xa
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/1
kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
Using raw in/out ioport access (sysfs - Input/output error)
Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?
kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6:
Device 'pci-assign' could not be initialized
2011-02-23 19:21:13.958: shutting down

dmesg:
http://pastebin.com/70D26xp4

This bit is different:

[  201.625221] uhci_hcd 0000:08:06.0: remove, state 4
[  201.625237] usb usb4: USB disconnect, address 1
[  201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
[  201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled
[  201.626028] pci-stub 0000:08:06.0: claimed by stub
[  201.631922] uhci_hcd 0000:08:06.1: remove, state 4
[  201.631937] usb usb9: USB disconnect, address 1
[  201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered
[  201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled
[  201.632419] pci-stub 0000:08:06.1: claimed by stub
[  201.638160] ehci_hcd 0000:08:06.2: remove, state 1
[  201.638172] usb usb10: USB disconnect, address 1
[  201.638178] usb 10-1: USB disconnect, address 2
[  201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully
deinitialized and disconnected.
[  201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered
[  201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled
[  201.725042] pci-stub 0000:08:06.2: claimed by stub
[  201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled
[  201.731838] firewire_ohci: Removed fw-ohci device.
[  201.732536] pci-stub 0000:08:0e.0: claimed by stub
[  202.303880] device vnet0 entered promiscuous mode
[  202.305184] virbr0: topology change detected, propagating
[  202.305193] virbr0: port 1(vnet0) entering forwarding state
[  202.305199] virbr0: port 1(vnet0) entering forwarding state
[  202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
[  202.470076] pci-stub 0000:08:06.0: restoring config space at offset
0x1 (was 0x2100000, writing 0x2100001)
[  202.697270] assign device 0:8:6.0
[  202.697325] deassign device 0:8:6.0
[  202.730080] pci-stub 0000:08:06.0: restoring config space at offset
0x1 (was 0x2100000, writing 0x2100001)
[  202.730107] pci-stub 0000:08:06.0: PCI INT A disabled

This time the pci-stub claimed lines are not all bunched up and there
is only one per device, rather than three per device.
Also for the first time it says "assign device 0:8:6.0" rather than
"assign device 0:8:6.0 failed"
It them immediately deassigns the device and stops.

test.log shows:

Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?

lspsci -vv for the relevant devices shows:
http://pastebin.com/EUtUMj8x

00:14.4 now appears to be using pci-stub as it's driver, as well as
08:06.1, 2, 3 but not 0e.0

Anyway, that's all for now.
I think I'll try 'amd_iommu_dump' next, does it write to dmesg?

Many Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Wed, Feb 23, 2011 at 7:44 PM, James Neave <roboj1m@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 12:11 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> * James Neave (roboj1m@gmail.com) wrote:
>>> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>> > * James Neave (roboj1m@gmail.com) wrote:
>>> >> Does anybody know the debug kernel switches for iommu?
>>> >
>>> > Two helpful kernel commandline options are:
>>> >
>>> > amd_iommu_dump debug (and drop "quiet")
>>> >
>>> > The problem is when you attach the device (function) you're getting
>>> > stuck up in conflicts with the existing domain for that function.
>>> >
>>> > My guess is that all the functions are behind a PCI to PCI bridge, so the alias
>>> > lookup is finding a conflict.
>>>
>>> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
>>> earlier email:
>>
>> Sorry, I missed that in your original mail, thanks for reposting.
>>
>>> cat /proc/interruts
>>> http://pastebin.com/LQdB3hms
>>>
>>> lspci -vvv
>>> http://pastebin.com/GJDkC8B4
>>
>>  00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>>
>>> lspci -t -v
>>> http://pastebin.com/Ftx8Hfjt
>>
>> Yup, that's what I expected:
>>
>>  +-14.4-[08]--+-06.0  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>  |            +-06.1  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>  |            +-06.2  VIA Technologies, Inc. USB 2.0
>>  |            \-0e.0  Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller
>>
>> I'd now expect to see (if you boot with amd_iommu_dump) some IVRS
>> details showing an alias range entry basically showing 08:* pointing
>> back to 00:14.4.  This means that from the point of view of the IOMMU the
>> devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if they
>> are 00:14.4.
>>
>> When you assign a device to a guest, the guest VM gets an IOMMU domain
>> (a context to manage IOMMU page table mappings) and the device is put
>> into that guest's IOMMU domain.  However, if the device is behind a
>> PCI-PCI bridge it will appear as an alias for the bridge itself.  The
>> bridge is a PCI device with an IOMMU domain.  When trying to assign a
>> device to a guest there's some sanity checking to verify that the device
>> (or its alias) aren't already under some IOMMU domain other than the
>> guest VM's IOMMU domain.
>>
>> I suspect this is what you are hitting.  You could test this theory by
>> adding 2 more devices to your guest -- the firewire device (08:0e.0)
>> and the PCI-PCI bridge itself (00:14.4).
>>
>> thanks,
>> -chris
>>
>
> Hi,
>
> OK, here we go again!
>
> Right, I've diabled apparmor (I think) with this:
>
> sudo invoke-rc.d apparmor stop
> sudo update-rc.d -f apparmor remove
>
> After a reboot I'm back to getting the error about pci-stub claiming the device.
> Apparmor being off made no difference to that (except there are no
> apparmor messages in dmesg)
>
> Then I try adding 08:0e.0 and 00:14.4 to the VM and I get this error message:
>
> libvirtError: this function is not supported by the connection driver:
> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
> available
>
> There is nothing written to test.log when you try to start the VM with
> 00:14.4 attached.
>
> At this point libvirt goes screwy and I have to restart it before I
> can remove 00:14.4 from the VM.
>
> However, once I've done THAT, starting the VM gets the different error
> message in test.log and a new dmesg:
>
> 2011-02-23 19:21:13.483: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
> 3,sockets=3,cores=1,threads=1 -name test -uuid
> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
> order=cd,menu=off -drive
> file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=57,id=hostnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga
> cirrus -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6
> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7
> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
> -device pci-assign,host=08:0e.0,id=hostdev3,configfd=61,bus=pci.0,addr=0xa
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> char device redirected to /dev/pts/1
> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
> Using raw in/out ioport access (sysfs - Input/output error)
> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?
> kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6:
> Device 'pci-assign' could not be initialized
> 2011-02-23 19:21:13.958: shutting down
>
> dmesg:
> http://pastebin.com/70D26xp4
>
> This bit is different:
>
> [  201.625221] uhci_hcd 0000:08:06.0: remove, state 4
> [  201.625237] usb usb4: USB disconnect, address 1
> [  201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
> [  201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled
> [  201.626028] pci-stub 0000:08:06.0: claimed by stub
> [  201.631922] uhci_hcd 0000:08:06.1: remove, state 4
> [  201.631937] usb usb9: USB disconnect, address 1
> [  201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered
> [  201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled
> [  201.632419] pci-stub 0000:08:06.1: claimed by stub
> [  201.638160] ehci_hcd 0000:08:06.2: remove, state 1
> [  201.638172] usb usb10: USB disconnect, address 1
> [  201.638178] usb 10-1: USB disconnect, address 2
> [  201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully
> deinitialized and disconnected.
> [  201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered
> [  201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled
> [  201.725042] pci-stub 0000:08:06.2: claimed by stub
> [  201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled
> [  201.731838] firewire_ohci: Removed fw-ohci device.
> [  201.732536] pci-stub 0000:08:0e.0: claimed by stub
> [  202.303880] device vnet0 entered promiscuous mode
> [  202.305184] virbr0: topology change detected, propagating
> [  202.305193] virbr0: port 1(vnet0) entering forwarding state
> [  202.305199] virbr0: port 1(vnet0) entering forwarding state
> [  202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  202.470076] pci-stub 0000:08:06.0: restoring config space at offset
> 0x1 (was 0x2100000, writing 0x2100001)
> [  202.697270] assign device 0:8:6.0
> [  202.697325] deassign device 0:8:6.0
> [  202.730080] pci-stub 0000:08:06.0: restoring config space at offset
> 0x1 (was 0x2100000, writing 0x2100001)
> [  202.730107] pci-stub 0000:08:06.0: PCI INT A disabled
>
> This time the pci-stub claimed lines are not all bunched up and there
> is only one per device, rather than three per device.
> Also for the first time it says "assign device 0:8:6.0" rather than
> "assign device 0:8:6.0 failed"
> It them immediately deassigns the device and stops.
>
> test.log shows:
>
> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?
>
> lspsci -vv for the relevant devices shows:
> http://pastebin.com/EUtUMj8x
>
> 00:14.4 now appears to be using pci-stub as it's driver, as well as
> 08:06.1, 2, 3 but not 0e.0
>
> Anyway, that's all for now.
> I think I'll try 'amd_iommu_dump' next, does it write to dmesg?
>
> Many Thanks,
>
> James.
>

OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
http://pastebin.com/JxEwvqRA

I've just figured out a sequence of "echo DEV > PATH" commands to call
for 14.4 gets me past the "claimed by pci-stub" error and gets me to
the "failed to assign IRQ" error.
I'm going to narrow down the required sequence and then post it.

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Wed, Feb 23, 2011 at 8:09 PM, James Neave <roboj1m@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 7:44 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Wed, Feb 23, 2011 at 12:11 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>> * James Neave (roboj1m@gmail.com) wrote:
>>>> On Tue, Feb 22, 2011 at 1:51 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>>> > * James Neave (roboj1m@gmail.com) wrote:
>>>> >> Does anybody know the debug kernel switches for iommu?
>>>> >
>>>> > Two helpful kernel commandline options are:
>>>> >
>>>> > amd_iommu_dump debug (and drop "quiet")
>>>> >
>>>> > The problem is when you attach the device (function) you're getting
>>>> > stuck up in conflicts with the existing domain for that function.
>>>> >
>>>> > My guess is that all the functions are behind a PCI to PCI bridge, so the alias
>>>> > lookup is finding a conflict.
>>>>
>>>> Yes, it's behind a PCI-PCI bridge I think, here's the blurb from an
>>>> earlier email:
>>>
>>> Sorry, I missed that in your original mail, thanks for reposting.
>>>
>>>> cat /proc/interruts
>>>> http://pastebin.com/LQdB3hms
>>>>
>>>> lspci -vvv
>>>> http://pastebin.com/GJDkC8B4
>>>
>>>  00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>>>
>>>> lspci -t -v
>>>> http://pastebin.com/Ftx8Hfjt
>>>
>>> Yup, that's what I expected:
>>>
>>>  +-14.4-[08]--+-06.0  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>>  |            +-06.1  VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
>>>  |            +-06.2  VIA Technologies, Inc. USB 2.0
>>>  |            \-0e.0  Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller
>>>
>>> I'd now expect to see (if you boot with amd_iommu_dump) some IVRS
>>> details showing an alias range entry basically showing 08:* pointing
>>> back to 00:14.4.  This means that from the point of view of the IOMMU the
>>> devices 08:06.0, 08:06.1, 08:06.2, 08:0e.0 will all show up as if they
>>> are 00:14.4.
>>>
>>> When you assign a device to a guest, the guest VM gets an IOMMU domain
>>> (a context to manage IOMMU page table mappings) and the device is put
>>> into that guest's IOMMU domain.  However, if the device is behind a
>>> PCI-PCI bridge it will appear as an alias for the bridge itself.  The
>>> bridge is a PCI device with an IOMMU domain.  When trying to assign a
>>> device to a guest there's some sanity checking to verify that the device
>>> (or its alias) aren't already under some IOMMU domain other than the
>>> guest VM's IOMMU domain.
>>>
>>> I suspect this is what you are hitting.  You could test this theory by
>>> adding 2 more devices to your guest -- the firewire device (08:0e.0)
>>> and the PCI-PCI bridge itself (00:14.4).
>>>
>>> thanks,
>>> -chris
>>>
>>
>> Hi,
>>
>> OK, here we go again!
>>
>> Right, I've diabled apparmor (I think) with this:
>>
>> sudo invoke-rc.d apparmor stop
>> sudo update-rc.d -f apparmor remove
>>
>> After a reboot I'm back to getting the error about pci-stub claiming the device.
>> Apparmor being off made no difference to that (except there are no
>> apparmor messages in dmesg)
>>
>> Then I try adding 08:0e.0 and 00:14.4 to the VM and I get this error message:
>>
>> libvirtError: this function is not supported by the connection driver:
>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>> available
>>
>> There is nothing written to test.log when you try to start the VM with
>> 00:14.4 attached.
>>
>> At this point libvirt goes screwy and I have to restart it before I
>> can remove 00:14.4 from the VM.
>>
>> However, once I've done THAT, starting the VM gets the different error
>> message in test.log and a new dmesg:
>>
>> 2011-02-23 19:21:13.483: starting up
>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
>> 3,sockets=3,cores=1,threads=1 -name test -uuid
>> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
>> order=cd,menu=off -drive
>> file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
>> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> -netdev tap,fd=57,id=hostnet0 -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga
>> cirrus -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6
>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7
>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>> -device pci-assign,host=08:0e.0,id=hostdev3,configfd=61,bus=pci.0,addr=0xa
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> char device redirected to /dev/pts/1
>> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
>> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
>> Using raw in/out ioport access (sysfs - Input/output error)
>> Failed to assign irq for "hostdev0": Operation not permitted
>> Perhaps you are assigning a device that shares an IRQ with another device?
>> kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6:
>> Device 'pci-assign' could not be initialized
>> 2011-02-23 19:21:13.958: shutting down
>>
>> dmesg:
>> http://pastebin.com/70D26xp4
>>
>> This bit is different:
>>
>> [  201.625221] uhci_hcd 0000:08:06.0: remove, state 4
>> [  201.625237] usb usb4: USB disconnect, address 1
>> [  201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
>> [  201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled
>> [  201.626028] pci-stub 0000:08:06.0: claimed by stub
>> [  201.631922] uhci_hcd 0000:08:06.1: remove, state 4
>> [  201.631937] usb usb9: USB disconnect, address 1
>> [  201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered
>> [  201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled
>> [  201.632419] pci-stub 0000:08:06.1: claimed by stub
>> [  201.638160] ehci_hcd 0000:08:06.2: remove, state 1
>> [  201.638172] usb usb10: USB disconnect, address 1
>> [  201.638178] usb 10-1: USB disconnect, address 2
>> [  201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully
>> deinitialized and disconnected.
>> [  201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered
>> [  201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled
>> [  201.725042] pci-stub 0000:08:06.2: claimed by stub
>> [  201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled
>> [  201.731838] firewire_ohci: Removed fw-ohci device.
>> [  201.732536] pci-stub 0000:08:0e.0: claimed by stub
>> [  202.303880] device vnet0 entered promiscuous mode
>> [  202.305184] virbr0: topology change detected, propagating
>> [  202.305193] virbr0: port 1(vnet0) entering forwarding state
>> [  202.305199] virbr0: port 1(vnet0) entering forwarding state
>> [  202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
>> [  202.470076] pci-stub 0000:08:06.0: restoring config space at offset
>> 0x1 (was 0x2100000, writing 0x2100001)
>> [  202.697270] assign device 0:8:6.0
>> [  202.697325] deassign device 0:8:6.0
>> [  202.730080] pci-stub 0000:08:06.0: restoring config space at offset
>> 0x1 (was 0x2100000, writing 0x2100001)
>> [  202.730107] pci-stub 0000:08:06.0: PCI INT A disabled
>>
>> This time the pci-stub claimed lines are not all bunched up and there
>> is only one per device, rather than three per device.
>> Also for the first time it says "assign device 0:8:6.0" rather than
>> "assign device 0:8:6.0 failed"
>> It them immediately deassigns the device and stops.
>>
>> test.log shows:
>>
>> Failed to assign irq for "hostdev0": Operation not permitted
>> Perhaps you are assigning a device that shares an IRQ with another device?
>>
>> lspsci -vv for the relevant devices shows:
>> http://pastebin.com/EUtUMj8x
>>
>> 00:14.4 now appears to be using pci-stub as it's driver, as well as
>> 08:06.1, 2, 3 but not 0e.0
>>
>> Anyway, that's all for now.
>> I think I'll try 'amd_iommu_dump' next, does it write to dmesg?
>>
>> Many Thanks,
>>
>> James.
>>
>
> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
> http://pastebin.com/JxEwvqRA
>
> I've just figured out a sequence of "echo DEV > PATH" commands to call
> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
> the "failed to assign IRQ" error.
> I'm going to narrow down the required sequence and then post it.
>
> Regards,
>
> James.
>

Hi,

Just out of interest, what kind of mileage would I expect out of
buying a shiny new PCIe tuner?
Can I pass through PCIe? Would it work better because it wouldn't be
behind a bridge? WOULD it not be behind a bridge?
As much as I'd hate to solve a problem with the application of money... :(
(OT question, on mailing lists should I use Reply All or just reply
and change the To address to kvm.vger.kernel.org?)

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> Just out of interest, what kind of mileage would I expect out of
> buying a shiny new PCIe tuner?

Hard to say.  One advantage would be if it's using MSI or MSI-X
interrupts.

> Can I pass through PCIe?

Often, yes (still some caveats w.r.t. extended config space I believe).

> Would it work better because it wouldn't be
> behind a bridge? WOULD it not be behind a bridge?

You should have a PCIe slot that does not sit behind a PCI-PCI bridge.

> As much as I'd hate to solve a problem with the application of money... :(

If you just want _one_ tuner to go to the guest, you should be able to
do that by unbinding the other devices and giving the guest just the one
usb controller (assuming just assigning the usb device itself is hitting
usb/qemu stack limitations).  The trick is to be sure to unbind any host
devices that are sharing interrupts with the one device you want the
guest to have.  With USB controllers you just have to be sure you know
which ports they map to so you don't kill a keyboard, mouse, external
disk, etc...

> (OT question, on mailing lists should I use Reply All or just reply
> and change the To address to kvm.vger.kernel.org?)

Reply all is best.

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
> http://pastebin.com/JxEwvqRA

Yeah, that's what I expected:

[    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
[    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7

That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
should all go into same iommu domain).

> I've just figured out a sequence of "echo DEV > PATH" commands to call
> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
> the "failed to assign IRQ" error.
> I'm going to narrow down the required sequence and then post it.

Kind of afraid to ask, but does it include:

(assuming 1002 4384 is the pci to pci bridge)
echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind

(this has the side effect of detaching the bridge from its domain)

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
>> http://pastebin.com/JxEwvqRA
>
> Yeah, that's what I expected:
>
> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
>
> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
> should all go into same iommu domain).
>
>> I've just figured out a sequence of "echo DEV > PATH" commands to call
>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
>> the "failed to assign IRQ" error.
>> I'm going to narrow down the required sequence and then post it.
>
> Kind of afraid to ask, but does it include:
>
> (assuming 1002 4384 is the pci to pci bridge)
> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
>
> (this has the side effect of detaching the bridge from its domain)
>
> thanks,
> -chris
>

Exact sequence is:

echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind

I take it this is a bad thing then?

> I assume this means that 00:14.4 is still left claimed by pci-stub?

Yes

> How are you determining this?  The lspci paste above has pci-stub for all
> of them.  The easiest thing might be to start with manually disabling
> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
> Then giving the guest only 08:06.1.

I determined it by being half asleep and not reading it properly... >.<
You're right, all 5 devices were using pci-stub

>> libvirtError: this function is not supported by the connection driver:
>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>> available

> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
> libvirt here).

What does that libvirt error mean? I can't find a definition.
Am I limiting myself by using libvirt? Would not using it help and how
would I go about not using it?

> Trouble now is that
> with shared IRQ we don't have a good way to handle that right now.

Game over then?
I've tried assigning the USB devices before, I couldn't do it because
qemu doesn't support USB2 devices.
I don't really understand where this IRQ conflict is, the firewire and
the USB2 device share IRQ22 but I'm assigning them both to the VM?
Is that still a problem?
I don't suppose there's any way to change which IRQ they use in the
BIOS or with a command is there?

I don't know if it means anything but this page:

http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200

Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
My knowledge it very limited on this subject so I don't know if that's
meaningless looking at the output from another person's lspci.

Anything left to try?

Regardless, many thanks for your help,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Fri, Feb 25, 2011 at 10:47 PM, James Neave <roboj1m@gmail.com> wrote:
> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> * James Neave (roboj1m@gmail.com) wrote:
>>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
>>> http://pastebin.com/JxEwvqRA
>>
>> Yeah, that's what I expected:
>>
>> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
>> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
>>
>> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
>> should all go into same iommu domain).
>>
>>> I've just figured out a sequence of "echo DEV > PATH" commands to call
>>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
>>> the "failed to assign IRQ" error.
>>> I'm going to narrow down the required sequence and then post it.
>>
>> Kind of afraid to ask, but does it include:
>>
>> (assuming 1002 4384 is the pci to pci bridge)
>> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
>> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
>>
>> (this has the side effect of detaching the bridge from its domain)
>>
>> thanks,
>> -chris
>>
>
> Exact sequence is:
>
> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
>
> I take it this is a bad thing then?
>
>> I assume this means that 00:14.4 is still left claimed by pci-stub?
>
> Yes
>
>> How are you determining this?  The lspci paste above has pci-stub for all
>> of them.  The easiest thing might be to start with manually disabling
>> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
>> Then giving the guest only 08:06.1.
>
> I determined it by being half asleep and not reading it properly... >.<
> You're right, all 5 devices were using pci-stub
>
>>> libvirtError: this function is not supported by the connection driver:
>>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>>> available
>
>> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
>> libvirt here).
>
> What does that libvirt error mean? I can't find a definition.
> Am I limiting myself by using libvirt? Would not using it help and how
> would I go about not using it?
>
>> Trouble now is that
>> with shared IRQ we don't have a good way to handle that right now.
>
> Game over then?
> I've tried assigning the USB devices before, I couldn't do it because
> qemu doesn't support USB2 devices.
> I don't really understand where this IRQ conflict is, the firewire and
> the USB2 device share IRQ22 but I'm assigning them both to the VM?
> Is that still a problem?
> I don't suppose there's any way to change which IRQ they use in the
> BIOS or with a command is there?
>
> I don't know if it means anything but this page:
>
> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
>
> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
> My knowledge it very limited on this subject so I don't know if that's
> meaningless looking at the output from another person's lspci.
>
> Anything left to try?
>
> Regardless, many thanks for your help,
>
> James.
>

On the off chance I tried disabling the firewire in the BIOS, which
leaves only my tuner card using IRQ 20, 21 and 22.
No difference, still complains about IRQs:

Using raw in/out ioport access (sysfs - Input/output error)
Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?

It does say "Operation not permitted" and that only "perhaps" I am
assigning a device that shares an IRQ.
Perhaps IRQ conflict it not the problem? They really are sitting on
their own. Another permissions problem perhaps?

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Fri, Feb 25, 2011 at 11:02 PM, James Neave <roboj1m@gmail.com> wrote:
> On Fri, Feb 25, 2011 at 10:47 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>>> * James Neave (roboj1m@gmail.com) wrote:
>>>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
>>>> http://pastebin.com/JxEwvqRA
>>>
>>> Yeah, that's what I expected:
>>>
>>> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
>>> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
>>>
>>> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
>>> should all go into same iommu domain).
>>>
>>>> I've just figured out a sequence of "echo DEV > PATH" commands to call
>>>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
>>>> the "failed to assign IRQ" error.
>>>> I'm going to narrow down the required sequence and then post it.
>>>
>>> Kind of afraid to ask, but does it include:
>>>
>>> (assuming 1002 4384 is the pci to pci bridge)
>>> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
>>> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
>>>
>>> (this has the side effect of detaching the bridge from its domain)
>>>
>>> thanks,
>>> -chris
>>>
>>
>> Exact sequence is:
>>
>> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
>> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
>>
>> I take it this is a bad thing then?
>>
>>> I assume this means that 00:14.4 is still left claimed by pci-stub?
>>
>> Yes
>>
>>> How are you determining this?  The lspci paste above has pci-stub for all
>>> of them.  The easiest thing might be to start with manually disabling
>>> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
>>> Then giving the guest only 08:06.1.
>>
>> I determined it by being half asleep and not reading it properly... >.<
>> You're right, all 5 devices were using pci-stub
>>
>>>> libvirtError: this function is not supported by the connection driver:
>>>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>>>> available
>>
>>> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
>>> libvirt here).
>>
>> What does that libvirt error mean? I can't find a definition.
>> Am I limiting myself by using libvirt? Would not using it help and how
>> would I go about not using it?
>>
>>> Trouble now is that
>>> with shared IRQ we don't have a good way to handle that right now.
>>
>> Game over then?
>> I've tried assigning the USB devices before, I couldn't do it because
>> qemu doesn't support USB2 devices.
>> I don't really understand where this IRQ conflict is, the firewire and
>> the USB2 device share IRQ22 but I'm assigning them both to the VM?
>> Is that still a problem?
>> I don't suppose there's any way to change which IRQ they use in the
>> BIOS or with a command is there?
>>
>> I don't know if it means anything but this page:
>>
>> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
>>
>> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
>> My knowledge it very limited on this subject so I don't know if that's
>> meaningless looking at the output from another person's lspci.
>>
>> Anything left to try?
>>
>> Regardless, many thanks for your help,
>>
>> James.
>>
>
> On the off chance I tried disabling the firewire in the BIOS, which
> leaves only my tuner card using IRQ 20, 21 and 22.
> No difference, still complains about IRQs:
>
> Using raw in/out ioport access (sysfs - Input/output error)
> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?
>
> It does say "Operation not permitted" and that only "perhaps" I am
> assigning a device that shares an IRQ.
> Perhaps IRQ conflict it not the problem? They really are sitting on
> their own. Another permissions problem perhaps?
>
> Regards,
>
> James.
>

I'm reading something about this error message being related to
libvirt and CAP_SYS_RAWIO?

http://www.mail-archive.com/kvm@vger.kernel.org/msg34338.html
http://www.google.co.uk/#hl=en&xhr=t&q=libvirt+CAP_SYS_RAWIO&cp=21&pf=p&sclient=psy&aq=f&aqi=&aql=&oq=libvirt+CAP_SYS_RAWIO&pbx=1&fp=2d8e3f69fec095f4

"When I patch libvirt to not drop the capabilities, everything works
as expected."

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> On Fri, Feb 25, 2011 at 11:02 PM, James Neave <roboj1m@gmail.com> wrote:
> > On Fri, Feb 25, 2011 at 10:47 PM, James Neave <roboj1m@gmail.com> wrote:
> >> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> >>> * James Neave (roboj1m@gmail.com) wrote:
> >>>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
> >>>> http://pastebin.com/JxEwvqRA
> >>>
> >>> Yeah, that's what I expected:
> >>>
> >>> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
> >>> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
> >>>
> >>> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
> >>> should all go into same iommu domain).
> >>>
> >>>> I've just figured out a sequence of "echo DEV > PATH" commands to call
> >>>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
> >>>> the "failed to assign IRQ" error.
> >>>> I'm going to narrow down the required sequence and then post it.
> >>>
> >>> Kind of afraid to ask, but does it include:
> >>>
> >>> (assuming 1002 4384 is the pci to pci bridge)
> >>> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
> >>> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
> >>>
> >>> (this has the side effect of detaching the bridge from its domain)
> >>>
> >>> thanks,
> >>> -chris
> >>>
> >>
> >> Exact sequence is:
> >>
> >> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
> >> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
> >>
> >> I take it this is a bad thing then?
> >>
> >>> I assume this means that 00:14.4 is still left claimed by pci-stub?
> >>
> >> Yes
> >>
> >>> How are you determining this?  The lspci paste above has pci-stub for all
> >>> of them.  The easiest thing might be to start with manually disabling
> >>> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
> >>> Then giving the guest only 08:06.1.
> >>
> >> I determined it by being half asleep and not reading it properly... >.<
> >> You're right, all 5 devices were using pci-stub
> >>
> >>>> libvirtError: this function is not supported by the connection driver:
> >>>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
> >>>> available
> >>
> >>> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
> >>> libvirt here).
> >>
> >> What does that libvirt error mean? I can't find a definition.
> >> Am I limiting myself by using libvirt? Would not using it help and how
> >> would I go about not using it?
> >>
> >>> Trouble now is that
> >>> with shared IRQ we don't have a good way to handle that right now.
> >>
> >> Game over then?
> >> I've tried assigning the USB devices before, I couldn't do it because
> >> qemu doesn't support USB2 devices.
> >> I don't really understand where this IRQ conflict is, the firewire and
> >> the USB2 device share IRQ22 but I'm assigning them both to the VM?
> >> Is that still a problem?
> >> I don't suppose there's any way to change which IRQ they use in the
> >> BIOS or with a command is there?
> >>
> >> I don't know if it means anything but this page:
> >>
> >> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
> >>
> >> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
> >> My knowledge it very limited on this subject so I don't know if that's
> >> meaningless looking at the output from another person's lspci.
> >>
> >> Anything left to try?
> >>
> >> Regardless, many thanks for your help,
> >>
> >> James.
> >>
> >
> > On the off chance I tried disabling the firewire in the BIOS, which
> > leaves only my tuner card using IRQ 20, 21 and 22.
> > No difference, still complains about IRQs:
> >
> > Using raw in/out ioport access (sysfs - Input/output error)
> > Failed to assign irq for "hostdev0": Operation not permitted
> > Perhaps you are assigning a device that shares an IRQ with another device?
> >
> > It does say "Operation not permitted" and that only "perhaps" I am
> > assigning a device that shares an IRQ.
> > Perhaps IRQ conflict it not the problem? They really are sitting on
> > their own. Another permissions problem perhaps?
> >
> > Regards,
> >
> > James.
> >
> 
> I'm reading something about this error message being related to
> libvirt and CAP_SYS_RAWIO?

Depending on how new your libvirt is, you can force it to stop dropping
capabilities.  Look for the config item "clear_emulator_capabilities"
in /etc/libvirt/qemu.conf.  Setting this to 0 would verify that's the
problem (and not a real shared irq...i thought i saw sharing on
/proc/interrupts though).

> 
> http://www.mail-archive.com/kvm@vger.kernel.org/msg34338.html
> http://www.google.co.uk/#hl=en&xhr=t&q=libvirt+CAP_SYS_RAWIO&cp=21&pf=p&sclient=psy&aq=f&aqi=&aql=&oq=libvirt+CAP_SYS_RAWIO&pbx=1&fp=2d8e3f69fec095f4
> 
> "When I patch libvirt to not drop the capabilities, everything works
> as expected."

Well, that's a good point.  We fixed that a while ago, but I'm not sure
your kernel has that fix.

2.6.35.10-dmar (btw, random nitpick, dmar == intel dma remapping engine,
aka vt-d not amd iommu ;)

This was fixed in 2.6.36, commit:

48bb09e KVM: remove CAP_SYS_RAWIO requirement from kvm_vm_ioctl_assign_irq

The last 2.6.35 stable release is 2.6.35.9 and does not have that fix.
So unless your .10-dmar has it, you could patch it in.

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Fri, Feb 25, 2011 at 11:31 PM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> On Fri, Feb 25, 2011 at 11:02 PM, James Neave <roboj1m@gmail.com> wrote:
>> > On Fri, Feb 25, 2011 at 10:47 PM, James Neave <roboj1m@gmail.com> wrote:
>> >> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
>> >>> * James Neave (roboj1m@gmail.com) wrote:
>> >>>> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
>> >>>> http://pastebin.com/JxEwvqRA
>> >>>
>> >>> Yeah, that's what I expected:
>> >>>
>> >>> [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
>> >>> [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
>> >>>
>> >>> That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
>> >>> should all go into same iommu domain).
>> >>>
>> >>>> I've just figured out a sequence of "echo DEV > PATH" commands to call
>> >>>> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
>> >>>> the "failed to assign IRQ" error.
>> >>>> I'm going to narrow down the required sequence and then post it.
>> >>>
>> >>> Kind of afraid to ask, but does it include:
>> >>>
>> >>> (assuming 1002 4384 is the pci to pci bridge)
>> >>> echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
>> >>> echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
>> >>>
>> >>> (this has the side effect of detaching the bridge from its domain)
>> >>>
>> >>> thanks,
>> >>> -chris
>> >>>
>> >>
>> >> Exact sequence is:
>> >>
>> >> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
>> >>
>> >> I take it this is a bad thing then?
>> >>
>> >>> I assume this means that 00:14.4 is still left claimed by pci-stub?
>> >>
>> >> Yes
>> >>
>> >>> How are you determining this?  The lspci paste above has pci-stub for all
>> >>> of them.  The easiest thing might be to start with manually disabling
>> >>> host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
>> >>> Then giving the guest only 08:06.1.
>> >>
>> >> I determined it by being half asleep and not reading it properly... >.<
>> >> You're right, all 5 devices were using pci-stub
>> >>
>> >>>> libvirtError: this function is not supported by the connection driver:
>> >>>> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
>> >>>> available
>> >>
>> >>> Right, libvirt is more restrictive than qemu-kvm (forgot you were using
>> >>> libvirt here).
>> >>
>> >> What does that libvirt error mean? I can't find a definition.
>> >> Am I limiting myself by using libvirt? Would not using it help and how
>> >> would I go about not using it?
>> >>
>> >>> Trouble now is that
>> >>> with shared IRQ we don't have a good way to handle that right now.
>> >>
>> >> Game over then?
>> >> I've tried assigning the USB devices before, I couldn't do it because
>> >> qemu doesn't support USB2 devices.
>> >> I don't really understand where this IRQ conflict is, the firewire and
>> >> the USB2 device share IRQ22 but I'm assigning them both to the VM?
>> >> Is that still a problem?
>> >> I don't suppose there's any way to change which IRQ they use in the
>> >> BIOS or with a command is there?
>> >>
>> >> I don't know if it means anything but this page:
>> >>
>> >> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
>> >>
>> >> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
>> >> My knowledge it very limited on this subject so I don't know if that's
>> >> meaningless looking at the output from another person's lspci.
>> >>
>> >> Anything left to try?
>> >>
>> >> Regardless, many thanks for your help,
>> >>
>> >> James.
>> >>
>> >
>> > On the off chance I tried disabling the firewire in the BIOS, which
>> > leaves only my tuner card using IRQ 20, 21 and 22.
>> > No difference, still complains about IRQs:
>> >
>> > Using raw in/out ioport access (sysfs - Input/output error)
>> > Failed to assign irq for "hostdev0": Operation not permitted
>> > Perhaps you are assigning a device that shares an IRQ with another device?
>> >
>> > It does say "Operation not permitted" and that only "perhaps" I am
>> > assigning a device that shares an IRQ.
>> > Perhaps IRQ conflict it not the problem? They really are sitting on
>> > their own. Another permissions problem perhaps?
>> >
>> > Regards,
>> >
>> > James.
>> >
>>
>> I'm reading something about this error message being related to
>> libvirt and CAP_SYS_RAWIO?
>
> Depending on how new your libvirt is, you can force it to stop dropping
> capabilities.  Look for the config item "clear_emulator_capabilities"
> in /etc/libvirt/qemu.conf.  Setting this to 0 would verify that's the
> problem (and not a real shared irq...i thought i saw sharing on
> /proc/interrupts though).
>
>>
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg34338.html
>> http://www.google.co.uk/#hl=en&xhr=t&q=libvirt+CAP_SYS_RAWIO&cp=21&pf=p&sclient=psy&aq=f&aqi=&aql=&oq=libvirt+CAP_SYS_RAWIO&pbx=1&fp=2d8e3f69fec095f4
>>
>> "When I patch libvirt to not drop the capabilities, everything works
>> as expected."
>
> Well, that's a good point.  We fixed that a while ago, but I'm not sure
> your kernel has that fix.
>
> 2.6.35.10-dmar (btw, random nitpick, dmar == intel dma remapping engine,
> aka vt-d not amd iommu ;)
>
> This was fixed in 2.6.36, commit:
>
> 48bb09e KVM: remove CAP_SYS_RAWIO requirement from kvm_vm_ioctl_assign_irq
>
> The last 2.6.35 stable release is 2.6.35.9 and does not have that fix.
> So unless your .10-dmar has it, you could patch it in.
>
> thanks,
> -chris
>

HOLY CRAP IT WORKS!!!! 8@

...almost...

OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
as it turns out, the rawio problem)
My VM came up, both the tuners were there and after the firmware
install I was able to tune and watch the slowest TV in the world over
VNC.

Thank god for that, i was really starting to believe that slashing out
a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
collosal waste of money.
"Sigh of relief"

Well, thank you all so much for helping me to get to this point!

And yes, I did say "almost works"

Looks like I've run straight into Chris' ref counting problem when
shutting the guest down.
Some sort of critical error barf was on the servers' screen when I
shut down the guest, appeared to be very similar to Chris' example, in
amd_iommu.c

I'd post it but the server locks up after it's been shown and needed
resetting. No idea how I would post that bit of dmesg as it gets reset
after each boot.

Is there a solution for this at the moment or will I have to wait for
it to be patched?

Many Thanks,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> HOLY CRAP IT WORKS!!!! 8@


Hey, great! ;)

> ...almost...
> 
> OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
> as it turns out, the rawio problem)
> My VM came up, both the tuners were there and after the firmware
> install I was able to tune and watch the slowest TV in the world over
> VNC.
> 
> Thank god for that, i was really starting to believe that slashing out
> a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
> collosal waste of money.
> "Sigh of relief"
> 
> Well, thank you all so much for helping me to get to this point!
> 
> And yes, I did say "almost works"
> 
> Looks like I've run straight into Chris' ref counting problem when
> shutting the guest down.
> Some sort of critical error barf was on the servers' screen when I
> shut down the guest, appeared to be very similar to Chris' example, in
> amd_iommu.c
> 
> I'd post it but the server locks up after it's been shown and needed
> resetting. No idea how I would post that bit of dmesg as it gets reset
> after each boot.
> 
> Is there a solution for this at the moment or will I have to wait for
> it to be patched?

No solution at the moment.  Will keep you posted.

thanks,
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 28, 2011 at 3:31 PM, Chris Wright <chrisw@sous-sol.org> wrote:
> * James Neave (roboj1m@gmail.com) wrote:
>> HOLY CRAP IT WORKS!!!! 8@
>
>
> Hey, great! ;)
>
>> ...almost...
>>
>> OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
>> as it turns out, the rawio problem)
>> My VM came up, both the tuners were there and after the firmware
>> install I was able to tune and watch the slowest TV in the world over
>> VNC.
>>
>> Thank god for that, i was really starting to believe that slashing out
>> a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
>> collosal waste of money.
>> "Sigh of relief"
>>
>> Well, thank you all so much for helping me to get to this point!
>>
>> And yes, I did say "almost works"
>>
>> Looks like I've run straight into Chris' ref counting problem when
>> shutting the guest down.
>> Some sort of critical error barf was on the servers' screen when I
>> shut down the guest, appeared to be very similar to Chris' example, in
>> amd_iommu.c
>>
>> I'd post it but the server locks up after it's been shown and needed
>> resetting. No idea how I would post that bit of dmesg as it gets reset
>> after each boot.
>>
>> Is there a solution for this at the moment or will I have to wait for
>> it to be patched?
>
> No solution at the moment.  Will keep you posted.
>
> thanks,
> -chris
>

Ah, I see, this is the side effect of detaching it from the domain,
means it fails here:

BUG_ON(!dev_data->domain);

in __detach_device because now domain is null?

I see what you mean about ref counting.

I was thinking about just hacking in a "fake_detach" method that just
knocks one off the device count for the moment so I can at least
shutdown the VM and reboot the host cleanly without having to hit the
reset button.
Or maybe just comment out the BUG_ON lines (assuming this is what
causes it to bomb out)
At least until someone who knows what they are doing fixes it properly anyway!

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 28, 2011 at 8:25 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 28, 2011 at 3:31 PM, Chris Wright <chrisw@sous-sol.org> wrote:
>> * James Neave (roboj1m@gmail.com) wrote:
>>> HOLY CRAP IT WORKS!!!! 8@
>>
>>
>> Hey, great! ;)
>>
>>> ...almost...
>>>
>>> OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
>>> as it turns out, the rawio problem)
>>> My VM came up, both the tuners were there and after the firmware
>>> install I was able to tune and watch the slowest TV in the world over
>>> VNC.
>>>
>>> Thank god for that, i was really starting to believe that slashing out
>>> a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
>>> collosal waste of money.
>>> "Sigh of relief"
>>>
>>> Well, thank you all so much for helping me to get to this point!
>>>
>>> And yes, I did say "almost works"
>>>
>>> Looks like I've run straight into Chris' ref counting problem when
>>> shutting the guest down.
>>> Some sort of critical error barf was on the servers' screen when I
>>> shut down the guest, appeared to be very similar to Chris' example, in
>>> amd_iommu.c
>>>
>>> I'd post it but the server locks up after it's been shown and needed
>>> resetting. No idea how I would post that bit of dmesg as it gets reset
>>> after each boot.
>>>
>>> Is there a solution for this at the moment or will I have to wait for
>>> it to be patched?
>>
>> No solution at the moment.  Will keep you posted.
>>
>> thanks,
>> -chris
>>
>
> Ah, I see, this is the side effect of detaching it from the domain,
> means it fails here:
>
> BUG_ON(!dev_data->domain);
>
> in __detach_device because now domain is null?
>
> I see what you mean about ref counting.
>
> I was thinking about just hacking in a "fake_detach" method that just
> knocks one off the device count for the moment so I can at least
> shutdown the VM and reboot the host cleanly without having to hit the
> reset button.
> Or maybe just comment out the BUG_ON lines (assuming this is what
> causes it to bomb out)
> At least until someone who knows what they are doing fixes it properly anyway!
>
> James.
>

Heh, well, funnily enough random hacking didn't work.
Think I'll give up and wait for someone who knows what they're doing!

I tried replacing two calls to BUG_ON with if() blocks that just
skipped that method instead of forcibly crashing the kernel.
Sadly it still crashes in amd_iommu_domain_destroy somewhere and I
don't understand how to read the stack trace.
I was hoping it would fail just enough that I would get back to the
command line for a clean reboot.
But I'm C# not C, although I can read it a little.

I wonder if there is any way to get past the "pci-stub" error without
detaching the PCI-PCI bridge from it's domain?

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 28, 2011 at 3:31 PM, Chris Wright <chrisw@sous-sol.org> wrote:
>
> * James Neave (roboj1m@gmail.com) wrote:
> > HOLY CRAP IT WORKS!!!! 8@
>
>
> Hey, great! ;)
>
> > ...almost...
> >
> > OK, "clear_emulator_capabilities=0" solved the IRQ problem (which was,
> > as it turns out, the rawio problem)
> > My VM came up, both the tuners were there and after the firmware
> > install I was able to tune and watch the slowest TV in the world over
> > VNC.
> >
> > Thank god for that, i was really starting to believe that slashing out
> > a lot of cash on my 890FX board and the fancy DDR3 ram it needed was a
> > collosal waste of money.
> > "Sigh of relief"
> >
> > Well, thank you all so much for helping me to get to this point!
> >
> > And yes, I did say "almost works"
> >
> > Looks like I've run straight into Chris' ref counting problem when
> > shutting the guest down.
> > Some sort of critical error barf was on the servers' screen when I
> > shut down the guest, appeared to be very similar to Chris' example, in
> > amd_iommu.c
> >
> > I'd post it but the server locks up after it's been shown and needed
> > resetting. No idea how I would post that bit of dmesg as it gets reset
> > after each boot.
> >
> > Is there a solution for this at the moment or will I have to wait for
> > it to be patched?
>
> No solution at the moment.  Will keep you posted.
>
> thanks,
> -chris

Hello all,

Just wondering if there has been any movement on this problem of
detaching PCI devices behind PCI-PCI bridges?

(Resent as I forgot to switch to plain text)

Regards,

James.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> On Fri, Feb 25, 2011 at 12:06 AM, Chris Wright <chrisw@sous-sol.org> wrote:
> > * James Neave (roboj1m@gmail.com) wrote:
> >> OK, here's my latest dmesg with amd_iommu_dump and debug with no quiet
> >> http://pastebin.com/JxEwvqRA
> >
> > Yeah, that's what I expected:
> >
> > [    0.724403] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 08:00.0 flags: 00 devid_to: 00:14.4
> > [    0.724439] AMD-Vi:   DEV_RANGE_END           devid: 08:1f.7
> >
> > That basically says 08:00.0 - 08:1f.7 will show up as 00:14.4 (and
> > should all go into same iommu domain).
> >
> >> I've just figured out a sequence of "echo DEV > PATH" commands to call
> >> for 14.4 gets me past the "claimed by pci-stub" error and gets me to
> >> the "failed to assign IRQ" error.
> >> I'm going to narrow down the required sequence and then post it.
> >
> > Kind of afraid to ask, but does it include:
> >
> > (assuming 1002 4384 is the pci to pci bridge)
> > echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
> > echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
> >
> > (this has the side effect of detaching the bridge from its domain)
> 
> Exact sequence is:
> 
> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind

OK, same, since driver is a symlink to pci-stub.

> I take it this is a bad thing then?

It just means the amd iommu driver might be susceptible to a refcounting
issue.  Indeed, here's what I do that  assigning a device below the
PCI-PCI bridge, then shutdown the guest:

[  406.535873] ------------[ cut here ]------------
[  406.536864] kernel BUG at arch/x86/kernel/amd_iommu.c:2460!
[  406.536864] invalid opcode: 0000 [#1] SMP 
[  406.536864] last sysfs file: /sys/devices/pci0000:00/0000:00:14.4/0000:03:06.0/device
[  406.536864] CPU 0 
[  406.536864] Modules linked in: kvm_amd kvm e1000e bnx2
[  406.536864] 
[  406.536864] Pid: 4265, comm: qemu-system-x86 Not tainted 2.6.37-rc6+ #61 Toonie/Toonie
[  406.536864] RIP: 0010:[<ffffffff81025e53>]  [<ffffffff81025e53>] amd_iommu_domain_destroy+0x75/0x9d
[  406.536864] RSP: 0018:ffff88013507fb78  EFLAGS: 00010202
[  406.536864] RAX: ffff8801346ebeb8 RBX: ffff8801346ebeb8 RCX: 0000000000014f67
[  406.536864] RDX: 0000000000000202 RSI: 0000000000000202 RDI: ffffffff81a118a0
[  406.536864] RBP: ffff88013507fba8 R08: 0000000000000000 R09: ffff88007900f8e8
[  406.536864] R10: ffff88013507f8d8 R11: 0000000000000006 R12: ffff8801346ebea8
[  406.536864] R13: ffff8800783b73a8 R14: 0000000000000202 R15: ffff880135089570
[  406.536864] FS:  00007fe794db76e0(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[  406.536864] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  406.536864] CR2: 0000000000000000 CR3: 000000007c6fb000 CR4: 00000000000006f0
[  406.536864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  406.536864] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  406.536864] Process qemu-system-x86 (pid: 4265, threadinfo ffff88013507e000, task ffff88013496b090)
[  406.536864] Stack:
[  406.536864]  0000000000000009 ffff880135089570 ffff88007c734ca0 0000000000000001
[  406.536864]  ffff88007c74e3c8 0000000000000002 ffff88013507fbc8 ffffffff813013b7
[  406.536864]  0000000000000001 ffff880135089570 ffff88013507fbe8 ffffffffa003f
d81
[  406.536864] Call Trace:
[  406.536864]  [<ffffffff813013b7>] iommu_domain_free+0x16/0x22
[  406.536864]  [<ffffffffa003fd81>] kvm_iommu_unmap_guest+0x22/0x28 [kvm]
[  406.536864]  [<ffffffffa00440fd>] kvm_arch_destroy_vm+0x15/0x119 [kvm]
[  406.536864]  [<ffffffffa003af59>] kvm_put_kvm+0xde/0x103 [kvm]
[  406.536864]  [<ffffffffa003b64e>] kvm_vcpu_release+0x13/0x17 [kvm]
[  406.536864]  [<ffffffff810e893a>] fput+0x11b/0x1bc
[  406.536864]  [<ffffffff810e5db9>] filp_close+0x67/0x72
[  406.536864]  [<ffffffff81040505>] put_files_struct+0x70/0xc3
[  406.536864]  [<ffffffff8104058c>] exit_files+0x34/0x39
[  406.536864]  [<ffffffff810418ec>] do_exit+0x267/0x72e
[  406.536864]  [<ffffffff8104994c>] ? lock_timer_base+0x26/0x4a
[  406.536864]  [<ffffffff8104be04>] ? freezing+0xe/0x10
[  406.536864]  [<ffffffff81041e47>] sys_exit_group+0x0/0x16
[  406.536864]  [<ffffffff8104dfce>] get_signal_to_deliver+0x31c/0x33b
[  406.536864]  [<ffffffff81001fc6>] do_notify_resume+0x8b/0x6c3
[  406.536864]  [<ffffffff8104be41>] ? set_tsk_thread_flag+0xd/0xf
[  406.536864]  [<ffffffff8104e6b5>] ? sys_rt_sigtimedwait+0x18e/0x208
[  406.536864]  [<ffffffff810efe00>] ? path_put+0x1d/0x22
[  406.536864]  [<ffffffff81002c58>] int_signal+0x12/0x17
[  406.536864] Code: 00 00 00 4c 89 eb 4d 8b 6d 00 49 8d 44 24 10 48 39
c3 75 df 4c 89 f6 48 c7 c7 a0 18 a1 81 e8 fa b5 56 00 41 83 7c 24 64 00
74 04 <0f> 0b eb fe 4c 89 e7 e8 2c f5 ff ff 4c 89 e7 e8 9e e2 ff ff 49 
[  406.536864] RIP  [<ffffffff81025e53>] amd_iommu_domain_destroy+0x75/0x9d
[  406.536864]  RSP <ffff88013507fb78>
[  406.854138] ---[ end trace 13c9f9241c8b376b ]---
[  406.859182] Fixing recursive fault but reboot is needed!

> > I assume this means that 00:14.4 is still left claimed by pci-stub?
> 
> Yes
> 
> > How are you determining this?  The lspci paste above has pci-stub for all
> > of them.  The easiest thing might be to start with manually disabling
> > host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
> > Then giving the guest only 08:06.1.
> 
> I determined it by being half asleep and not reading it properly... >.<
> You're right, all 5 devices were using pci-stub

Heh, ok ;)

> >> libvirtError: this function is not supported by the connection driver:
> >> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
> >> available
> 
> > Right, libvirt is more restrictive than qemu-kvm (forgot you were using
> > libvirt here).
> 
> What does that libvirt error mean? I can't find a definition.

libvirt is being very strict and erring on the side of safety.  It's
looking at each device and verifying that there is some method for
resetting it.  Not resetting the device between users can lead to
information leakage or robustness issues (worst cases would be a guest
learning host or other guest secrets or a guest maliciously leaving the
device in an unusual state that caused next user -- host or another
guest -- to crash, etc).  It is not seeing the ability to do FLR
(Function Level Reset), PM (Power Management) reset (switch to Power
state off, then back on basically), and can't to a secondary bus reset
because the device 00:14.4 is on the root bus.

> Am I limiting myself by using libvirt? Would not using it help and how
> would I go about not using it?

I think if you can do just a single device, and ensure that it's not
sharing an IRQ with any other device, then 

> > Trouble now is that
> > with shared IRQ we don't have a good way to handle that right now.
> 
> Game over then?

Yeah, unless you can do just a single device (and don't need the other
devices in the host).

> I've tried assigning the USB devices before, I couldn't do it because
> qemu doesn't support USB2 devices.
> I don't really understand where this IRQ conflict is, the firewire and
> the USB2 device share IRQ22 but I'm assigning them both to the VM?
> Is that still a problem?

You'll see them on the host in /proc/interrupts.  The problem is that if
you have two devices sharing an interrupt and they are each driven by
different guest (or one guest and one host), you subject a system that
relied on cooperating drivers to properly ack the interrupt to abuse.

> I don't suppose there's any way to change which IRQ they use in the
> BIOS or with a command is there?
> 
> I don't know if it means anything but this page:
> 
> http://linuxtv.org/wiki/index.php/Hauppauge_WinTV-HVR-2200
> 
> Has the lspci output for the HVR-2200 which mentions MSI and IRQ255.
> My knowledge it very limited on this subject so I don't know if that's
> meaningless looking at the output from another person's lspci.

Yeah, that suggests the device will use MSI interrupts.

> Anything left to try?

Just the idea of reducing the problem to a single device to the guest
(and may have to do the sysfs black magic above).

thanks.
-chris

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* James Neave (roboj1m@gmail.com) wrote:
> libvirtError: this function is not supported by the connection driver:
> Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset
> available

Right, libvirt is more restrictive than qemu-kvm (forgot you were using
libvirt here).

> There is nothing written to test.log when you try to start the VM with
> 00:14.4 attached.
> 
> At this point libvirt goes screwy and I have to restart it before I
> can remove 00:14.4 from the VM.

I assume this means that 00:14.4 is still left claimed by pci-stub?

> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?
> kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6:

Believe it or not this is progress ;)  You have passed the point that
it was failing before (the iommu domain issue).  Trouble now is that
with shared IRQ we don't have a good way to handle that right now.

> Device 'pci-assign' could not be initialized


> 2011-02-23 19:21:13.958: shutting down
> 
> dmesg:
> http://pastebin.com/70D26xp4
> 
> This bit is different:
> 
> [  201.625221] uhci_hcd 0000:08:06.0: remove, state 4
> [  201.625237] usb usb4: USB disconnect, address 1
> [  201.625514] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
> [  201.625595] uhci_hcd 0000:08:06.0: PCI INT A disabled
> [  201.626028] pci-stub 0000:08:06.0: claimed by stub
> [  201.631922] uhci_hcd 0000:08:06.1: remove, state 4
> [  201.631937] usb usb9: USB disconnect, address 1
> [  201.632195] uhci_hcd 0000:08:06.1: USB bus 9 deregistered
> [  201.632274] uhci_hcd 0000:08:06.1: PCI INT B disabled
> [  201.632419] pci-stub 0000:08:06.1: claimed by stub
> [  201.638160] ehci_hcd 0000:08:06.2: remove, state 1
> [  201.638172] usb usb10: USB disconnect, address 1
> [  201.638178] usb 10-1: USB disconnect, address 2
> [  201.721626] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T successfully
> deinitialized and disconnected.
> [  201.721990] ehci_hcd 0000:08:06.2: USB bus 10 deregistered
> [  201.722126] ehci_hcd 0000:08:06.2: PCI INT C disabled
> [  201.725042] pci-stub 0000:08:06.2: claimed by stub
> [  201.731830] firewire_ohci 0000:08:0e.0: PCI INT A disabled
> [  201.731838] firewire_ohci: Removed fw-ohci device.
> [  201.732536] pci-stub 0000:08:0e.0: claimed by stub
> [  202.303880] device vnet0 entered promiscuous mode
> [  202.305184] virbr0: topology change detected, propagating
> [  202.305193] virbr0: port 1(vnet0) entering forwarding state
> [  202.305199] virbr0: port 1(vnet0) entering forwarding state
> [  202.433007] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  202.470076] pci-stub 0000:08:06.0: restoring config space at offset
> 0x1 (was 0x2100000, writing 0x2100001)
> [  202.697270] assign device 0:8:6.0
> [  202.697325] deassign device 0:8:6.0
> [  202.730080] pci-stub 0000:08:06.0: restoring config space at offset
> 0x1 (was 0x2100000, writing 0x2100001)
> [  202.730107] pci-stub 0000:08:06.0: PCI INT A disabled
> 
> This time the pci-stub claimed lines are not all bunched up and there
> is only one per device, rather than three per device.
> Also for the first time it says "assign device 0:8:6.0" rather than
> "assign device 0:8:6.0 failed"
> It them immediately deassigns the device and stops.
> 
> test.log shows:
> 
> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?
> 
> lspsci -vv for the relevant devices shows:
> http://pastebin.com/EUtUMj8x
> 
> 00:14.4 now appears to be using pci-stub as it's driver, as well as
> 08:06.1, 2, 3 but not 0e.0

How are you determining this?  The lspci paste above has pci-stub for all
of them.  The easiest thing might be to start with manually disabling
host driver and reassigning pci-stub to: 00:14.4, 08: 06.2,3 and 0e.0
Then giving the guest only 08:06.1.

> Anyway, that's all for now.

Thanks for testing.

> I think I'll try 'amd_iommu_dump' next, does it write to dmesg?

Yes it does.

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[Chris Wright]

* Alex Williamson (alex.williamson@redhat.com) wrote:
> I don't know why you're getting -EBUSY for this device, but maybe we can
> start from a clean slate and see if it helps.  Here's what I would
> suggest:

I bet this is an AMD IOMMU box.  Can we get full dmesg?

Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave]

On Mon, Feb 21, 2011 at 11:28 PM, Chris Wright <chrisw@sous-sol.org> wrote:
> * Alex Williamson (alex.williamson@redhat.com) wrote:
>> I don't know why you're getting -EBUSY for this device, but maybe we can
>> start from a clean slate and see if it helps.  Here's what I would
>> suggest:
>
> I bet this is an AMD IOMMU box.  Can we get full dmesg?
>

Hi!

Good guess, it is indeed an 890FX based board.

Here's the latest dmesg:
http://pastebin.com/0ZSP31uf

Lots and other dumps are available further up this thread.

fwiw, I've somehow managed to get it to do something different.
After passing though the soundcard, installing Ubuntu 10.10 on the VM
and trying some of the other 14.x PCI devices I managed to really
upset libvirt.
After restarting it on the host server, I added 08:06.0, 1 and 2 again
and got slightly further:

[ 5805.521230] firewire_ohci: Removed fw-ohci device.
[ 5812.092791] pci-stub 0000:08:06.0: claimed by stub
[ 5812.092861] pci-stub 0000:08:06.1: claimed by stub
[ 5812.093107] pci-stub 0000:08:06.0: claimed by stub
[ 5812.099889] pci-stub 0000:08:06.1: claimed by stub
[ 5812.102623] pci-stub 0000:08:06.2: claimed by stub
[ 5812.102723] pci-stub 0000:08:06.2: claimed by stub
[ 5812.265948] type=1400 audit(1298331554.277:41): apparmor="STATUS"
operation="profile_load"
name="libvirt-307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc" pid=3227
comm="apparmor_parser"
[ 5812.653784] device vnet0 entered promiscuous mode
[ 5812.655088] virbr0: topology change detected, propagating
[ 5812.655097] virbr0: port 1(vnet0) entering forwarding state
[ 5812.655103] virbr0: port 1(vnet0) entering forwarding state
[ 5812.781203] pci-stub 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
[ 5812.820087] pci-stub 0000:08:06.0: restoring config space at offset
0x1 (was 0x2100000, writing 0x2100001)
[ 5813.048427] assign device 0:8:6.0
[ 5813.048463] type=1400 audit(1298331555.057:42): apparmor="DENIED"
operation="capable" parent=1
profile="libvirt-307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc" pid=3236
comm="kvm" capability=17  capname="sys_rawio"
[ 5813.048505] deassign device 0:8:6.0
[ 5813.080089] pci-stub 0000:08:06.0: restoring config space at offset
0x1 (was 0x2100000, writing 0x2100001)
[ 5813.080116] pci-stub 0000:08:06.0: PCI INT A disabled
[ 5813.277511] type=1400 audit(1298331555.287:43): apparmor="STATUS"
operation="profile_remove"
name="libvirt-307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc" pid=3251
comm="apparmor_parser"
[ 5813.340516] uhci_hcd 0000:08:06.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
[ 5813.340534] uhci_hcd 0000:08:06.0: UHCI Host Controller
[ 5813.340660] uhci_hcd 0000:08:06.0: new USB bus registered, assigned
bus number 4
[ 5813.340708] uhci_hcd 0000:08:06.0: irq 20, io base 0x00007f00
[ 5813.340717] uhci_hcd 0000:08:06.0: unable to allocate consistent
memory for frame list
[ 5813.340858] uhci_hcd 0000:08:06.0: startup error -16
[ 5813.340950] uhci_hcd 0000:08:06.0: USB bus 4 deregistered
[ 5813.341044] uhci_hcd 0000:08:06.0: PCI INT A disabled
[ 5813.341049] uhci_hcd 0000:08:06.0: init 0000:08:06.0 fail, -16

'apparmor="DENIED"', I've fixed those before by adding to
/etc/apparmor/ profiles for libvirt.
I guess that means I have to add rw access for sys_rawio?

That's all for tonight, probably won't get any more time on this until
Wednesday 19:00 GMT.

Many Thanks,

James.