PCI hotplug related issues

PCI hotplug related issues

[Anil Rao]

[-- Attachment #1.1: Type: text/plain, Size: 2965 bytes --]

Hello,

I am encountering an interesting problem when using PCI-hotplug to assign devices to a VM (HVM) in a Xen environment (SLES 11 SP2).

1. If the VM is powered on before the PCI device is brought on-line, the guest OS is unable to see the device (I have tried using
    the acpiphp/pciehp drivers in the guest OS, as well as manually rescanning the PCI bus from within the VM).

    However, Xen reports that the hot-plug operation was successful. When I try to verify the PCI device state from Xen's perspective
    by looking  at a) the list of (remaining) assignable devices and b) list of devices attached to the VM, the xm and xl tool-sets report
    that the PCI device is indeed assigned to the VM.

2. If the VM is powered on after the PCI device (an FPGA in this case) is brought on-line, the guest OS inside the VM can see and
    configure the device as expected (using the appropriate driver for the device).

Upon examining the output of /var/log/xen/qemu-dm-VMName.log, I see that in the former case QEMU returned a failure, while in the latter case it reported success (see below).


Case 1: VM was powered on 'before' PCI device (FPGA) was brought on-line
-------------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Error: couldn't locate device in libpci structures


Case 2: VM was powered on 'after' PCI device (FPGA) was brought on-line
----------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6000000)
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6100000)
pt_msi_setup: msi mapped with pirq 84 pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!

I have seen this behavior when running both WindRiver and Ubuntu based operating environments inside the VM. It doesn't seem related to the type of guest OS.

I have two questions, that I hoped someone on this list can answer for me.

A) Does Xen support case 1 (hotplugging a PCI device to a running VM that was powered-on before the device was added to
    the host)?

B) Why is Xen reporting that the hotplug operation was successful and that the device is assigned to the VM, when QEMU reported
    an error? Shouldn't the Xen hotplug operation have failed as well?

Any help on this matter will be most appreciated.

Thanks.
-Anil


[-- Attachment #1.2: Type: text/html, Size: 4111 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PCI hotplug related issues

[Jan Beulich]

>>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote:
> A) Does Xen support case 1 (hotplugging a PCI device to a running VM that 
> was powered-on before the device was added to
>     the host)?

As far as I'm aware, only hotplug from the perspective of the guest
is really supported/tested.

But the issue here really seems less Xen related, but more towards
libpci behavior (as according to qemu this is where the device isn't
known). As that's a component Xen/qemu-dm simply builds upon,
failure there (e.g. lack of hotplug awareness) would surface as a
Xen misbehavior. Quickly going through the list of provided symbols
of libpci.so doesn't show anything that looks hotplug related (e.g.
some sort of notification registration). But of course it's possible
that I'm overlooking something here, and that in fact qemu-dm's
interaction with libpci is simply incomplete.

Jan

Re: PCI hotplug related issues

[Anil Rao]

Thanks for the reply. 

After encountering this problem, I was afraid that only VM-level hotplug is supported by Xen (and not true hotplug at the host-level). Can someone kindly confirm that this is the case? Our current effort depends on host-level hotplug of a PCI device that can (later) be assigned (hotplugged) to a running VM, so a definitive answer will help us to avoid going down the wrong path.

At the same time, I still think its odd that the Xen xm pci-attach command returned success when the attempted hotplug operation actually failed (as reported by the error message printed in the qemu-dm log file). Also, as I have indicated in my earlier post, Xen actually believes that the PCI device is assigned to the target VM (verified this by using xm pci-list to find out the list of devices attached to a VM), which again is very misleading. 

-Anil



-----Original Message-----
From: Jan Beulich [mailto:JBeulich@suse.com] 
Sent: Tuesday, June 05, 2012 1:43 AM
To: Anil Rao
Cc: xen-devel
Subject: Re: [Xen-devel] PCI hotplug related issues

>>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote:
> A) Does Xen support case 1 (hotplugging a PCI device to a running VM 
> that was powered-on before the device was added to
>     the host)?

As far as I'm aware, only hotplug from the perspective of the guest is really supported/tested.

But the issue here really seems less Xen related, but more towards libpci behavior (as according to qemu this is where the device isn't known). As that's a component Xen/qemu-dm simply builds upon, failure there (e.g. lack of hotplug awareness) would surface as a Xen misbehavior. Quickly going through the list of provided symbols of libpci.so doesn't show anything that looks hotplug related (e.g.
some sort of notification registration). But of course it's possible that I'm overlooking something here, and that in fact qemu-dm's interaction with libpci is simply incomplete.

Jan

Re: PCI hotplug related issues

[Jan Beulich]

>>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote:
> After encountering this problem, I was afraid that only VM-level hotplug is 
> supported by Xen (and not true hotplug at the host-level). Can someone kindly 
> confirm that this is the case? Our current effort depends on host-level 
> hotplug of a PCI device that can (later) be assigned (hotplugged) to a 
> running VM, so a definitive answer will help us to avoid going down the wrong 
> path.

That is not a proper statement - host side holtplug is still supported,
just that currently it may need to happen before the target VM gets
created. Overcoming limitations like this is certainly desirable, but will
obviously require someone to actually invest work - the question
would hence be whether, given that you're interested in this feature
and posted on xen-devel, you wouldn't also want to look into
addressing this (provided my brief analysis of the situation is correct
at all).

Jan

Re: PCI hotplug related issues

[Ian Campbell]

On Tue, 2012-06-05 at 09:42 +0100, Jan Beulich wrote:
> >>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote:
> > A) Does Xen support case 1 (hotplugging a PCI device to a running VM that 
> > was powered-on before the device was added to
> >     the host)?
> 
> As far as I'm aware, only hotplug from the perspective of the guest
> is really supported/tested.
> 
> But the issue here really seems less Xen related, but more towards
> libpci behavior (as according to qemu this is where the device isn't
> known). As that's a component Xen/qemu-dm simply builds upon,
> failure there (e.g. lack of hotplug awareness) would surface as a
> Xen misbehavior. Quickly going through the list of provided symbols
> of libpci.so doesn't show anything that looks hotplug related (e.g.
> some sort of notification registration). But of course it's possible
> that I'm overlooking something here, and that in fact qemu-dm's
> interaction with libpci is simply incomplete.

Your theory tallies with what I remember from looking at PCI hotplug
using a stub qemu -- IIRC libpci scans the bus at start of day and has
no capability to refresh the list of known devices after that. This
means that on host hotplug (actually mini-os hotplug in the case I was
looking at) the device doesn't become known to qemu.

Anyway the upshot is that AFAIK even for non-stub qemu either (or both)
qemu and libpci need to be taught about host PCI hotplug somehow.

Ian.

Re: PCI hotplug related issues

[Ian Campbell]

On Wed, 2012-06-06 at 08:36 +0100, Jan Beulich wrote:
> >>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote:
> > After encountering this problem, I was afraid that only VM-level hotplug is 
> > supported by Xen (and not true hotplug at the host-level). Can someone kindly 
> > confirm that this is the case? Our current effort depends on host-level 
> > hotplug of a PCI device that can (later) be assigned (hotplugged) to a 
> > running VM, so a definitive answer will help us to avoid going down the wrong 
> > path.
> 
> That is not a proper statement - host side holtplug is still supported,
> just that currently it may need to happen before the target VM gets
> created. Overcoming limitations like this is certainly desirable, but will
> obviously require someone to actually invest work - the question
> would hence be whether, given that you're interested in this feature
> and posted on xen-devel, you wouldn't also want to look into
> addressing this (provided my brief analysis of the situation is correct
> at all).

Yes, patches to make this work would be gratefully received.

As an aside, I wonder if upstream qemu does anything better here -- that
would probably be the right place to look at adding this new feature in
any case.

Ian.

Re: PCI hotplug related issues

[Anil Rao]

 
Thanks, Jan and Ian for your confirmation and suggestions. For the moment, I have incorporated a workaround that is allowing us to proceed with our particular usage of Xen based VMs. 

When I free up a little bit at work, I don't mind taking a stab at getting this to work. I'll keep the list posted.

Anil


-----Original Message-----
From: Ian Campbell [mailto:Ian.Campbell@citrix.com] 
Sent: Wednesday, June 06, 2012 2:29 AM
To: Jan Beulich
Cc: Anil Rao; xen-devel
Subject: Re: [Xen-devel] PCI hotplug related issues

On Wed, 2012-06-06 at 08:36 +0100, Jan Beulich wrote:
> >>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote:
> > After encountering this problem, I was afraid that only VM-level 
> > hotplug is supported by Xen (and not true hotplug at the 
> > host-level). Can someone kindly confirm that this is the case? Our 
> > current effort depends on host-level hotplug of a PCI device that 
> > can (later) be assigned (hotplugged) to a running VM, so a 
> > definitive answer will help us to avoid going down the wrong path.
> 
> That is not a proper statement - host side holtplug is still 
> supported, just that currently it may need to happen before the target 
> VM gets created. Overcoming limitations like this is certainly 
> desirable, but will obviously require someone to actually invest work 
> - the question would hence be whether, given that you're interested in 
> this feature and posted on xen-devel, you wouldn't also want to look 
> into addressing this (provided my brief analysis of the situation is 
> correct at all).

Yes, patches to make this work would be gratefully received.

As an aside, I wonder if upstream qemu does anything better here -- that would probably be the right place to look at adding this new feature in any case.

Ian.