Xen interrupts when using passthrough

Xen interrupts when using passthrough

[Eli Cohen]

Hi,
I am trying to debug a problem where a guest does not receive the
correct interrupt. It's a passtrough system where the guest is given a
virtual function and is using MSIX to recieve the interrupts. I am
trying to understand how is the guest supposed to recieve the
interrupt: Is it handled by dom0 which is using event channels to pass
the interrupt to domU, or is handled directly by domU. Please send
pointers to the code.

Thanks in advance,
eli;

Re: Xen interrupts when using passthrough

[Andrew Cooper]

On 04/10/11 17:29, Eli Cohen wrote:
> Hi,
> I am trying to debug a problem where a guest does not receive the
> correct interrupt. It's a passtrough system where the guest is given a
> virtual function and is using MSIX to recieve the interrupts.

Do you mean that it is receiving interrupts for another device, or are
you trying to pass through multiple interrupts for a device and the
interrupt is appearing in the wrong event channel?

Could you describe your hardware setup in detail please, especially
AMD/Intel, IOMMU features in use according to Xen, and the PCI topology
of the box.

>  I am
> trying to understand how is the guest supposed to recieve the
> interrupt: Is it handled by dom0 which is using event channels to pass
> the interrupt to domU, or is handled directly by domU. Please send
> pointers to the code.

(I am not very familiar with guest IRQs so the following might not be
accurate)

For a PCI passthrough system, the interrupts should be bound directly to
domU, with no dom0 interaction.  A PIRQ is bound to an event channel
using the EVTCHNOP_bind_pirq in common/event_channel.c, which should be
set up by qemu for the relevant domU, when it is interpreting the PCI
config space reads/writes.

> Thanks in advance,
> eli;
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: Xen interrupts when using passthrough

[Eli Cohen]

Andrew, thanks for your response.

On Tue, Oct 04, 2011 at 05:50:17PM +0100, Andrew Cooper wrote:
> 
> Do you mean that it is receiving interrupts for another device, or are
> you trying to pass through multiple interrupts for a device and the
> interrupt is appearing in the wrong event channel?
> 
I am not sure I understand your question.
> Could you describe your hardware setup in detail please, especially
> AMD/Intel, IOMMU features in use according to Xen, and the PCI topology
> of the box.

Here's a description. I have a PCI device which supports SRIOV, so
there are several virtual functions besides the primary function. The
device generates interrupts using MSI-X. I bind one of the virtual
functions to the guest. The PCI device driver at the guest requests
msix vectors from kernel and gets two vectors. Then it calls
request_irq for those vectors.
When the device generates msix interrupt using the address/data in
it's msi-x table. However, the guest kernel complains the no handler
is registered for this vector. Specifically, I request_irq for vectors
48 and 49. When the device issues interrupt for vector 49, the guest
complains that it received interrupt for vector 89 which has no
handler routine.
> 
> >  I am
> > trying to understand how is the guest supposed to recieve the
> > interrupt: Is it handled by dom0 which is using event channels to pass
> > the interrupt to domU, or is handled directly by domU. Please send
> > pointers to the code.
> 
> (I am not very familiar with guest IRQs so the following might not be
> accurate)
> 
> For a PCI passthrough system, the interrupts should be bound directly to
> domU, with no dom0 interaction.  A PIRQ is bound to an event channel
> using the EVTCHNOP_bind_pirq in common/event_channel.c, which should be
> set up by qemu for the relevant domU, when it is interpreting the PCI
> config space reads/writes.
My geust kernel is 2.6.32 based and I can't see any file with the name
event_channel.c.

Re: Xen interrupts when using passthrough

[Jan Beulich]

>>> On 04.10.11 at 21:00, Eli Cohen <eli@dev.mellanox.co.il> wrote:
> Here's a description. I have a PCI device which supports SRIOV, so
> there are several virtual functions besides the primary function. The
> device generates interrupts using MSI-X. I bind one of the virtual
> functions to the guest. The PCI device driver at the guest requests
> msix vectors from kernel and gets two vectors. Then it calls
> request_irq for those vectors.
> When the device generates msix interrupt using the address/data in
> it's msi-x table. However, the guest kernel complains the no handler
> is registered for this vector. Specifically, I request_irq for vectors
> 48 and 49. When the device issues interrupt for vector 49, the guest
> complains that it received interrupt for vector 89 which has no
> handler routine.

Quite possibly a mixup of Xen and Linux IRQ numbers. You may want
to check Xen's 'i' and 'e' debug key output for whether this might be
the case.

> My geust kernel is 2.6.32 based

Upstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or
yet something else?

> and I can't see any file with the name
> event_channel.c.

That's a hypervisor source file, not a Linux one.

Jan

Re: Xen interrupts when using passthrough

[Eli Cohen]

On Wed, Oct 05, 2011 at 08:27:07AM +0100, Jan Beulich wrote:
> 
> Quite possibly a mixup of Xen and Linux IRQ numbers. You may want
> to check Xen's 'i' and 'e' debug key output for whether this might be
> the case.

Can you tell how this is done - sorry, I'm not a xen expert.

> 
> > My geust kernel is 2.6.32 based
> 
> Upstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or
> yet something else?
> 
> > and I can't see any file with the name
> > event_channel.c.
> 
> That's a hypervisor source file, not a Linux one.
> 

So how do you think interrupts should reach the guest. Is it through
event channels (i.e dom0 receives them first) or do they go directly
to the guest with dom0 unaware that there was an interrupt?

Re: Xen interrupts when using passthrough

[Jan Beulich]

>>> On 05.10.11 at 10:26, Eli Cohen <eli@dev.mellanox.co.il> wrote:
> On Wed, Oct 05, 2011 at 08:27:07AM +0100, Jan Beulich wrote:
>> 
>> Quite possibly a mixup of Xen and Linux IRQ numbers. You may want
>> to check Xen's 'i' and 'e' debug key output for whether this might be
>> the case.
> 
> Can you tell how this is done - sorry, I'm not a xen expert.

Either by sending the characters from the serial console (after switching
input to Xen), or by using "{xl,xm] debug-key" followed by
"{xm,xl} dmesg" to view the output.

>> 
>> > My geust kernel is 2.6.32 based
>> 
>> Upstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or
>> yet something else?
>> 
>> > and I can't see any file with the name
>> > event_channel.c.
>> 
>> That's a hypervisor source file, not a Linux one.
>> 
> 
> So how do you think interrupts should reach the guest. Is it through
> event channels (i.e dom0 receives them first) or do they go directly
> to the guest with dom0 unaware that there was an interrupt?

Dom0 shouldn't get involved at all - Xen does the necessary conversion.
Dom0 may only be involved in setting things up. Whether they get
received via event channels or virtual interrupts depends on the kind
of guest.

Jan

Re: Xen interrupts when using passthrough

[Eli Cohen]

On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote:
> 
> Either by sending the characters from the serial console (after switching
> input to Xen), or by using "{xl,xm] debug-key" followed by
> "{xm,xl} dmesg" to view the output.
> 
Thanks.
> 
> Dom0 shouldn't get involved at all - Xen does the necessary conversion.
> Dom0 may only be involved in setting things up. Whether they get
> received via event channels or virtual interrupts depends on the kind
> of guest.
> 
I think I used the wrong terminology not distinguishing between the
hypervisor and dom0 so I will ask again just to make sure. Is the
hypervisor involved in passing the interrupts? Does it depend on
whether the guest is paravirtualized or not? The guest I am using is
paravirtualized (i.e uses xen hypercalls), so does it mean that it
uses event channels or is there a possibilty that it does not use
event channels.

Re: Xen interrupts when using passthrough

[Jan Beulich]

>>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote:
> On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote:
>> Dom0 shouldn't get involved at all - Xen does the necessary conversion.
>> Dom0 may only be involved in setting things up. Whether they get
>> received via event channels or virtual interrupts depends on the kind
>> of guest.
>> 
> I think I used the wrong terminology not distinguishing between the
> hypervisor and dom0 so I will ask again just to make sure. Is the
> hypervisor involved in passing the interrupts?

Yes, of course. The hypervisor receives all physical interrupts.

> Does it depend on whether the guest is paravirtualized or not?

Yes, at least to some degree.

> The guest I am using is
> paravirtualized (i.e uses xen hypercalls), so does it mean that it
> uses event channels or is there a possibilty that it does not use
> event channels.

Yes and no: PV guests can receive interrupts only via the event
channel mechanism.

Jan

Re: Xen interrupts when using passthrough

[Eli Cohen]

Thanks for the information.

On Wed, Oct 05, 2011 at 10:09:01AM +0100, Jan Beulich wrote:
> >>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote:
> > On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote:
> >> Dom0 shouldn't get involved at all - Xen does the necessary conversion.
> >> Dom0 may only be involved in setting things up. Whether they get
> >> received via event channels or virtual interrupts depends on the kind
> >> of guest.
> >> 
> > I think I used the wrong terminology not distinguishing between the
> > hypervisor and dom0 so I will ask again just to make sure. Is the
> > hypervisor involved in passing the interrupts?
> 
> Yes, of course. The hypervisor receives all physical interrupts.
> 
> > Does it depend on whether the guest is paravirtualized or not?
> 
> Yes, at least to some degree.
> 
> > The guest I am using is
> > paravirtualized (i.e uses xen hypercalls), so does it mean that it
> > uses event channels or is there a possibilty that it does not use
> > event channels.
> 
> Yes and no: PV guests can receive interrupts only via the event
> channel mechanism.
> 
> Jan

Re: Xen interrupts when using passthrough

[Ian Campbell]

On Wed, 2011-10-05 at 10:09 +0100, Jan Beulich wrote:
> >>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote:
> > On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote:
> >> Dom0 shouldn't get involved at all - Xen does the necessary conversion.
> >> Dom0 may only be involved in setting things up. Whether they get
> >> received via event channels or virtual interrupts depends on the kind
> >> of guest.
> >> 
> > I think I used the wrong terminology not distinguishing between the
> > hypervisor and dom0 so I will ask again just to make sure. Is the
> > hypervisor involved in passing the interrupts?
> 
> Yes, of course. The hypervisor receives all physical interrupts.
> 
> > Does it depend on whether the guest is paravirtualized or not?
> 
> Yes, at least to some degree.
> 
> > The guest I am using is
> > paravirtualized (i.e uses xen hypercalls), so does it mean that it
> > uses event channels or is there a possibilty that it does not use
> > event channels.
> 
> Yes and no: PV guests can receive interrupts only via the event
> channel mechanism.

While an HVM guest can receive interrupts via any of:
      * Emulated interrupt injection (i.e. from the emulated APIC)
      * Event channels, injected via the INTx interrupt on the PCI
        platform device (itself injected by the previous option)
      * Event channels, injected directly to a vector.

The first is available to any HVM guest, the middle one is the older
style PV Driver for HVM guest mechanism while the third is an additional
feature of more recent Linux kernels using the PVHVM mechanism.

Ian.