Purpose of translate MSI interrupt into INTx for guest passthrough

Purpose of translate MSI interrupt into INTx for guest passthrough

[Roger Pau Monné]

Hello,

While trying to do some cleanup of the Xen interrupt support for pci
pass though I came across the MSI to INTx translation that Xen is in
theory capable of performing (ie: use a physical MSI interrupt source
and inject that as an INTx to a guest).

AFAICT such functionality is not wired up to the toolstack, so it's
hard to tell what's the indented purpose, or whether it has seen any
usage.

The commit that introduces it seems to be:

11f8ac5ebf passthrough: MSI-INTx translation for HVM

Which contains your SoB. Could you elaborate a bit on what was the
intended purpose, and whether this has never been used?

Thanks, Roger.

Re: Purpose of translate MSI interrupt into INTx for guest passthrough

[Roger Pau Monné]

Dropping Qing He as this address bounces.

On Tue, Jan 12, 2021 at 03:10:57PM +0100, Roger Pau Monné wrote:
> Hello,
> 
> While trying to do some cleanup of the Xen interrupt support for pci
> pass though I came across the MSI to INTx translation that Xen is in
> theory capable of performing (ie: use a physical MSI interrupt source
> and inject that as an INTx to a guest).
> 
> AFAICT such functionality is not wired up to the toolstack, so it's
> hard to tell what's the indented purpose, or whether it has seen any
> usage.

So apparently it is wired up to the toolstack for qemu-traditional,
albeit it's disabled by default. There's some documentation in
xl-pci-configuration:

"When enabled, MSI-INTx translation will always enable MSI on the PCI
device regardless of whether the guest uses INTx or MSI."

So the main purpose seem to be to always use the MSI interrupt source
regardless of whether the guest is using INTx or MSI. Maybe the
purpose was to workaround some bugs when using INTx? Or buggy devices
with INTx interrupts?

qemu-upstream doesn't seem to support it anymore, so I would still
like to remove it if we get consensus.

Roger

Re: Purpose of translate MSI interrupt into INTx for guest passthrough

[Jason Andryuk]

On Tue, Jan 12, 2021 at 9:25 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> Dropping Qing He as this address bounces.
>
> On Tue, Jan 12, 2021 at 03:10:57PM +0100, Roger Pau Monné wrote:
> > Hello,
> >
> > While trying to do some cleanup of the Xen interrupt support for pci
> > pass though I came across the MSI to INTx translation that Xen is in
> > theory capable of performing (ie: use a physical MSI interrupt source
> > and inject that as an INTx to a guest).
> >
> > AFAICT such functionality is not wired up to the toolstack, so it's
> > hard to tell what's the indented purpose, or whether it has seen any
> > usage.
>
> So apparently it is wired up to the toolstack for qemu-traditional,
> albeit it's disabled by default. There's some documentation in
> xl-pci-configuration:
>
> "When enabled, MSI-INTx translation will always enable MSI on the PCI
> device regardless of whether the guest uses INTx or MSI."
>
> So the main purpose seem to be to always use the MSI interrupt source
> regardless of whether the guest is using INTx or MSI. Maybe the
> purpose was to workaround some bugs when using INTx? Or buggy devices
> with INTx interrupts?
>
> qemu-upstream doesn't seem to support it anymore, so I would still
> like to remove it if we get consensus.

The cover letter from
http://old-list-archives.xenproject.org/archives/html/xen-devel/2009-01/msg00228.html
"""
This patchset enables MSI-INTx interrupt translation for HVM
domains. The intention of the patch is to use MSI as the physical
interrupt mechanism for passthrough device as much as possible,
thus reducing the pirq sharing among domains.

When MSI is globally enabled, if the device has the MSI capability
but doesn't used by the guest, hypervisor will try to user MSI as
the underlying pirq and inject translated INTx irq to guest
vioapic. When guest itself enabled MSI or MSI-X, the translation
is automatically turned off.

Add a config file option to disable/enable this feature. Also, in
order to allow the user to override the option per device, a
per-device option mechanism is implemented and an MSI-INTx option
is added
"""

It seems like it could be a good idea, but I don't know if it presents
compatibility issues when actually used.

As you say, it's not supported by qemu-upstream, so maybe it should
just be dropped.

Regards,
Jason

Re: Purpose of translate MSI interrupt into INTx for guest passthrough

[Roger Pau Monné]

On Tue, Jan 12, 2021 at 09:48:17AM -0500, Jason Andryuk wrote:
> On Tue, Jan 12, 2021 at 9:25 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > Dropping Qing He as this address bounces.
> >
> > On Tue, Jan 12, 2021 at 03:10:57PM +0100, Roger Pau Monné wrote:
> > > Hello,
> > >
> > > While trying to do some cleanup of the Xen interrupt support for pci
> > > pass though I came across the MSI to INTx translation that Xen is in
> > > theory capable of performing (ie: use a physical MSI interrupt source
> > > and inject that as an INTx to a guest).
> > >
> > > AFAICT such functionality is not wired up to the toolstack, so it's
> > > hard to tell what's the indented purpose, or whether it has seen any
> > > usage.
> >
> > So apparently it is wired up to the toolstack for qemu-traditional,
> > albeit it's disabled by default. There's some documentation in
> > xl-pci-configuration:
> >
> > "When enabled, MSI-INTx translation will always enable MSI on the PCI
> > device regardless of whether the guest uses INTx or MSI."
> >
> > So the main purpose seem to be to always use the MSI interrupt source
> > regardless of whether the guest is using INTx or MSI. Maybe the
> > purpose was to workaround some bugs when using INTx? Or buggy devices
> > with INTx interrupts?
> >
> > qemu-upstream doesn't seem to support it anymore, so I would still
> > like to remove it if we get consensus.
> 
> The cover letter from
> http://old-list-archives.xenproject.org/archives/html/xen-devel/2009-01/msg00228.html
> """
> This patchset enables MSI-INTx interrupt translation for HVM
> domains. The intention of the patch is to use MSI as the physical
> interrupt mechanism for passthrough device as much as possible,
> thus reducing the pirq sharing among domains.
> 
> When MSI is globally enabled, if the device has the MSI capability
> but doesn't used by the guest, hypervisor will try to user MSI as
> the underlying pirq and inject translated INTx irq to guest
> vioapic. When guest itself enabled MSI or MSI-X, the translation
> is automatically turned off.
> 
> Add a config file option to disable/enable this feature. Also, in
> order to allow the user to override the option per device, a
> per-device option mechanism is implemented and an MSI-INTx option
> is added
> """
> 
> It seems like it could be a good idea, but I don't know if it presents
> compatibility issues when actually used.

Hm, MSI interrupts are edge triggered, while INTx is (usually) level.
Also devices capable of multiple MSI vectors will be limited to a
single one, and I'm not sure whether the transition from translated
MSI to INTx into multiple MSIs would work correctly, as seems tricky.

> As you say, it's not supported by qemu-upstream, so maybe it should
> just be dropped.

I don't really see much value in forcing Xen to always use MSI
regardless of whether the guest is using INTx or MSI, and it's likely
to cause more issues than benefits.

IMO I think we should get rid of this, as the only real value is
having Xen using MSI intend of INTx, but it's not introducing any kind
of functionality from a guest PoV.

Thanks, Roger.

Re: Purpose of translate MSI interrupt into INTx for guest passthrough

[Andrew Cooper]

On 12/01/2021 15:51, Roger Pau Monné wrote:
> On Tue, Jan 12, 2021 at 09:48:17AM -0500, Jason Andryuk wrote:
>> On Tue, Jan 12, 2021 at 9:25 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> Dropping Qing He as this address bounces.
>>>
>>> On Tue, Jan 12, 2021 at 03:10:57PM +0100, Roger Pau Monné wrote:
>>>> Hello,
>>>>
>>>> While trying to do some cleanup of the Xen interrupt support for pci
>>>> pass though I came across the MSI to INTx translation that Xen is in
>>>> theory capable of performing (ie: use a physical MSI interrupt source
>>>> and inject that as an INTx to a guest).
>>>>
>>>> AFAICT such functionality is not wired up to the toolstack, so it's
>>>> hard to tell what's the indented purpose, or whether it has seen any
>>>> usage.
>>> So apparently it is wired up to the toolstack for qemu-traditional,
>>> albeit it's disabled by default. There's some documentation in
>>> xl-pci-configuration:
>>>
>>> "When enabled, MSI-INTx translation will always enable MSI on the PCI
>>> device regardless of whether the guest uses INTx or MSI."
>>>
>>> So the main purpose seem to be to always use the MSI interrupt source
>>> regardless of whether the guest is using INTx or MSI. Maybe the
>>> purpose was to workaround some bugs when using INTx? Or buggy devices
>>> with INTx interrupts?
>>>
>>> qemu-upstream doesn't seem to support it anymore, so I would still
>>> like to remove it if we get consensus.
>> The cover letter from
>> http://old-list-archives.xenproject.org/archives/html/xen-devel/2009-01/msg00228.html
>> """
>> This patchset enables MSI-INTx interrupt translation for HVM
>> domains. The intention of the patch is to use MSI as the physical
>> interrupt mechanism for passthrough device as much as possible,
>> thus reducing the pirq sharing among domains.
>>
>> When MSI is globally enabled, if the device has the MSI capability
>> but doesn't used by the guest, hypervisor will try to user MSI as
>> the underlying pirq and inject translated INTx irq to guest
>> vioapic. When guest itself enabled MSI or MSI-X, the translation
>> is automatically turned off.
>>
>> Add a config file option to disable/enable this feature. Also, in
>> order to allow the user to override the option per device, a
>> per-device option mechanism is implemented and an MSI-INTx option
>> is added
>> """
>>
>> It seems like it could be a good idea, but I don't know if it presents
>> compatibility issues when actually used.
> Hm, MSI interrupts are edge triggered, while INTx is (usually) level.
> Also devices capable of multiple MSI vectors will be limited to a
> single one, and I'm not sure whether the transition from translated
> MSI to INTx into multiple MSIs would work correctly, as seems tricky.
>
>> As you say, it's not supported by qemu-upstream, so maybe it should
>> just be dropped.
> I don't really see much value in forcing Xen to always use MSI
> regardless of whether the guest is using INTx or MSI, and it's likely
> to cause more issues than benefits.
>
> IMO I think we should get rid of this, as the only real value is
> having Xen using MSI intend of INTx, but it's not introducing any kind
> of functionality from a guest PoV.

I find this feature very dubious.

While I agree that reducing INTx sharing between domains is obviously a
good thing, I don't see how the device can possibly be expected to work
if the in-guest driver doesn't have an accurate idea of what's going on.

There are up to 4 INTx lines, but absolutely nothing to suggest that
these would logically map to the first 4 MSIs enabled on the device. 
Even in the simplified case of only INTA and a single MSI, there's
nothing to suggest that the device will behave in the same way when it
comes to generating interrupts.

The difference between edge and line interrupts forces the guest driver
to explicitly de-assert the (what it thinks is a) line interrupt in a
device-specific manner, which in turn risks confusing the device which
is configured in MSI mode.

Also, it means Xen's emulation of edge=>line semantics has no clue when
to correctly deassert the line, without having device-specific knowledge
in the hypervisor, and appropriate trap&emulate intercepts on whatever
the devices deassert mechanism is.

I don't see how a feature like this can ever have worked, other than by
shear luck.

~Andrew

Re: Purpose of translate MSI interrupt into INTx for guest passthrough

[Roger Pau Monné]

On Wed, Jan 13, 2021 at 09:00:47PM +0000, Andrew Cooper wrote:
> On 12/01/2021 15:51, Roger Pau Monné wrote:
> > On Tue, Jan 12, 2021 at 09:48:17AM -0500, Jason Andryuk wrote:
> >> On Tue, Jan 12, 2021 at 9:25 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>> Dropping Qing He as this address bounces.
> >>>
> >>> On Tue, Jan 12, 2021 at 03:10:57PM +0100, Roger Pau Monné wrote:
> >>>> Hello,
> >>>>
> >>>> While trying to do some cleanup of the Xen interrupt support for pci
> >>>> pass though I came across the MSI to INTx translation that Xen is in
> >>>> theory capable of performing (ie: use a physical MSI interrupt source
> >>>> and inject that as an INTx to a guest).
> >>>>
> >>>> AFAICT such functionality is not wired up to the toolstack, so it's
> >>>> hard to tell what's the indented purpose, or whether it has seen any
> >>>> usage.
> >>> So apparently it is wired up to the toolstack for qemu-traditional,
> >>> albeit it's disabled by default. There's some documentation in
> >>> xl-pci-configuration:
> >>>
> >>> "When enabled, MSI-INTx translation will always enable MSI on the PCI
> >>> device regardless of whether the guest uses INTx or MSI."
> >>>
> >>> So the main purpose seem to be to always use the MSI interrupt source
> >>> regardless of whether the guest is using INTx or MSI. Maybe the
> >>> purpose was to workaround some bugs when using INTx? Or buggy devices
> >>> with INTx interrupts?
> >>>
> >>> qemu-upstream doesn't seem to support it anymore, so I would still
> >>> like to remove it if we get consensus.
> >> The cover letter from
> >> http://old-list-archives.xenproject.org/archives/html/xen-devel/2009-01/msg00228.html
> >> """
> >> This patchset enables MSI-INTx interrupt translation for HVM
> >> domains. The intention of the patch is to use MSI as the physical
> >> interrupt mechanism for passthrough device as much as possible,
> >> thus reducing the pirq sharing among domains.
> >>
> >> When MSI is globally enabled, if the device has the MSI capability
> >> but doesn't used by the guest, hypervisor will try to user MSI as
> >> the underlying pirq and inject translated INTx irq to guest
> >> vioapic. When guest itself enabled MSI or MSI-X, the translation
> >> is automatically turned off.
> >>
> >> Add a config file option to disable/enable this feature. Also, in
> >> order to allow the user to override the option per device, a
> >> per-device option mechanism is implemented and an MSI-INTx option
> >> is added
> >> """
> >>
> >> It seems like it could be a good idea, but I don't know if it presents
> >> compatibility issues when actually used.
> > Hm, MSI interrupts are edge triggered, while INTx is (usually) level.
> > Also devices capable of multiple MSI vectors will be limited to a
> > single one, and I'm not sure whether the transition from translated
> > MSI to INTx into multiple MSIs would work correctly, as seems tricky.
> >
> >> As you say, it's not supported by qemu-upstream, so maybe it should
> >> just be dropped.
> > I don't really see much value in forcing Xen to always use MSI
> > regardless of whether the guest is using INTx or MSI, and it's likely
> > to cause more issues than benefits.
> >
> > IMO I think we should get rid of this, as the only real value is
> > having Xen using MSI intend of INTx, but it's not introducing any kind
> > of functionality from a guest PoV.
> 
> I find this feature very dubious.
> 
> While I agree that reducing INTx sharing between domains is obviously a
> good thing, I don't see how the device can possibly be expected to work
> if the in-guest driver doesn't have an accurate idea of what's going on.
> 
> There are up to 4 INTx lines, but absolutely nothing to suggest that
> these would logically map to the first 4 MSIs enabled on the device.

Multiple MSI interrupts routed to INTx will never work IMO.

> Even in the simplified case of only INTA and a single MSI, there's
> nothing to suggest that the device will behave in the same way when it
> comes to generating interrupts.

IIRC (at least for plain MSI with message groups) it's possible to
enable a single vector even when multiple ones are available, and that
single vector will be used to deliver all events, kind of similar to
the PCI INTx stuff. But as you note MSI is edge triggered while INTx is
level, so there's likely all kind of weird interactions that could
happen from the 'translation' done by Xen.

I'm up for removing it, we don't have it in qemu-upstream and no one
has complained, so I think it's a good sign that this feature was
unused.

Unless someone objects I will prepare a patch to remove it.

Thanks, Roger.