ath11k VFIO MSI issue with DMAR

ath11k VFIO MSI issue with DMAR

[Jiaxun Yang]

Hi all,

I was trying to do PCIe passthrough for a QCA6390 wireless card.
However I constantly get:

[ 230.587950] DMAR: DRHD: handling fault status reg 2
[ 230.587860] DMAR: [INTR-REMAP] Request device [0x00:0x01] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Error on host kernel.

Did a little bit investigation, I found that ath11k hardware is not following
MSI address and data programmed by PCI config space. It maintains MSI address
and data in it's own internal registers (programmed in 
ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup).

It means when signalling MSI IRQ, it won't write to the doorbell that expected by
host hardware (in this case, Intel IOMMU DHRD's doorbell for IRQ remapping) but
the doorbell that VFIO made up for guest.

I came up with a simple workaround on host kernel that just handle DRHD 0x25 fault
and deliver MSI to guest according to fault address and data.

So I've got a couple of questions about moving forward:
1. Is this hardware behaviour a violation of PCIe spec?
2. Is it expected for IOMMU to handle those ill-formed MSI requests? If not is
it possible to get this workaround upstreamed?
3. For Qualcomm guys, is it possible to fix this behaviour in ath11k driver or firmware?

Thanks

- Jiaxun

ath11k VFIO MSI issue with DMAR

[Jiaxun Yang]

Hi all,

I was trying to do PCIe passthrough for a QCA6390 wireless card.
However I constantly get:

[ 230.587950] DMAR: DRHD: handling fault status reg 2
[ 230.587860] DMAR: [INTR-REMAP] Request device [0x00:0x01] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Error on host kernel.

Did a little bit investigation, I found that ath11k hardware is not following
MSI address and data programmed by PCI config space. It maintains MSI address
and data in it's own internal registers (programmed in 
ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup).

It means when signalling MSI IRQ, it won't write to the doorbell that expected by
host hardware (in this case, Intel IOMMU DHRD's doorbell for IRQ remapping) but
the doorbell that VFIO made up for guest.

I came up with a simple workaround on host kernel that just handle DRHD 0x25 fault
and deliver MSI to guest according to fault address and data.

So I've got a couple of questions about moving forward:
1. Is this hardware behaviour a violation of PCIe spec?
2. Is it expected for IOMMU to handle those ill-formed MSI requests? If not is
it possible to get this workaround upstreamed?
3. For Qualcomm guys, is it possible to fix this behaviour in ath11k driver or firmware?

Thanks

- Jiaxun

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k VFIO MSI issue with DMAR

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

On Tue, 2022-09-13 at 15:09 +0100, Jiaxun Yang wrote:
> 
> So I've got a couple of questions about moving forward:
> 
> 1. Is this hardware behaviour a violation of PCIe spec?
> 
> 2. Is it expected for IOMMU to handle those ill-formed MSI requests? If not is
> 
> it possible to get this workaround upstreamed?
> 
> 3. For Qualcomm guys, is it possible to fix this behaviour in ath11k driver or firmware?

The IOMMU can't handle those requests. The "compatibility format" MSI
directly identifies the *guest* APICID and vector to deliver.

In an ideal world maybe the IOMMU could have been designed to handle
those MSIs directly. The PCIe requester-id could identify the guest and
thus route the interrupt via appropriately-designed tables. Then the
guest could be permitted to compose the intended MSI message for itself
and write it directly to the device's MSI(X) table.

But that isn't how it was done. Instead, the device has to be
programmed to submit "remappable format" MSIs instead. That's what the
IOMMU needs to look up the translation (and the contents of the
remappable format MSI are mostly just an index into the IOMMU table).

So instead of letting the guest program the MSI directly into the
hardware, a VMM will *trap* the writes to the MSI(X) table and create
an interrupt route, then write the *actual* hardware with a remappable
format MSI that will trigger the correct guest interrupt.

Your best bet is probably to have the hypervisor trap accesses to the
ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup
registers and do the same trick of allocating a host IRQ route and
writing the converted MSI message.

The alternative is to build some kind of enlightenment to let the guest
ask the hypervisor to tell it the correct remappable format message to
use... but that's going to get very horrid very quickly. If your
hardware allows you to read the MSI(X) table via some other indirect
route which isn't trapped by the hypervisor to give you back the
"original" thing you wrote, perhaps you could write to an unused MSI
and then read back what the hardware *actually* contains... then write
that to the special register? But that's kind of awful.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]

Re: ath11k VFIO MSI issue with DMAR

[David Woodhouse]

[-- Attachment #1.1: Type: text/plain, Size: 2230 bytes --]

On Tue, 2022-09-13 at 15:09 +0100, Jiaxun Yang wrote:
> 
> So I've got a couple of questions about moving forward:
> 
> 1. Is this hardware behaviour a violation of PCIe spec?
> 
> 2. Is it expected for IOMMU to handle those ill-formed MSI requests? If not is
> 
> it possible to get this workaround upstreamed?
> 
> 3. For Qualcomm guys, is it possible to fix this behaviour in ath11k driver or firmware?

The IOMMU can't handle those requests. The "compatibility format" MSI
directly identifies the *guest* APICID and vector to deliver.

In an ideal world maybe the IOMMU could have been designed to handle
those MSIs directly. The PCIe requester-id could identify the guest and
thus route the interrupt via appropriately-designed tables. Then the
guest could be permitted to compose the intended MSI message for itself
and write it directly to the device's MSI(X) table.

But that isn't how it was done. Instead, the device has to be
programmed to submit "remappable format" MSIs instead. That's what the
IOMMU needs to look up the translation (and the contents of the
remappable format MSI are mostly just an index into the IOMMU table).

So instead of letting the guest program the MSI directly into the
hardware, a VMM will *trap* the writes to the MSI(X) table and create
an interrupt route, then write the *actual* hardware with a remappable
format MSI that will trigger the correct guest interrupt.

Your best bet is probably to have the hypervisor trap accesses to the
ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup
registers and do the same trick of allocating a host IRQ route and
writing the converted MSI message.

The alternative is to build some kind of enlightenment to let the guest
ask the hypervisor to tell it the correct remappable format message to
use... but that's going to get very horrid very quickly. If your
hardware allows you to read the MSI(X) table via some other indirect
route which isn't trapped by the hypervisor to give you back the
"original" thing you wrote, perhaps you could write to an unused MSI
and then read back what the hardware *actually* contains... then write
that to the special register? But that's kind of awful.

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]

[-- Attachment #2: Type: text/plain, Size: 102 bytes --]

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k VFIO MSI issue with DMAR

[Jiaxun Yang]

在2022年9月13日九月 下午3:55，David Woodhouse写道：
> On Tue, 2022-09-13 at 15:09 +0100, Jiaxun Yang wrote:
[...]
>
> Your best bet is probably to have the hypervisor trap accesses to the
> ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup
> registers and do the same trick of allocating a host IRQ route and
> writing the converted MSI message.
>
> The alternative is to build some kind of enlightenment to let the guest
> ask the hypervisor to tell it the correct remappable format message to
> use... but that's going to get very horrid very quickly. If your
> hardware allows you to read the MSI(X) table via some other indirect
> route which isn't trapped by the hypervisor to give you back the
> "original" thing you wrote, perhaps you could write to an unused MSI
> and then read back what the hardware *actually* contains... then write
> that to the special register? But that's kind of awful.

Indeed both ideas sound awful...

My current approach is raise a guest APIC interrupt when 0x25 fault happens.
Though I'm currently hardcoding guest APIC destination but I think it's possible
to check fault MSI address/data from fault event address/data registers against
DMAR interrupt mapping table to determine actual destination?

It's bit of trap and emulation, think that will enable DMAR driver to handle such
situtation.

Thanks.

>
> 附件:
> * smime.p7s

-- 
- Jiaxun

Re: ath11k VFIO MSI issue with DMAR

[Jiaxun Yang]

在2022年9月13日九月 下午3:55，David Woodhouse写道：
> On Tue, 2022-09-13 at 15:09 +0100, Jiaxun Yang wrote:
[...]
>
> Your best bet is probably to have the hypervisor trap accesses to the
> ath11k_ce_srng_msi_ring_params_setup and ath11k_dp_srng_msi_setup
> registers and do the same trick of allocating a host IRQ route and
> writing the converted MSI message.
>
> The alternative is to build some kind of enlightenment to let the guest
> ask the hypervisor to tell it the correct remappable format message to
> use... but that's going to get very horrid very quickly. If your
> hardware allows you to read the MSI(X) table via some other indirect
> route which isn't trapped by the hypervisor to give you back the
> "original" thing you wrote, perhaps you could write to an unused MSI
> and then read back what the hardware *actually* contains... then write
> that to the special register? But that's kind of awful.

Indeed both ideas sound awful...

My current approach is raise a guest APIC interrupt when 0x25 fault happens.
Though I'm currently hardcoding guest APIC destination but I think it's possible
to check fault MSI address/data from fault event address/data registers against
DMAR interrupt mapping table to determine actual destination?

It's bit of trap and emulation, think that will enable DMAR driver to handle such
situtation.

Thanks.

>
> 附件:
> * smime.p7s

-- 
- Jiaxun

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k