[PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[Jeffrey Hugo]

If the allocation of multiple MSI vectors for multi-MSI fails in the core
PCI framework, the framework will retry the allocation as a single MSI
vector, assuming that meets the min_vecs specified by the requesting
driver.

Hyper-V advertises that multi-MSI is supported, but reuses the VECTOR
domain to implement that for x86.  The VECTOR domain does not support
multi-MSI, so the alloc will always fail and fallback to a single MSI
allocation.

In short, Hyper-V advertises a capability it does not implement.

Hyper-V can support multi-MSI because it coordinates with the hypervisor
to map the MSIs in the IOMMU's interrupt remapper, which is something the
VECTOR domain does not have.  Therefore the fix is simple - copy what the
x86 IOMMU drivers (AMD/Intel-IR) do by removing
X86_IRQ_ALLOC_CONTIGUOUS_VECTORS after calling the VECTOR domain's
pci_msi_prepare().

Fixes: 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")
Signed-off-by: Jeffrey Hugo <quic_jhugo@quicinc.com>
Reviewed-by: Dexuan Cui <decui@microsoft.com>
---

v2:
-Fix grammatical mistake in added comment

 drivers/pci/controller/pci-hyperv.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index d270a204..1cbe24b 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -614,7 +614,16 @@ static void hv_set_msi_entry_from_desc(union hv_msi_entry *msi_entry,
 static int hv_msi_prepare(struct irq_domain *domain, struct device *dev,
 			  int nvec, msi_alloc_info_t *info)
 {
-	return pci_msi_prepare(domain, dev, nvec, info);
+	int ret = pci_msi_prepare(domain, dev, nvec, info);
+
+	/*
+	 * By using the interrupt remapper in the hypervisor IOMMU, contiguous
+	 * CPU vectors is not needed for multi-MSI
+	 */
+	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
+		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
+
+	return ret;
 }
 
 /**
-- 
2.7.4

Re: [PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[Jeffrey Hugo]

On 4/13/2022 7:36 AM, Jeffrey Hugo wrote:
> If the allocation of multiple MSI vectors for multi-MSI fails in the core
> PCI framework, the framework will retry the allocation as a single MSI
> vector, assuming that meets the min_vecs specified by the requesting
> driver.
> 
> Hyper-V advertises that multi-MSI is supported, but reuses the VECTOR
> domain to implement that for x86.  The VECTOR domain does not support
> multi-MSI, so the alloc will always fail and fallback to a single MSI
> allocation.
> 
> In short, Hyper-V advertises a capability it does not implement.
> 
> Hyper-V can support multi-MSI because it coordinates with the hypervisor
> to map the MSIs in the IOMMU's interrupt remapper, which is something the
> VECTOR domain does not have.  Therefore the fix is simple - copy what the
> x86 IOMMU drivers (AMD/Intel-IR) do by removing
> X86_IRQ_ALLOC_CONTIGUOUS_VECTORS after calling the VECTOR domain's
> pci_msi_prepare().
> 
> Fixes: 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")
> Signed-off-by: Jeffrey Hugo <quic_jhugo@quicinc.com>
> Reviewed-by: Dexuan Cui <decui@microsoft.com>
> ---

Ping?

I don't see this in -next, nor have I seen any replies.  It is possible 
I have missed some kind of update, but currently I'm wondering if this 
change is progressing or not.  If there is some kind of process used in 
this area, I'm not familiar with it, so I would appreciate an introduction.

Thanks

-Jeff

Re: [PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[Wei Liu]

On Wed, Apr 20, 2022 at 08:13:22AM -0600, Jeffrey Hugo wrote:
> On 4/13/2022 7:36 AM, Jeffrey Hugo wrote:
> > If the allocation of multiple MSI vectors for multi-MSI fails in the core
> > PCI framework, the framework will retry the allocation as a single MSI
> > vector, assuming that meets the min_vecs specified by the requesting
> > driver.
> > 
> > Hyper-V advertises that multi-MSI is supported, but reuses the VECTOR
> > domain to implement that for x86.  The VECTOR domain does not support
> > multi-MSI, so the alloc will always fail and fallback to a single MSI
> > allocation.
> > 
> > In short, Hyper-V advertises a capability it does not implement.
> > 
> > Hyper-V can support multi-MSI because it coordinates with the hypervisor
> > to map the MSIs in the IOMMU's interrupt remapper, which is something the
> > VECTOR domain does not have.  Therefore the fix is simple - copy what the
> > x86 IOMMU drivers (AMD/Intel-IR) do by removing
> > X86_IRQ_ALLOC_CONTIGUOUS_VECTORS after calling the VECTOR domain's
> > pci_msi_prepare().
> > 
> > Fixes: 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")
> > Signed-off-by: Jeffrey Hugo <quic_jhugo@quicinc.com>
> > Reviewed-by: Dexuan Cui <decui@microsoft.com>
> > ---
> 
> Ping?
> 
> I don't see this in -next, nor have I seen any replies.  It is possible I
> have missed some kind of update, but currently I'm wondering if this change
> is progressing or not.  If there is some kind of process used in this area,
> I'm not familiar with it, so I would appreciate an introduction.

I expect the PCI maintainers to pick this up. If I don't see this picked
up in this week I will apply it to hyperv-next.

Thanks,
Wei.

> 
> Thanks
> 
> -Jeff

Re: [PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[Wei Liu]

On Mon, Apr 25, 2022 at 03:33:44PM +0000, Wei Liu wrote:
> On Wed, Apr 20, 2022 at 08:13:22AM -0600, Jeffrey Hugo wrote:
> > On 4/13/2022 7:36 AM, Jeffrey Hugo wrote:
> > > If the allocation of multiple MSI vectors for multi-MSI fails in the core
> > > PCI framework, the framework will retry the allocation as a single MSI
> > > vector, assuming that meets the min_vecs specified by the requesting
> > > driver.
> > > 
> > > Hyper-V advertises that multi-MSI is supported, but reuses the VECTOR
> > > domain to implement that for x86.  The VECTOR domain does not support
> > > multi-MSI, so the alloc will always fail and fallback to a single MSI
> > > allocation.
> > > 
> > > In short, Hyper-V advertises a capability it does not implement.
> > > 
> > > Hyper-V can support multi-MSI because it coordinates with the hypervisor
> > > to map the MSIs in the IOMMU's interrupt remapper, which is something the
> > > VECTOR domain does not have.  Therefore the fix is simple - copy what the
> > > x86 IOMMU drivers (AMD/Intel-IR) do by removing
> > > X86_IRQ_ALLOC_CONTIGUOUS_VECTORS after calling the VECTOR domain's
> > > pci_msi_prepare().
> > > 
> > > Fixes: 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")
> > > Signed-off-by: Jeffrey Hugo <quic_jhugo@quicinc.com>
> > > Reviewed-by: Dexuan Cui <decui@microsoft.com>
> > > ---
> > 
> > Ping?
> > 
> > I don't see this in -next, nor have I seen any replies.  It is possible I
> > have missed some kind of update, but currently I'm wondering if this change
> > is progressing or not.  If there is some kind of process used in this area,
> > I'm not familiar with it, so I would appreciate an introduction.
> 
> I expect the PCI maintainers to pick this up. If I don't see this picked
> up in this week I will apply it to hyperv-next.

Actually I will pick this up via hyperv-next, because there is another
series which will also touch this driver but at the some time depend on
vmbus changes. I can fix up any potential conflicts easily.

Thanks,
Wei.

Re: [PATCH v2] PCI: hv: Fix multi-MSI to allow more than one MSI vector

[Jeffrey Hugo]

On 4/25/2022 9:49 AM, Wei Liu wrote:
> On Mon, Apr 25, 2022 at 03:33:44PM +0000, Wei Liu wrote:
>> On Wed, Apr 20, 2022 at 08:13:22AM -0600, Jeffrey Hugo wrote:
>>> On 4/13/2022 7:36 AM, Jeffrey Hugo wrote:
>>>> If the allocation of multiple MSI vectors for multi-MSI fails in the core
>>>> PCI framework, the framework will retry the allocation as a single MSI
>>>> vector, assuming that meets the min_vecs specified by the requesting
>>>> driver.
>>>>
>>>> Hyper-V advertises that multi-MSI is supported, but reuses the VECTOR
>>>> domain to implement that for x86.  The VECTOR domain does not support
>>>> multi-MSI, so the alloc will always fail and fallback to a single MSI
>>>> allocation.
>>>>
>>>> In short, Hyper-V advertises a capability it does not implement.
>>>>
>>>> Hyper-V can support multi-MSI because it coordinates with the hypervisor
>>>> to map the MSIs in the IOMMU's interrupt remapper, which is something the
>>>> VECTOR domain does not have.  Therefore the fix is simple - copy what the
>>>> x86 IOMMU drivers (AMD/Intel-IR) do by removing
>>>> X86_IRQ_ALLOC_CONTIGUOUS_VECTORS after calling the VECTOR domain's
>>>> pci_msi_prepare().
>>>>
>>>> Fixes: 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")
>>>> Signed-off-by: Jeffrey Hugo <quic_jhugo@quicinc.com>
>>>> Reviewed-by: Dexuan Cui <decui@microsoft.com>
>>>> ---
>>>
>>> Ping?
>>>
>>> I don't see this in -next, nor have I seen any replies.  It is possible I
>>> have missed some kind of update, but currently I'm wondering if this change
>>> is progressing or not.  If there is some kind of process used in this area,
>>> I'm not familiar with it, so I would appreciate an introduction.
>>
>> I expect the PCI maintainers to pick this up. If I don't see this picked
>> up in this week I will apply it to hyperv-next.
> 
> Actually I will pick this up via hyperv-next, because there is another
> series which will also touch this driver but at the some time depend on
> vmbus changes. I can fix up any potential conflicts easily.

Sounds good to me.  Let me know if you do run into conflicts, and I can 
help.

-Jeff