[PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices

[PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices

[Nirmal Patel]

Make sure VMD assigns proper IRQ domain to the child devices during
device enumeration. DMAR errors were observed when interrupt remapping
is enabled by intel_iommu because of the fact that VMD child devices
are on different IRQ domain than all other PCI devices.

Nirmal Patel (2):
  PCI: vmd: Assign VMD IRQ domain before enumeration
  PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
    remapping if interrupt remapping is enabled by IOMMU.")

 drivers/pci/controller/vmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

-- 
2.26.2

[PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

[Nirmal Patel]

VMD creates and assigns a separate IRQ domain when MSI-X remapping is
enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when
MSI-X remapping is disabled resulting child devices getting default
PCI-MSI IRQ domain. Now when interrupt remapping is enabled by
intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain
including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain
when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was
enabled on Intel Icelake CPUs. For instance:

  DMAR: DRHD: handling fault status reg 2
  DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index eb05cceab964..5015adc04d19 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 
-- 
2.26.2

[PATCH 2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")

[Nirmal Patel]

Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if
interrupt remapping is enabled by IOMMU.")

The commit 2565e5b69c44 was added as a workaround to enable MSI-X
remapping if IOMMU enables interrupt remapping. VMD does not assign
proper IRQ domain to child devices when MSI-X remapping is disabled.
There is no dependency between MSI-X remapping by VMD and interrupt
remapping by IOMMU.

Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 5015adc04d19..94a14a3d7e55 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -6,7 +6,6 @@
 
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)
-- 
2.26.2

Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

[Lorenzo Pieralisi]

[Adding Marc, to keep an eye on IRQ domain usage]

On Mon, May 02, 2022 at 01:48:59AM -0700, Nirmal Patel wrote:
> VMD creates and assigns a separate IRQ domain when MSI-X remapping is
> enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when
> MSI-X remapping is disabled resulting child devices getting default
> PCI-MSI IRQ domain. Now when interrupt remapping is enabled by
> intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain
> including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain
> when VMD create a root bus and configures child devices.

I would encourage you to rewrite this log, it is unclear - granted,
I don't know intel-iommu internals - but IMHO if you explain the
issue and the fix thoroughly this could avoid repeating what
you have to do in patch(2).

Please describe how VMD handles IRQ domains and how you are fixing
that.

Thanks,
Lorenzo

> As a result DMAR errors were observed when interrupt remapping was
> enabled on Intel Icelake CPUs. For instance:
> 
>   DMAR: DRHD: handling fault status reg 2
>   DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request
> 
> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> ---
>  drivers/pci/controller/vmd.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index eb05cceab964..5015adc04d19 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>  	vmd_attach_resources(vmd);
>  	if (vmd->irq_domain)
>  		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
> +	else
> +		dev_set_msi_domain(&vmd->bus->dev,
> +				   dev_get_msi_domain(&vmd->dev->dev));
>  
>  	vmd_acpi_begin();
>  
> -- 
> 2.26.2
> 

Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

[Patel, Nirmal]

On 5/11/2022 2:10 AM, Lorenzo Pieralisi wrote:
> [Adding Marc, to keep an eye on IRQ domain usage]
>
> On Mon, May 02, 2022 at 01:48:59AM -0700, Nirmal Patel wrote:
>> VMD creates and assigns a separate IRQ domain when MSI-X remapping is
>> enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when
>> MSI-X remapping is disabled resulting child devices getting default
>> PCI-MSI IRQ domain. Now when interrupt remapping is enabled by
>> intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain
>> including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain
>> when VMD create a root bus and configures child devices.
> I would encourage you to rewrite this log, it is unclear - granted,
> I don't know intel-iommu internals - but IMHO if you explain the
> issue and the fix thoroughly this could avoid repeating what
> you have to do in patch(2).
>
> Please describe how VMD handles IRQ domains and how you are fixing
> that.
>
> Thanks,
> Lorenzo
I will add more information to the commit logs.

Thanks,
nirmal

>> As a result DMAR errors were observed when interrupt remapping was
>> enabled on Intel Icelake CPUs. For instance:
>>
>>   DMAR: DRHD: handling fault status reg 2
>>   DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request
>>
>> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
>> ---
>>  drivers/pci/controller/vmd.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
>> index eb05cceab964..5015adc04d19 100644
>> --- a/drivers/pci/controller/vmd.c
>> +++ b/drivers/pci/controller/vmd.c
>> @@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>>  	vmd_attach_resources(vmd);
>>  	if (vmd->irq_domain)
>>  		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
>> +	else
>> +		dev_set_msi_domain(&vmd->bus->dev,
>> +				   dev_get_msi_domain(&vmd->dev->dev));
>>  
>>  	vmd_acpi_begin();
>>  
>> -- 
>> 2.26.2
>>