* [PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices
@ 2022-05-02 8:48 Nirmal Patel
2022-05-02 8:48 ` [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration Nirmal Patel
2022-05-02 8:49 ` [PATCH 2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") Nirmal Patel
0 siblings, 2 replies; 5+ messages in thread
From: Nirmal Patel @ 2022-05-02 8:48 UTC (permalink / raw)
To: linux-pci; +Cc: Nirmal Patel
Make sure VMD assigns proper IRQ domain to the child devices during
device enumeration. DMAR errors were observed when interrupt remapping
is enabled by intel_iommu because of the fact that VMD child devices
are on different IRQ domain than all other PCI devices.
Nirmal Patel (2):
PCI: vmd: Assign VMD IRQ domain before enumeration
PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
remapping if interrupt remapping is enabled by IOMMU.")
drivers/pci/controller/vmd.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
--
2.26.2
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration 2022-05-02 8:48 [PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices Nirmal Patel @ 2022-05-02 8:48 ` Nirmal Patel 2022-05-11 9:10 ` Lorenzo Pieralisi 2022-05-02 8:49 ` [PATCH 2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") Nirmal Patel 1 sibling, 1 reply; 5+ messages in thread From: Nirmal Patel @ 2022-05-02 8:48 UTC (permalink / raw) To: linux-pci; +Cc: Nirmal Patel VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices. As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance: DMAR: DRHD: handling fault status reg 2 DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com> --- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index eb05cceab964..5015adc04d19 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) vmd_attach_resources(vmd); if (vmd->irq_domain) dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); + else + dev_set_msi_domain(&vmd->bus->dev, + dev_get_msi_domain(&vmd->dev->dev)); vmd_acpi_begin(); -- 2.26.2 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration 2022-05-02 8:48 ` [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration Nirmal Patel @ 2022-05-11 9:10 ` Lorenzo Pieralisi 2022-05-11 16:57 ` Patel, Nirmal 0 siblings, 1 reply; 5+ messages in thread From: Lorenzo Pieralisi @ 2022-05-11 9:10 UTC (permalink / raw) To: Nirmal Patel; +Cc: linux-pci, maz [Adding Marc, to keep an eye on IRQ domain usage] On Mon, May 02, 2022 at 01:48:59AM -0700, Nirmal Patel wrote: > VMD creates and assigns a separate IRQ domain when MSI-X remapping is > enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when > MSI-X remapping is disabled resulting child devices getting default > PCI-MSI IRQ domain. Now when interrupt remapping is enabled by > intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain > including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain > when VMD create a root bus and configures child devices. I would encourage you to rewrite this log, it is unclear - granted, I don't know intel-iommu internals - but IMHO if you explain the issue and the fix thoroughly this could avoid repeating what you have to do in patch(2). Please describe how VMD handles IRQ domains and how you are fixing that. Thanks, Lorenzo > As a result DMAR errors were observed when interrupt remapping was > enabled on Intel Icelake CPUs. For instance: > > DMAR: DRHD: handling fault status reg 2 > DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request > > Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com> > --- > drivers/pci/controller/vmd.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > index eb05cceab964..5015adc04d19 100644 > --- a/drivers/pci/controller/vmd.c > +++ b/drivers/pci/controller/vmd.c > @@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > vmd_attach_resources(vmd); > if (vmd->irq_domain) > dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); > + else > + dev_set_msi_domain(&vmd->bus->dev, > + dev_get_msi_domain(&vmd->dev->dev)); > > vmd_acpi_begin(); > > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration 2022-05-11 9:10 ` Lorenzo Pieralisi @ 2022-05-11 16:57 ` Patel, Nirmal 0 siblings, 0 replies; 5+ messages in thread From: Patel, Nirmal @ 2022-05-11 16:57 UTC (permalink / raw) To: Lorenzo Pieralisi; +Cc: linux-pci, maz On 5/11/2022 2:10 AM, Lorenzo Pieralisi wrote: > [Adding Marc, to keep an eye on IRQ domain usage] > > On Mon, May 02, 2022 at 01:48:59AM -0700, Nirmal Patel wrote: >> VMD creates and assigns a separate IRQ domain when MSI-X remapping is >> enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when >> MSI-X remapping is disabled resulting child devices getting default >> PCI-MSI IRQ domain. Now when interrupt remapping is enabled by >> intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain >> including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain >> when VMD create a root bus and configures child devices. > I would encourage you to rewrite this log, it is unclear - granted, > I don't know intel-iommu internals - but IMHO if you explain the > issue and the fix thoroughly this could avoid repeating what > you have to do in patch(2). > > Please describe how VMD handles IRQ domains and how you are fixing > that. > > Thanks, > Lorenzo I will add more information to the commit logs. Thanks, nirmal >> As a result DMAR errors were observed when interrupt remapping was >> enabled on Intel Icelake CPUs. For instance: >> >> DMAR: DRHD: handling fault status reg 2 >> DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request >> >> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com> >> --- >> drivers/pci/controller/vmd.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c >> index eb05cceab964..5015adc04d19 100644 >> --- a/drivers/pci/controller/vmd.c >> +++ b/drivers/pci/controller/vmd.c >> @@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) >> vmd_attach_resources(vmd); >> if (vmd->irq_domain) >> dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); >> + else >> + dev_set_msi_domain(&vmd->bus->dev, >> + dev_get_msi_domain(&vmd->dev->dev)); >> >> vmd_acpi_begin(); >> >> -- >> 2.26.2 >> ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") 2022-05-02 8:48 [PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices Nirmal Patel 2022-05-02 8:48 ` [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration Nirmal Patel @ 2022-05-02 8:49 ` Nirmal Patel 1 sibling, 0 replies; 5+ messages in thread From: Nirmal Patel @ 2022-05-02 8:49 UTC (permalink / raw) To: linux-pci; +Cc: Nirmal Patel Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") The commit 2565e5b69c44 was added as a workaround to enable MSI-X remapping if IOMMU enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSI-X remapping is disabled. There is no dependency between MSI-X remapping by VMD and interrupt remapping by IOMMU. Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com> --- drivers/pci/controller/vmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 5015adc04d19..94a14a3d7e55 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -6,7 +6,6 @@ #include <linux/device.h> #include <linux/interrupt.h> -#include <linux/iommu.h> #include <linux/irq.h> #include <linux/kernel.h> #include <linux/module.h> @@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ - if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) || - !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) || + if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) || offset[0] || offset[1]) { ret = vmd_alloc_irqs(vmd); if (ret) -- 2.26.2 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-05-11 16:57 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-02 8:48 [PATCH 0/2] PCI: vmd: IRQ domain assignment to sub devices Nirmal Patel
2022-05-02 8:48 ` [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration Nirmal Patel
2022-05-11 9:10 ` Lorenzo Pieralisi
2022-05-11 16:57 ` Patel, Nirmal
2022-05-02 8:49 ` [PATCH 2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") Nirmal Patel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).