[PATCH iommufd v3 0/9] Remove IOMMU_CAP_INTR_REMAP

[PATCH iommufd v3 0/9] Remove IOMMU_CAP_INTR_REMAP

[Jason Gunthorpe]

Currently the kernel has two ways to signal the "isolated MSI" concept
that IOMMU_CAP_INTR_REMAP and irq_domain_check_msi_remap() both lay claim
to.

Harmonize these into a single irq_domain based check under
msi_device_has_isolated_msi().

In real HW "isolated MSI" is implemented in a few different ways:

 - x86 uses "interrupt remapping" which is a block that sits between
   the device and APIC, that can "remap" the MSI MemWr. AMD uses per-RID
   tables to implement isolation while Intel stores the authorized RID in
   each IRTE entry. Part of the remapping is discarding, HW will not
   forward MSIs that don't positively match the tables.

 - ARM GICv3 ITS integrates the concept of an out-of-band "device ID"
   directly into the interrupt controller logic. The tables the GIC checks
   that determine how to deliver the interrupt through the ITS device table
   and interrupt translation tables allow limiting which interrupts device
   IDs can trigger.

 - S390 has unconditionally claimed it has isolated MSI through the iommu
   driver. This is a weaker version of the other arches in that it only
   works between "gisa" domains. See zpci_set_airq() and

    https://lore.kernel.org/r/31af8174-35e9-ebeb-b9ef-74c90d4bfd93@linux.ibm.com/

After this series the "isolated MSI" is tagged based only on the
irq_domains that the interrupt travels through. For x86 enabling interrupt
remapping causes IR irq_domains to be installed in the path, and they can
carry the IRQ_DOMAIN_FLAG_ISOLATED_MSI. For ARM the GICv3 ITS itself
already sets the flag when it is running in a isolated mode, and S390
simply sets it always through an arch hook since it doesn't use
irq_domains at all.

This removes the intrusion of IRQ subsystem information into the iommu
drivers. Linux's iommu_domains abstraction has no bearing at all on the
security of MSI. Even if HW linked to the IOMMU may implement the security
on x86 implementations, Linux models that HW through the irq_domain, not
the iommu_domain.

This is on github: https://github.com/jgunthorpe/linux/commits/secure_msi

v3:
 - Add missing #include to iommu.c
 - Update the comment in msi_device_has_isolated_msi() when
   arch_is_isolated_msi() is added
 - Rebase to v6.2-rc2
v2: https://lore.kernel.org/r/0-v2-10ad79761833+40588-secure_msi_jgg@nvidia.com
 - Rename secure_msi to isolated_msi
 - Add iommu_group_has_isolated_msi() as a core function to support
   VFIO/iommufd. It checks that the group has a consisent isolated_msi
   to catch driver bugs.
 - Revise comment and commit messages for clarity
 - Drop the VFIO iteration patch since iommu_group_has_isolated_msi() just
   does it.
 - Link to Matthew's discussion about S390 and explain it is less secure
v1: https://lore.kernel.org/r/0-v1-9e466539c244+47b5-secure_msi_jgg@nvidia.com

Jason Gunthorpe (9):
  irq: Add msi_device_has_isolated_msi()
  iommu: Add iommu_group_has_isolated_msi()
  vfio/type1: Convert to iommu_group_has_isolated_msi()
  iommufd: Convert to msi_device_has_isolated_msi()
  irq: Remove unused irq_domain_check_msi_remap() code
  irq: Rename IRQ_DOMAIN_MSI_REMAP to IRQ_DOMAIN_ISOLATED_MSI
  iommu/x86: Replace IOMMU_CAP_INTR_REMAP with
    IRQ_DOMAIN_FLAG_ISOLATED_MSI
  irq/s390: Add arch_is_isolated_msi() for s390
  iommu: Remove IOMMU_CAP_INTR_REMAP

 arch/s390/include/asm/msi.h         | 17 +++++++++++++
 drivers/iommu/amd/iommu.c           |  5 ++--
 drivers/iommu/intel/iommu.c         |  2 --
 drivers/iommu/intel/irq_remapping.c |  3 ++-
 drivers/iommu/iommu.c               | 24 ++++++++++++++++++
 drivers/iommu/iommufd/device.c      |  4 +--
 drivers/iommu/s390-iommu.c          |  2 --
 drivers/irqchip/irq-gic-v3-its.c    |  4 +--
 drivers/vfio/vfio_iommu_type1.c     | 16 +++---------
 include/linux/iommu.h               |  2 +-
 include/linux/irqdomain.h           | 29 +++------------------
 include/linux/msi.h                 | 17 +++++++++++++
 kernel/irq/irqdomain.c              | 39 -----------------------------
 kernel/irq/msi.c                    | 27 ++++++++++++++++++++
 14 files changed, 100 insertions(+), 91 deletions(-)
 create mode 100644 arch/s390/include/asm/msi.h


base-commit: 88603b6dc419445847923fcb7fe5080067a30f98
-- 
2.39.0

[PATCH iommufd v3 1/9] irq: Add msi_device_has_isolated_msi()

[Jason Gunthorpe]

This will replace irq_domain_check_msi_remap() in following patches.

The new API makes it more clear what "msi_remap" actually means from a
functional perspective instead of identifying an implementation specific
HW feature.

Isolated MSI means that HW modeled by an irq_domain on the path from the
initiating device to the CPU will validate that the MSI message specifies
an interrupt number that the device is authorized to trigger. This must
block devices from triggering interrupts they are not authorized to
trigger.  Currently authorization means the MSI vector is one assigned to
the device.

This is interesting for securing VFIO use cases where a rouge MSI (eg
created by abusing a normal PCI MemWr DMA) must not allow the VFIO
userspace to impact outside its security domain, eg userspace triggering
interrupts on kernel drivers, a VM triggering interrupts on the
hypervisor, or a VM triggering interrupts on another VM.

As this is actually modeled as a per-irq_domain property, not a global
platform property, correct the interface to accept the device parameter
and scan through only the part of the irq_domains hierarchy originating
from the source device.

Locate the new code in msi.c as it naturally only works with
CONFIG_GENERIC_MSI_IRQ, which also requires CONFIG_IRQ_DOMAIN and
IRQ_DOMAIN_HIERARCHY.

Cc: Eric Auger <eric.auger@redhat.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Tomasz Nowicki <tomasz.nowicki@caviumnetworks.com>
Cc: Bharat Bhushan <bharat.bhushan@nxp.com>
Cc: Will Deacon <will.deacon@arm.com>
Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 include/linux/msi.h | 13 +++++++++++++
 kernel/irq/msi.c    | 27 +++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/include/linux/msi.h b/include/linux/msi.h
index a112b913fff949..e8a3f3a8a7f427 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -649,6 +649,19 @@ int platform_msi_device_domain_alloc(struct irq_domain *domain, unsigned int vir
 void platform_msi_device_domain_free(struct irq_domain *domain, unsigned int virq,
 				     unsigned int nvec);
 void *platform_msi_get_host_data(struct irq_domain *domain);
+
+bool msi_device_has_isolated_msi(struct device *dev);
+#else /* CONFIG_GENERIC_MSI_IRQ */
+static inline bool msi_device_has_isolated_msi(struct device *dev)
+{
+	/*
+	 * Arguably if the platform does not enable MSI support then it has
+	 * "isolated MSI", as an interrupt controller that cannot receive MSIs
+	 * is inherently isolated by our definition. As nobody seems to needs
+	 * this be conservative and return false anyhow.
+	 */
+	return false;
+}
 #endif /* CONFIG_GENERIC_MSI_IRQ */
 
 /* PCI specific interfaces */
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 955267bbc2be63..dfb5d40abac9e8 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -1623,3 +1623,30 @@ struct msi_domain_info *msi_get_domain_info(struct irq_domain *domain)
 {
 	return (struct msi_domain_info *)domain->host_data;
 }
+
+/**
+ * msi_device_has_isolated_msi - True if the device has isolated MSI
+ * @dev: The device to check
+ *
+ * Isolated MSI means that HW modeled by an irq_domain on the path from the
+ * initiating device to the CPU will validate that the MSI message specifies an
+ * interrupt number that the device is authorized to trigger. This must block
+ * devices from triggering interrupts they are not authorized to trigger.
+ * Currently authorization means the MSI vector is one assigned to the device.
+ *
+ * This is interesting for securing VFIO use cases where a rouge MSI (eg created
+ * by abusing a normal PCI MemWr DMA) must not allow the VFIO userspace to
+ * impact outside its security domain, eg userspace triggering interrupts on
+ * kernel drivers, a VM triggering interrupts on the hypervisor, or a VM
+ * triggering interrupts on another VM.
+ */
+bool msi_device_has_isolated_msi(struct device *dev)
+{
+	struct irq_domain *domain = dev_get_msi_domain(dev);
+
+	for (; domain; domain = domain->parent)
+		if (domain->flags & IRQ_DOMAIN_FLAG_MSI_REMAP)
+			return true;
+	return false;
+}
+EXPORT_SYMBOL_GPL(msi_device_has_isolated_msi);
-- 
2.39.0

[PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Jason Gunthorpe]

Compute the isolated_msi over all the devices in the IOMMU group because
iommufd and vfio both need to know that the entire group is isolated
before granting access to it.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommu.c | 26 ++++++++++++++++++++++++++
 include/linux/iommu.h |  1 +
 2 files changed, 27 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index de91dd88705bd3..7f744904e02f4d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -30,6 +30,7 @@
 #include <linux/cc_platform.h>
 #include <trace/events/iommu.h>
 #include <linux/sched/mm.h>
+#include <linux/msi.h>
 
 #include "dma-iommu.h"
 
@@ -1897,6 +1898,31 @@ bool device_iommu_capable(struct device *dev, enum iommu_cap cap)
 }
 EXPORT_SYMBOL_GPL(device_iommu_capable);
 
+/**
+ * iommu_group_has_isolated_msi() - Compute msi_device_has_isolated_msi()
+ *       for a group
+ * @group: Group to query
+ *
+ * IOMMU groups should not have differing values of
+ * msi_device_has_isolated_msi() for devices in a group. However nothing
+ * directly prevents this, so ensure mistakes don't result in isolation failures
+ * by checking that all the devices are the same.
+ */
+bool iommu_group_has_isolated_msi(struct iommu_group *group)
+{
+	struct group_device *group_dev;
+	bool ret = true;
+
+	mutex_lock(&group->mutex);
+	list_for_each_entry(group_dev, &group->devices, list)
+		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
+		       device_iommu_capable(group_dev->dev,
+					    IOMMU_CAP_INTR_REMAP);
+	mutex_unlock(&group->mutex);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_group_has_isolated_msi);
+
 /**
  * iommu_set_fault_handler() - set a fault handler for an iommu domain
  * @domain: iommu domain
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 46e1347bfa2286..9b7a9fa5ad28d3 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -455,6 +455,7 @@ static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
 extern int bus_iommu_probe(struct bus_type *bus);
 extern bool iommu_present(struct bus_type *bus);
 extern bool device_iommu_capable(struct device *dev, enum iommu_cap cap);
+extern bool iommu_group_has_isolated_msi(struct iommu_group *group);
 extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
 extern struct iommu_group *iommu_group_get_by_id(int id);
 extern void iommu_domain_free(struct iommu_domain *domain);
-- 
2.39.0

[PATCH iommufd v3 3/9] vfio/type1: Convert to iommu_group_has_isolated_msi()

[Jason Gunthorpe]

Trivially use the new API.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio_iommu_type1.c | 16 +++-------------
 1 file changed, 3 insertions(+), 13 deletions(-)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 23c24fe98c00d4..393b27a3bd87ee 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -37,7 +37,6 @@
 #include <linux/vfio.h>
 #include <linux/workqueue.h>
 #include <linux/notifier.h>
-#include <linux/irqdomain.h>
 #include "vfio.h"
 
 #define DRIVER_VERSION  "0.2"
@@ -2160,12 +2159,6 @@ static void vfio_iommu_iova_insert_copy(struct vfio_iommu *iommu,
 	list_splice_tail(iova_copy, iova);
 }
 
-/* Redundantly walks non-present capabilities to simplify caller */
-static int vfio_iommu_device_capable(struct device *dev, void *data)
-{
-	return device_iommu_capable(dev, (enum iommu_cap)data);
-}
-
 static int vfio_iommu_domain_alloc(struct device *dev, void *data)
 {
 	struct iommu_domain **domain = data;
@@ -2180,7 +2173,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 	struct vfio_iommu *iommu = iommu_data;
 	struct vfio_iommu_group *group;
 	struct vfio_domain *domain, *d;
-	bool resv_msi, msi_remap;
+	bool resv_msi;
 	phys_addr_t resv_msi_base = 0;
 	struct iommu_domain_geometry *geo;
 	LIST_HEAD(iova_copy);
@@ -2278,11 +2271,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 	INIT_LIST_HEAD(&domain->group_list);
 	list_add(&group->next, &domain->group_list);
 
-	msi_remap = irq_domain_check_msi_remap() ||
-		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
-					     vfio_iommu_device_capable);
-
-	if (!allow_unsafe_interrupts && !msi_remap) {
+	if (!allow_unsafe_interrupts &&
+	    !iommu_group_has_isolated_msi(iommu_group)) {
 		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
 		       __func__);
 		ret = -EPERM;
-- 
2.39.0

[PATCH iommufd v3 4/9] iommufd: Convert to msi_device_has_isolated_msi()

[Jason Gunthorpe]

Trivially use the new API.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommufd/device.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c
index d81f93a321afcb..9f3b9674d72e81 100644
--- a/drivers/iommu/iommufd/device.c
+++ b/drivers/iommu/iommufd/device.c
@@ -4,7 +4,6 @@
 #include <linux/iommufd.h>
 #include <linux/slab.h>
 #include <linux/iommu.h>
-#include <linux/irqdomain.h>
 
 #include "io_pagetable.h"
 #include "iommufd_private.h"
@@ -169,8 +168,7 @@ static int iommufd_device_setup_msi(struct iommufd_device *idev,
 	 * operation from the device (eg a simple DMA) cannot trigger an
 	 * interrupt outside this iommufd context.
 	 */
-	if (!device_iommu_capable(idev->dev, IOMMU_CAP_INTR_REMAP) &&
-	    !irq_domain_check_msi_remap()) {
+	if (!iommu_group_has_isolated_msi(idev->group)) {
 		if (!allow_unsafe_interrupts)
 			return -EPERM;
 
-- 
2.39.0

[PATCH iommufd v3 5/9] irq: Remove unused irq_domain_check_msi_remap() code

[Jason Gunthorpe]

After converting the users of irq_domain_check_msi_remap() it and the
helpers are no longer needed.

The new version does not require all the #ifdef helpers and inlines
because CONFIG_GENERIC_MSI_IRQ always requires CONFIG_IRQ_DOMAIN and
IRQ_DOMAIN_HIERARCHY.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 include/linux/irqdomain.h | 23 -----------------------
 kernel/irq/irqdomain.c    | 39 ---------------------------------------
 2 files changed, 62 deletions(-)

diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index a372086750ca55..b04ce03d3bb69f 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -276,7 +276,6 @@ struct irq_domain *irq_domain_create_legacy(struct fwnode_handle *fwnode,
 					    void *host_data);
 extern struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 						   enum irq_domain_bus_token bus_token);
-extern bool irq_domain_check_msi_remap(void);
 extern void irq_set_default_host(struct irq_domain *host);
 extern struct irq_domain *irq_get_default_host(void);
 extern int irq_domain_alloc_descs(int virq, unsigned int nr_irqs,
@@ -559,13 +558,6 @@ static inline bool irq_domain_is_msi(struct irq_domain *domain)
 	return domain->flags & IRQ_DOMAIN_FLAG_MSI;
 }
 
-static inline bool irq_domain_is_msi_remap(struct irq_domain *domain)
-{
-	return domain->flags & IRQ_DOMAIN_FLAG_MSI_REMAP;
-}
-
-extern bool irq_domain_hierarchical_is_msi_remap(struct irq_domain *domain);
-
 static inline bool irq_domain_is_msi_parent(struct irq_domain *domain)
 {
 	return domain->flags & IRQ_DOMAIN_FLAG_MSI_PARENT;
@@ -611,17 +603,6 @@ static inline bool irq_domain_is_msi(struct irq_domain *domain)
 	return false;
 }
 
-static inline bool irq_domain_is_msi_remap(struct irq_domain *domain)
-{
-	return false;
-}
-
-static inline bool
-irq_domain_hierarchical_is_msi_remap(struct irq_domain *domain)
-{
-	return false;
-}
-
 static inline bool irq_domain_is_msi_parent(struct irq_domain *domain)
 {
 	return false;
@@ -641,10 +622,6 @@ static inline struct irq_domain *irq_find_matching_fwnode(
 {
 	return NULL;
 }
-static inline bool irq_domain_check_msi_remap(void)
-{
-	return false;
-}
 #endif /* !CONFIG_IRQ_DOMAIN */
 
 #endif /* _LINUX_IRQDOMAIN_H */
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index 8fe1da9614ee8d..10495495158210 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -436,31 +436,6 @@ struct irq_domain *irq_find_matching_fwspec(struct irq_fwspec *fwspec,
 }
 EXPORT_SYMBOL_GPL(irq_find_matching_fwspec);
 
-/**
- * irq_domain_check_msi_remap - Check whether all MSI irq domains implement
- * IRQ remapping
- *
- * Return: false if any MSI irq domain does not support IRQ remapping,
- * true otherwise (including if there is no MSI irq domain)
- */
-bool irq_domain_check_msi_remap(void)
-{
-	struct irq_domain *h;
-	bool ret = true;
-
-	mutex_lock(&irq_domain_mutex);
-	list_for_each_entry(h, &irq_domain_list, link) {
-		if (irq_domain_is_msi(h) &&
-		    !irq_domain_hierarchical_is_msi_remap(h)) {
-			ret = false;
-			break;
-		}
-	}
-	mutex_unlock(&irq_domain_mutex);
-	return ret;
-}
-EXPORT_SYMBOL_GPL(irq_domain_check_msi_remap);
-
 /**
  * irq_set_default_host() - Set a "default" irq domain
  * @domain: default domain pointer
@@ -1815,20 +1790,6 @@ static void irq_domain_check_hierarchy(struct irq_domain *domain)
 	if (domain->ops->alloc)
 		domain->flags |= IRQ_DOMAIN_FLAG_HIERARCHY;
 }
-
-/**
- * irq_domain_hierarchical_is_msi_remap - Check if the domain or any
- * parent has MSI remapping support
- * @domain: domain pointer
- */
-bool irq_domain_hierarchical_is_msi_remap(struct irq_domain *domain)
-{
-	for (; domain; domain = domain->parent) {
-		if (irq_domain_is_msi_remap(domain))
-			return true;
-	}
-	return false;
-}
 #else	/* CONFIG_IRQ_DOMAIN_HIERARCHY */
 /**
  * irq_domain_get_irq_data - Get irq_data associated with @virq and @domain
-- 
2.39.0

[PATCH iommufd v3 6/9] irq: Rename IRQ_DOMAIN_MSI_REMAP to IRQ_DOMAIN_ISOLATED_MSI

[Jason Gunthorpe]

What x86 calls "interrupt remapping" is one way to achieve isolated MSI,
make it clear this is talking about isolated MSI, no matter how it is
achieved. This matches the new driver facing API name of
msi_device_has_isolated_msi()

No functional change.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/irqchip/irq-gic-v3-its.c | 4 ++--
 include/linux/irqdomain.h        | 6 ++++--
 kernel/irq/msi.c                 | 2 +-
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 973ede0197e36f..b4069f825a9b73 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4692,7 +4692,7 @@ static bool __maybe_unused its_enable_quirk_socionext_synquacer(void *data)
 		}
 
 		/* the pre-ITS breaks isolation, so disable MSI remapping */
-		its->msi_domain_flags &= ~IRQ_DOMAIN_FLAG_MSI_REMAP;
+		its->msi_domain_flags &= ~IRQ_DOMAIN_FLAG_ISOLATED_MSI;
 		return true;
 	}
 	return false;
@@ -5074,7 +5074,7 @@ static int __init its_probe_one(struct resource *res,
 	its->cmd_write = its->cmd_base;
 	its->fwnode_handle = handle;
 	its->get_msi_base = its_irq_get_msi_base;
-	its->msi_domain_flags = IRQ_DOMAIN_FLAG_MSI_REMAP;
+	its->msi_domain_flags = IRQ_DOMAIN_FLAG_ISOLATED_MSI;
 
 	its_enable_quirks(its);
 
diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index b04ce03d3bb69f..0a3e974b7288d0 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -192,8 +192,10 @@ enum {
 	/* Irq domain implements MSIs */
 	IRQ_DOMAIN_FLAG_MSI		= (1 << 4),
 
-	/* Irq domain implements MSI remapping */
-	IRQ_DOMAIN_FLAG_MSI_REMAP	= (1 << 5),
+	/*
+	 * Irq domain implements isolated MSI, see msi_device_has_isolated_msi()
+	 */
+	IRQ_DOMAIN_FLAG_ISOLATED_MSI	= (1 << 5),
 
 	/* Irq domain doesn't translate anything */
 	IRQ_DOMAIN_FLAG_NO_MAP		= (1 << 6),
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index dfb5d40abac9e8..ac5e224a11b9aa 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -1645,7 +1645,7 @@ bool msi_device_has_isolated_msi(struct device *dev)
 	struct irq_domain *domain = dev_get_msi_domain(dev);
 
 	for (; domain; domain = domain->parent)
-		if (domain->flags & IRQ_DOMAIN_FLAG_MSI_REMAP)
+		if (domain->flags & IRQ_DOMAIN_FLAG_ISOLATED_MSI)
 			return true;
 	return false;
 }
-- 
2.39.0

[PATCH iommufd v3 7/9] iommu/x86: Replace IOMMU_CAP_INTR_REMAP with IRQ_DOMAIN_FLAG_ISOLATED_MSI

[Jason Gunthorpe]

On x86 platforms when the HW can support interrupt remapping the iommu
driver creates an irq_domain for the IR hardware and creates a child MSI
irq_domain.

When the global irq_remapping_enabled is set, the IR MSI domain is
assigned to the PCI devices (by intel_irq_remap_add_device(), or
amd_iommu_set_pci_msi_domain()) making those devices have the isolated MSI
property.

Due to how interrupt domains work, setting IRQ_DOMAIN_FLAG_ISOLATED_MSI on
the parent IR domain will cause all struct devices attached to it to
return true from msi_device_has_isolated_msi(). This replaces the
IOMMU_CAP_INTR_REMAP flag as all places using IOMMU_CAP_INTR_REMAP also
call msi_device_has_isolated_msi()

Set the flag and delete the cap.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/amd/iommu.c           | 5 ++---
 drivers/iommu/intel/iommu.c         | 2 --
 drivers/iommu/intel/irq_remapping.c | 3 ++-
 3 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index cbeaab55c0dbcc..321d50e9df5b4a 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2271,8 +2271,6 @@ static bool amd_iommu_capable(struct device *dev, enum iommu_cap cap)
 	switch (cap) {
 	case IOMMU_CAP_CACHE_COHERENCY:
 		return true;
-	case IOMMU_CAP_INTR_REMAP:
-		return (irq_remapping_enabled == 1);
 	case IOMMU_CAP_NOEXEC:
 		return false;
 	case IOMMU_CAP_PRE_BOOT_PROTECTION:
@@ -3671,7 +3669,8 @@ int amd_iommu_create_irq_domain(struct amd_iommu *iommu)
 	}
 
 	irq_domain_update_bus_token(iommu->ir_domain,  DOMAIN_BUS_AMDVI);
-	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT;
+	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT |
+				   IRQ_DOMAIN_FLAG_ISOLATED_MSI;
 
 	if (amd_iommu_np_cache)
 		iommu->ir_domain->msi_parent_ops = &virt_amdvi_msi_parent_ops;
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 59df7e42fd533c..7cfab5fd5e5964 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -4464,8 +4464,6 @@ static bool intel_iommu_capable(struct device *dev, enum iommu_cap cap)
 	switch (cap) {
 	case IOMMU_CAP_CACHE_COHERENCY:
 		return true;
-	case IOMMU_CAP_INTR_REMAP:
-		return irq_remapping_enabled == 1;
 	case IOMMU_CAP_PRE_BOOT_PROTECTION:
 		return dmar_platform_optin();
 	case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
index f58f5f57af782b..6d01fa078c36fc 100644
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -573,7 +573,8 @@ static int intel_setup_irq_remapping(struct intel_iommu *iommu)
 	}
 
 	irq_domain_update_bus_token(iommu->ir_domain,  DOMAIN_BUS_DMAR);
-	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT;
+	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT |
+				   IRQ_DOMAIN_FLAG_ISOLATED_MSI;
 
 	if (cap_caching_mode(iommu->cap))
 		iommu->ir_domain->msi_parent_ops = &virt_dmar_msi_parent_ops;
-- 
2.39.0

[PATCH iommufd v3 8/9] irq/s390: Add arch_is_isolated_msi() for s390

[Jason Gunthorpe]

s390 doesn't use irq_domains, so it has no place to set
IRQ_DOMAIN_FLAG_ISOLATED_MSI. Instead of continuing to abuse the iommu
subsystem to convey this information add a simple define which s390 can
make statically true. The define will cause msi_device_has_isolated() to
return true.

Remove IOMMU_CAP_INTR_REMAP from the s390 iommu driver.

Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 arch/s390/include/asm/msi.h | 17 +++++++++++++++++
 drivers/iommu/s390-iommu.c  |  2 --
 include/linux/msi.h         | 10 +++++++---
 kernel/irq/msi.c            |  2 +-
 4 files changed, 25 insertions(+), 6 deletions(-)
 create mode 100644 arch/s390/include/asm/msi.h

diff --git a/arch/s390/include/asm/msi.h b/arch/s390/include/asm/msi.h
new file mode 100644
index 00000000000000..399343ed9ffbc6
--- /dev/null
+++ b/arch/s390/include/asm/msi.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_S390_MSI_H
+#define _ASM_S390_MSI_H
+#include <asm-generic/msi.h>
+
+/*
+ * Work around S390 not using irq_domain at all so we can't set
+ * IRQ_DOMAIN_FLAG_ISOLATED_MSI. See for an explanation how it works:
+ *
+ * https://lore.kernel.org/r/31af8174-35e9-ebeb-b9ef-74c90d4bfd93@linux.ibm.com/
+ *
+ * Note this is less isolated than the ARM/x86 versions as userspace can trigger
+ * MSI belonging to kernel devices within the same gisa.
+ */
+#define arch_is_isolated_msi() true
+
+#endif
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index ed33c6cce08362..bb00580a30d84d 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -34,8 +34,6 @@ static bool s390_iommu_capable(struct device *dev, enum iommu_cap cap)
 	switch (cap) {
 	case IOMMU_CAP_CACHE_COHERENCY:
 		return true;
-	case IOMMU_CAP_INTR_REMAP:
-		return true;
 	default:
 		return false;
 	}
diff --git a/include/linux/msi.h b/include/linux/msi.h
index e8a3f3a8a7f427..13c9b74a4575aa 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -48,6 +48,10 @@ typedef struct arch_msi_msg_data {
 } __attribute__ ((packed)) arch_msi_msg_data_t;
 #endif
 
+#ifndef arch_is_isolated_msi
+#define arch_is_isolated_msi() false
+#endif
+
 /**
  * msi_msg - Representation of a MSI message
  * @address_lo:		Low 32 bits of msi message address
@@ -657,10 +661,10 @@ static inline bool msi_device_has_isolated_msi(struct device *dev)
 	/*
 	 * Arguably if the platform does not enable MSI support then it has
 	 * "isolated MSI", as an interrupt controller that cannot receive MSIs
-	 * is inherently isolated by our definition. As nobody seems to needs
-	 * this be conservative and return false anyhow.
+	 * is inherently isolated by our definition. The default definition for
+	 * arch_is_isolated_msi() is conservative and returns false anyhow.
 	 */
-	return false;
+	return arch_is_isolated_msi();
 }
 #endif /* CONFIG_GENERIC_MSI_IRQ */
 
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index ac5e224a11b9aa..4dec57fc4ea639 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -1647,6 +1647,6 @@ bool msi_device_has_isolated_msi(struct device *dev)
 	for (; domain; domain = domain->parent)
 		if (domain->flags & IRQ_DOMAIN_FLAG_ISOLATED_MSI)
 			return true;
-	return false;
+	return arch_is_isolated_msi();
 }
 EXPORT_SYMBOL_GPL(msi_device_has_isolated_msi);
-- 
2.39.0

[PATCH iommufd v3 9/9] iommu: Remove IOMMU_CAP_INTR_REMAP

[Jason Gunthorpe]

No iommu driver implements this any more, get rid of it.

Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommu.c | 4 +---
 include/linux/iommu.h | 1 -
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 7f744904e02f4d..834e6ecf3e5197 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1915,9 +1915,7 @@ bool iommu_group_has_isolated_msi(struct iommu_group *group)
 
 	mutex_lock(&group->mutex);
 	list_for_each_entry(group_dev, &group->devices, list)
-		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
-		       device_iommu_capable(group_dev->dev,
-					    IOMMU_CAP_INTR_REMAP);
+		ret &= msi_device_has_isolated_msi(group_dev->dev);
 	mutex_unlock(&group->mutex);
 	return ret;
 }
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 9b7a9fa5ad28d3..933cc57bfc4818 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -120,7 +120,6 @@ static inline bool iommu_is_dma_domain(struct iommu_domain *domain)
 
 enum iommu_cap {
 	IOMMU_CAP_CACHE_COHERENCY,	/* IOMMU_CACHE is supported */
-	IOMMU_CAP_INTR_REMAP,		/* IOMMU supports interrupt isolation */
 	IOMMU_CAP_NOEXEC,		/* IOMMU_NOEXEC flag */
 	IOMMU_CAP_PRE_BOOT_PROTECTION,	/* Firmware says it used the IOMMU for
 					   DMA protection and we should too */
-- 
2.39.0

Re: [PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Baolu Lu]

On 1/6/2023 3:33 AM, Jason Gunthorpe wrote:
> Compute the isolated_msi over all the devices in the IOMMU group because
> iommufd and vfio both need to know that the entire group is isolated
> before granting access to it.
>
> Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/iommu/iommu.c | 26 ++++++++++++++++++++++++++
>   include/linux/iommu.h |  1 +
>   2 files changed, 27 insertions(+)
>
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index de91dd88705bd3..7f744904e02f4d 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -30,6 +30,7 @@
>   #include <linux/cc_platform.h>
>   #include <trace/events/iommu.h>
>   #include <linux/sched/mm.h>
> +#include <linux/msi.h>
>   
>   #include "dma-iommu.h"
>   
> @@ -1897,6 +1898,31 @@ bool device_iommu_capable(struct device *dev, enum iommu_cap cap)
>   }
>   EXPORT_SYMBOL_GPL(device_iommu_capable);
>   
> +/**
> + * iommu_group_has_isolated_msi() - Compute msi_device_has_isolated_msi()
> + *       for a group
> + * @group: Group to query
> + *
> + * IOMMU groups should not have differing values of
> + * msi_device_has_isolated_msi() for devices in a group. However nothing
> + * directly prevents this, so ensure mistakes don't result in isolation failures
> + * by checking that all the devices are the same.
> + */
> +bool iommu_group_has_isolated_msi(struct iommu_group *group)
> +{
> +	struct group_device *group_dev;
> +	bool ret = true;
> +
> +	mutex_lock(&group->mutex);
> +	list_for_each_entry(group_dev, &group->devices, list)
> +		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> +		       device_iommu_capable(group_dev->dev,
> +					    IOMMU_CAP_INTR_REMAP);
> +	mutex_unlock(&group->mutex);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(iommu_group_has_isolated_msi);
> +
>   /**
>    * iommu_set_fault_handler() - set a fault handler for an iommu domain
>    * @domain: iommu domain
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index 46e1347bfa2286..9b7a9fa5ad28d3 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -455,6 +455,7 @@ static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
>   extern int bus_iommu_probe(struct bus_type *bus);
>   extern bool iommu_present(struct bus_type *bus);
>   extern bool device_iommu_capable(struct device *dev, enum iommu_cap cap);
> +extern bool iommu_group_has_isolated_msi(struct iommu_group *group);

This lacks a static inline definition when CONFIG_IOMMU_API is false?

>   extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
>   extern struct iommu_group *iommu_group_get_by_id(int id);
>   extern void iommu_domain_free(struct iommu_domain *domain);

Others look good to me. With above addressed,

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

--

Best regards,

baolu

Re: [PATCH iommufd v3 3/9] vfio/type1: Convert to iommu_group_has_isolated_msi()

[Baolu Lu]

On 1/6/2023 3:33 AM, Jason Gunthorpe wrote:
> Trivially use the new API.
>
> Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/vfio_iommu_type1.c | 16 +++-------------
>   1 file changed, 3 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 23c24fe98c00d4..393b27a3bd87ee 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -37,7 +37,6 @@
>   #include <linux/vfio.h>
>   #include <linux/workqueue.h>
>   #include <linux/notifier.h>
> -#include <linux/irqdomain.h>
>   #include "vfio.h"
>   
>   #define DRIVER_VERSION  "0.2"
> @@ -2160,12 +2159,6 @@ static void vfio_iommu_iova_insert_copy(struct vfio_iommu *iommu,
>   	list_splice_tail(iova_copy, iova);
>   }
>   
> -/* Redundantly walks non-present capabilities to simplify caller */
> -static int vfio_iommu_device_capable(struct device *dev, void *data)
> -{
> -	return device_iommu_capable(dev, (enum iommu_cap)data);
> -}
> -
>   static int vfio_iommu_domain_alloc(struct device *dev, void *data)
>   {
>   	struct iommu_domain **domain = data;
> @@ -2180,7 +2173,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>   	struct vfio_iommu *iommu = iommu_data;
>   	struct vfio_iommu_group *group;
>   	struct vfio_domain *domain, *d;
> -	bool resv_msi, msi_remap;
> +	bool resv_msi;
>   	phys_addr_t resv_msi_base = 0;
>   	struct iommu_domain_geometry *geo;
>   	LIST_HEAD(iova_copy);
> @@ -2278,11 +2271,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>   	INIT_LIST_HEAD(&domain->group_list);
>   	list_add(&group->next, &domain->group_list);
>   
> -	msi_remap = irq_domain_check_msi_remap() ||
> -		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
> -					     vfio_iommu_device_capable);
> -
> -	if (!allow_unsafe_interrupts && !msi_remap) {
> +	if (!allow_unsafe_interrupts &&
> +	    !iommu_group_has_isolated_msi(iommu_group)) {
>   		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
>   		       __func__);
>   		ret = -EPERM;

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

--

Best regards,

baolu

Re: [PATCH iommufd v3 4/9] iommufd: Convert to msi_device_has_isolated_msi()

[Baolu Lu]

On 1/6/2023 3:33 AM, Jason Gunthorpe wrote:
> Trivially use the new API.
>
> Tested-by: Matthew Rosato<mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian<kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe<jgg@nvidia.com>
> ---
>   drivers/iommu/iommufd/device.c | 4 +---
>   1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c
> index d81f93a321afcb..9f3b9674d72e81 100644
> --- a/drivers/iommu/iommufd/device.c
> +++ b/drivers/iommu/iommufd/device.c
> @@ -4,7 +4,6 @@
>   #include <linux/iommufd.h>
>   #include <linux/slab.h>
>   #include <linux/iommu.h>
> -#include <linux/irqdomain.h>
>   
>   #include "io_pagetable.h"
>   #include "iommufd_private.h"
> @@ -169,8 +168,7 @@ static int iommufd_device_setup_msi(struct iommufd_device *idev,
>   	 * operation from the device (eg a simple DMA) cannot trigger an
>   	 * interrupt outside this iommufd context.
>   	 */
> -	if (!device_iommu_capable(idev->dev, IOMMU_CAP_INTR_REMAP) &&
> -	    !irq_domain_check_msi_remap()) {
> +	if (!iommu_group_has_isolated_msi(idev->group)) {
>   		if (!allow_unsafe_interrupts)
>   			return -EPERM;

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

--

Best regards,

baolu

Re: [PATCH iommufd v3 7/9] iommu/x86: Replace IOMMU_CAP_INTR_REMAP with IRQ_DOMAIN_FLAG_ISOLATED_MSI

[Baolu Lu]

On 1/6/2023 3:33 AM, Jason Gunthorpe wrote:
> On x86 platforms when the HW can support interrupt remapping the iommu
> driver creates an irq_domain for the IR hardware and creates a child MSI
> irq_domain.
> 
> When the global irq_remapping_enabled is set, the IR MSI domain is
> assigned to the PCI devices (by intel_irq_remap_add_device(), or
> amd_iommu_set_pci_msi_domain()) making those devices have the isolated MSI
> property.
> 
> Due to how interrupt domains work, setting IRQ_DOMAIN_FLAG_ISOLATED_MSI on
> the parent IR domain will cause all struct devices attached to it to
> return true from msi_device_has_isolated_msi(). This replaces the
> IOMMU_CAP_INTR_REMAP flag as all places using IOMMU_CAP_INTR_REMAP also
> call msi_device_has_isolated_msi()
> 
> Set the flag and delete the cap.
> 
> Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/iommu/amd/iommu.c           | 5 ++---
>   drivers/iommu/intel/iommu.c         | 2 --
>   drivers/iommu/intel/irq_remapping.c | 3 ++-
>   3 files changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index cbeaab55c0dbcc..321d50e9df5b4a 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2271,8 +2271,6 @@ static bool amd_iommu_capable(struct device *dev, enum iommu_cap cap)
>   	switch (cap) {
>   	case IOMMU_CAP_CACHE_COHERENCY:
>   		return true;
> -	case IOMMU_CAP_INTR_REMAP:
> -		return (irq_remapping_enabled == 1);
>   	case IOMMU_CAP_NOEXEC:
>   		return false;
>   	case IOMMU_CAP_PRE_BOOT_PROTECTION:
> @@ -3671,7 +3669,8 @@ int amd_iommu_create_irq_domain(struct amd_iommu *iommu)
>   	}
>   
>   	irq_domain_update_bus_token(iommu->ir_domain,  DOMAIN_BUS_AMDVI);
> -	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT;
> +	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT |
> +				   IRQ_DOMAIN_FLAG_ISOLATED_MSI;
>   
>   	if (amd_iommu_np_cache)
>   		iommu->ir_domain->msi_parent_ops = &virt_amdvi_msi_parent_ops;
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index 59df7e42fd533c..7cfab5fd5e5964 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -4464,8 +4464,6 @@ static bool intel_iommu_capable(struct device *dev, enum iommu_cap cap)
>   	switch (cap) {
>   	case IOMMU_CAP_CACHE_COHERENCY:
>   		return true;
> -	case IOMMU_CAP_INTR_REMAP:
> -		return irq_remapping_enabled == 1;
>   	case IOMMU_CAP_PRE_BOOT_PROTECTION:
>   		return dmar_platform_optin();
>   	case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
> diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
> index f58f5f57af782b..6d01fa078c36fc 100644
> --- a/drivers/iommu/intel/irq_remapping.c
> +++ b/drivers/iommu/intel/irq_remapping.c
> @@ -573,7 +573,8 @@ static int intel_setup_irq_remapping(struct intel_iommu *iommu)
>   	}
>   
>   	irq_domain_update_bus_token(iommu->ir_domain,  DOMAIN_BUS_DMAR);
> -	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT;
> +	iommu->ir_domain->flags |= IRQ_DOMAIN_FLAG_MSI_PARENT |
> +				   IRQ_DOMAIN_FLAG_ISOLATED_MSI;
>   
>   	if (cap_caching_mode(iommu->cap))
>   		iommu->ir_domain->msi_parent_ops = &virt_dmar_msi_parent_ops;

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

--
Best regards,
baolu

Re: [PATCH iommufd v3 9/9] iommu: Remove IOMMU_CAP_INTR_REMAP

[Baolu Lu]

On 1/6/2023 3:33 AM, Jason Gunthorpe wrote:
> No iommu driver implements this any more, get rid of it.
> 
> Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/iommu/iommu.c | 4 +---
>   include/linux/iommu.h | 1 -
>   2 files changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index 7f744904e02f4d..834e6ecf3e5197 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1915,9 +1915,7 @@ bool iommu_group_has_isolated_msi(struct iommu_group *group)
>   
>   	mutex_lock(&group->mutex);
>   	list_for_each_entry(group_dev, &group->devices, list)
> -		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> -		       device_iommu_capable(group_dev->dev,
> -					    IOMMU_CAP_INTR_REMAP);
> +		ret &= msi_device_has_isolated_msi(group_dev->dev);
>   	mutex_unlock(&group->mutex);
>   	return ret;
>   }
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index 9b7a9fa5ad28d3..933cc57bfc4818 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -120,7 +120,6 @@ static inline bool iommu_is_dma_domain(struct iommu_domain *domain)
>   
>   enum iommu_cap {
>   	IOMMU_CAP_CACHE_COHERENCY,	/* IOMMU_CACHE is supported */
> -	IOMMU_CAP_INTR_REMAP,		/* IOMMU supports interrupt isolation */
>   	IOMMU_CAP_NOEXEC,		/* IOMMU_NOEXEC flag */
>   	IOMMU_CAP_PRE_BOOT_PROTECTION,	/* Firmware says it used the IOMMU for
>   					   DMA protection and we should too */

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

--
Best regards,
baolu

Re: [PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Jason Gunthorpe]

On Fri, Jan 06, 2023 at 07:28:46PM +0800, Baolu Lu wrote:

> > diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> > index 46e1347bfa2286..9b7a9fa5ad28d3 100644
> > --- a/include/linux/iommu.h
> > +++ b/include/linux/iommu.h
> > @@ -455,6 +455,7 @@ static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
> >   extern int bus_iommu_probe(struct bus_type *bus);
> >   extern bool iommu_present(struct bus_type *bus);
> >   extern bool device_iommu_capable(struct device *dev, enum iommu_cap cap);
> > +extern bool iommu_group_has_isolated_msi(struct iommu_group *group);
> 
> This lacks a static inline definition when CONFIG_IOMMU_API is false?

It is not needed, the call sites are all compilation protected by
CONFIG_IOMMU_API already.

Thanks,
Jason

Re: [PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Baolu Lu]

On 1/6/2023 9:12 PM, Jason Gunthorpe wrote:
> On Fri, Jan 06, 2023 at 07:28:46PM +0800, Baolu Lu wrote:
> 
>>> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
>>> index 46e1347bfa2286..9b7a9fa5ad28d3 100644
>>> --- a/include/linux/iommu.h
>>> +++ b/include/linux/iommu.h
>>> @@ -455,6 +455,7 @@ static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
>>>    extern int bus_iommu_probe(struct bus_type *bus);
>>>    extern bool iommu_present(struct bus_type *bus);
>>>    extern bool device_iommu_capable(struct device *dev, enum iommu_cap cap);
>>> +extern bool iommu_group_has_isolated_msi(struct iommu_group *group);
>> This lacks a static inline definition when CONFIG_IOMMU_API is false?
> It is not needed, the call sites are all compilation protected by
> CONFIG_IOMMU_API already.

Thanks for the explanation. It's okay to me as it has been considered.

--
Best regards,
baolu

Re: [PATCH iommufd v3 1/9] irq: Add msi_device_has_isolated_msi()

[Thomas Gleixner]

Jason!

On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:

> Subject: Re: [PATCH iommufd v3 1/9] irq: Add msi_device_has_isolated_msi()

Nit: The correct prefix for this is 'genirq/msi:' 

Other than that:

Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Re: [PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Thomas Gleixner]

On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:
> +
> +	mutex_lock(&group->mutex);
> +	list_for_each_entry(group_dev, &group->devices, list)
> +		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> +		       device_iommu_capable(group_dev->dev,
> +					    IOMMU_CAP_INTR_REMAP);

Nit. This really wants brackets even if they are not required by the
language. Why?

Brackets can be omitted for a single line statement in the loop/if path,
but this

> +		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> +		       device_iommu_capable(group_dev->dev,
> +					    IOMMU_CAP_INTR_REMAP);

is visually a multi line statement. So having brackets makes visual
parsing of this construct way clearer:

	list_for_each_entry(group_dev, &group->devices, list) {
		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
		       device_iommu_capable(group_dev->dev, IOMMU_CAP_INTR_REMAP);
	}

Also get rid of that extra line break. We lifted the 80 characters line
length quite some while ago and made it 100.

Thanks,

        tglx

Re: [PATCH iommufd v3 5/9] irq: Remove unused irq_domain_check_msi_remap() code

[Thomas Gleixner]

On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:

Prefix: genirq/irqdomain:

git log --online $FILEPATH gives you usually a pretty good hint for the prefix.

Other than that:

Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Re: [PATCH iommufd v3 6/9] irq: Rename IRQ_DOMAIN_MSI_REMAP to IRQ_DOMAIN_ISOLATED_MSI

[Thomas Gleixner]

On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:

Comment about prefix allplies here as well...

Other than that:

Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Re: [PATCH iommufd v3 7/9] iommu/x86: Replace IOMMU_CAP_INTR_REMAP with IRQ_DOMAIN_FLAG_ISOLATED_MSI

[Thomas Gleixner]

On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:
> On x86 platforms when the HW can support interrupt remapping the iommu
> driver creates an irq_domain for the IR hardware and creates a child MSI
> irq_domain.
>
> When the global irq_remapping_enabled is set, the IR MSI domain is
> assigned to the PCI devices (by intel_irq_remap_add_device(), or
> amd_iommu_set_pci_msi_domain()) making those devices have the isolated MSI
> property.
>
> Due to how interrupt domains work, setting IRQ_DOMAIN_FLAG_ISOLATED_MSI on
> the parent IR domain will cause all struct devices attached to it to
> return true from msi_device_has_isolated_msi(). This replaces the
> IOMMU_CAP_INTR_REMAP flag as all places using IOMMU_CAP_INTR_REMAP also
> call msi_device_has_isolated_msi()
>
> Set the flag and delete the cap.
>
> Tested-by: Matthew Rosato <mjrosato@linux.ibm.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Acked-by: Thomas Gleixner <tglx@linutronix.de>

Re: [PATCH iommufd v3 2/9] iommu: Add iommu_group_has_isolated_msi()

[Jason Gunthorpe]

On Wed, Jan 11, 2023 at 07:19:34PM +0100, Thomas Gleixner wrote:
> On Thu, Jan 05 2023 at 15:33, Jason Gunthorpe wrote:
> > +
> > +	mutex_lock(&group->mutex);
> > +	list_for_each_entry(group_dev, &group->devices, list)
> > +		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> > +		       device_iommu_capable(group_dev->dev,
> > +					    IOMMU_CAP_INTR_REMAP);
> 
> Nit. This really wants brackets even if they are not required by the
> language. Why?
> 
> Brackets can be omitted for a single line statement in the loop/if path,
> but this
> 
> > +		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
> > +		       device_iommu_capable(group_dev->dev,
> > +					    IOMMU_CAP_INTR_REMAP);
> 
> is visually a multi line statement. So having brackets makes visual
> parsing of this construct way clearer:

Sure, though this is all undone by the last patch which does make it
visually a single statement:

 	list_for_each_entry(group_dev, &group->devices, list)
-		ret &= msi_device_has_isolated_msi(group_dev->dev) ||
-		       device_iommu_capable(group_dev->dev,
-					    IOMMU_CAP_INTR_REMAP);
+		ret &= msi_device_has_isolated_msi(group_dev->dev);

> Also get rid of that extra line break. We lifted the 80 characters line
> length quite some while ago and made it 100.

Ah, there are so many different opinions on that!

I got everything else, I'll stick this in linux-next via the iommufd
tree so more bots can check it - if anyone else has remarks I can make
a v4

Thanks,
Jason