[PATCH v3 00/14] iommu: Permit modular builds of ARM SMMU[v3] drivers

[PATCH v3 00/14] iommu: Permit modular builds of ARM SMMU[v3] drivers

[Will Deacon]

Hi everyone,

This is version three of the patches I previously posted here:

  v1: https://lore.kernel.org/lkml/20191030145112.19738-1-will@kernel.org/
  v2: https://lore.kernel.org/lkml/20191108151608.20932-1-will@kernel.org

Changes since v2 include:

  * Tested successfully on real hardware (!)
  * Handle re-registering of IOMMU bus ops
  * Unregister IOMMU on unload
  * Prevent forced unbinding of SMMU drivers via sysfs
  * Update my email address in MODULE_AUTHOR

Note that if you want to rely on the 'of_devlink' support in linux-next
for dependent probe ordering, then you'll need an extra patch [1] to
support PCI devices upstream of an SMMU. I'm routing that one via Greg.

Will

[1] https://lore.kernel.org/lkml/20191120190028.4722-1-will@kernel.org

Cc: Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc: Jordan Crouse <jcrouse@codeaurora.org>
Cc: John Garry <john.garry@huawei.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Saravana Kannan <saravanak@google.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Isaac J. Manjarres" <isaacm@codeaurora.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Joerg Roedel <joro@8bytes.org>

--->8

Will Deacon (14):
  drivers/iommu: Export core IOMMU API symbols to permit modular drivers
  iommu/of: Request ACS from the PCI core when configuring IOMMU linkage
  PCI: Export pci_ats_disabled() as a GPL symbol to modules
  drivers/iommu: Take a ref to the IOMMU driver prior to ->add_device()
  iommu/of: Take a ref to the IOMMU driver during ->of_xlate()
  drivers/iommu: Allow IOMMU bus ops to be unregistered
  Revert "iommu/arm-smmu: Make arm-smmu-v3 explicitly non-modular"
  Revert "iommu/arm-smmu: Make arm-smmu explicitly non-modular"
  iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers
  iommu/arm-smmu-v3: Unregister IOMMU and bus ops on device removal
  iommu/arm-smmu-v3: Allow building as a module
  iommu/arm-smmu: Unregister IOMMU and bus ops on device removal
  iommu/arm-smmu: Allow building as a module
  iommu/arm-smmu: Update my email address in MODULE_AUTHOR()

 drivers/iommu/Kconfig       |  16 ++++-
 drivers/iommu/Makefile      |   3 +-
 drivers/iommu/arm-smmu-v3.c |  93 +++++++++++++++++---------
 drivers/iommu/arm-smmu.c    | 127 +++++++++++++++++++++++++-----------
 drivers/iommu/iommu-sysfs.c |   5 ++
 drivers/iommu/iommu.c       |  32 ++++++++-
 drivers/iommu/of_iommu.c    |  17 +++--
 drivers/pci/pci.c           |   1 +
 include/linux/iommu.h       |   2 +
 9 files changed, 215 insertions(+), 81 deletions(-)

-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 01/14] drivers/iommu: Export core IOMMU API symbols to permit modular drivers

[Will Deacon]

Building IOMMU drivers as modules requires that the core IOMMU API
symbols are exported as GPL symbols.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/iommu-sysfs.c | 5 +++++
 drivers/iommu/iommu.c       | 8 ++++++++
 2 files changed, 13 insertions(+)

diff --git a/drivers/iommu/iommu-sysfs.c b/drivers/iommu/iommu-sysfs.c
index e436ff813e7e..99869217fbec 100644
--- a/drivers/iommu/iommu-sysfs.c
+++ b/drivers/iommu/iommu-sysfs.c
@@ -87,6 +87,7 @@ int iommu_device_sysfs_add(struct iommu_device *iommu,
 	put_device(iommu->dev);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(iommu_device_sysfs_add);
 
 void iommu_device_sysfs_remove(struct iommu_device *iommu)
 {
@@ -94,6 +95,8 @@ void iommu_device_sysfs_remove(struct iommu_device *iommu)
 	device_unregister(iommu->dev);
 	iommu->dev = NULL;
 }
+EXPORT_SYMBOL_GPL(iommu_device_sysfs_remove);
+
 /*
  * IOMMU drivers can indicate a device is managed by a given IOMMU using
  * this interface.  A link to the device will be created in the "devices"
@@ -119,6 +122,7 @@ int iommu_device_link(struct iommu_device *iommu, struct device *link)
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(iommu_device_link);
 
 void iommu_device_unlink(struct iommu_device *iommu, struct device *link)
 {
@@ -128,3 +132,4 @@ void iommu_device_unlink(struct iommu_device *iommu, struct device *link)
 	sysfs_remove_link(&link->kobj, "iommu");
 	sysfs_remove_link_from_group(&iommu->dev->kobj, "devices", dev_name(link));
 }
+EXPORT_SYMBOL_GPL(iommu_device_unlink);
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d658c7c6a2ab..c1aadb570145 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -141,6 +141,7 @@ int iommu_device_register(struct iommu_device *iommu)
 	spin_unlock(&iommu_device_lock);
 	return 0;
 }
+EXPORT_SYMBOL_GPL(iommu_device_register);
 
 void iommu_device_unregister(struct iommu_device *iommu)
 {
@@ -148,6 +149,7 @@ void iommu_device_unregister(struct iommu_device *iommu)
 	list_del(&iommu->list);
 	spin_unlock(&iommu_device_lock);
 }
+EXPORT_SYMBOL_GPL(iommu_device_unregister);
 
 static struct iommu_param *iommu_get_dev_param(struct device *dev)
 {
@@ -886,6 +888,7 @@ struct iommu_group *iommu_group_ref_get(struct iommu_group *group)
 	kobject_get(group->devices_kobj);
 	return group;
 }
+EXPORT_SYMBOL_GPL(iommu_group_ref_get);
 
 /**
  * iommu_group_put - Decrement group reference
@@ -1259,6 +1262,7 @@ struct iommu_group *generic_device_group(struct device *dev)
 {
 	return iommu_group_alloc();
 }
+EXPORT_SYMBOL_GPL(generic_device_group);
 
 /*
  * Use standard PCI bus topology, isolation features, and DMA alias quirks
@@ -1326,6 +1330,7 @@ struct iommu_group *pci_device_group(struct device *dev)
 	/* No shared group found, allocate new */
 	return iommu_group_alloc();
 }
+EXPORT_SYMBOL_GPL(pci_device_group);
 
 /* Get the IOMMU group for device on fsl-mc bus */
 struct iommu_group *fsl_mc_device_group(struct device *dev)
@@ -1338,6 +1343,7 @@ struct iommu_group *fsl_mc_device_group(struct device *dev)
 		group = iommu_group_alloc();
 	return group;
 }
+EXPORT_SYMBOL_GPL(fsl_mc_device_group);
 
 /**
  * iommu_group_get_for_dev - Find or create the IOMMU group for a device
@@ -1406,6 +1412,7 @@ struct iommu_group *iommu_group_get_for_dev(struct device *dev)
 
 	return group;
 }
+EXPORT_SYMBOL(iommu_group_get_for_dev);
 
 struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
 {
@@ -2185,6 +2192,7 @@ struct iommu_resv_region *iommu_alloc_resv_region(phys_addr_t start,
 	region->type = type;
 	return region;
 }
+EXPORT_SYMBOL_GPL(iommu_alloc_resv_region);
 
 static int
 request_default_domain_for_dev(struct device *dev, unsigned long type)
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 02/14] iommu/of: Request ACS from the PCI core when configuring IOMMU linkage

[Will Deacon]

To avoid having to export 'pci_request_acs()' to modular IOMMU drivers,
move the call into the 'of_dma_configure()' path in a similar manner to
the way in which ACS is configured when probing via ACPI/IORT.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 614a93aa5305..78faa9f73a91 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -177,6 +177,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			.np = master_np,
 		};
 
+		pci_request_acs();
 		err = pci_for_each_dma_alias(to_pci_dev(dev),
 					     of_pci_iommu_init, &info);
 	} else if (dev_is_fsl_mc(dev)) {
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 03/14] PCI: Export pci_ats_disabled() as a GPL symbol to modules

[Will Deacon]

Building drivers for ATS-aware IOMMUs as modules requires access to
pci_ats_disabled(). Export it as a GPL symbol to get things working.

Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/pci/pci.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index a97e2571a527..4fbe5b576dd8 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -123,6 +123,7 @@ bool pci_ats_disabled(void)
 {
 	return pcie_ats_disabled;
 }
+EXPORT_SYMBOL_GPL(pci_ats_disabled);
 
 /* Disable bridge_d3 for all PCIe ports */
 static bool pci_bridge_d3_disable;
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 04/14] drivers/iommu: Take a ref to the IOMMU driver prior to ->add_device()

[Will Deacon]

To avoid accidental removal of an active IOMMU driver module, take a
reference to the driver module in 'iommu_probe_device()' immediately
prior to invoking the '->add_device()' callback and hold it until the
after the device has been removed by '->remove_device()'.

Suggested-by: Joerg Roedel <joro@8bytes.org>
Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/iommu.c | 19 +++++++++++++++++--
 include/linux/iommu.h |  2 ++
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index c1aadb570145..4bfecfbbe2cf 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -22,6 +22,7 @@
 #include <linux/bitops.h>
 #include <linux/property.h>
 #include <linux/fsl/mc.h>
+#include <linux/module.h>
 #include <trace/events/iommu.h>
 
 static struct kset *iommu_group_kset;
@@ -185,10 +186,21 @@ int iommu_probe_device(struct device *dev)
 	if (!iommu_get_dev_param(dev))
 		return -ENOMEM;
 
+	if (!try_module_get(ops->owner)) {
+		ret = -EINVAL;
+		goto err_free_dev_param;
+	}
+
 	ret = ops->add_device(dev);
 	if (ret)
-		iommu_free_dev_param(dev);
+		goto err_module_put;
+
+	return 0;
 
+err_module_put:
+	module_put(ops->owner);
+err_free_dev_param:
+	iommu_free_dev_param(dev);
 	return ret;
 }
 
@@ -199,7 +211,10 @@ void iommu_release_device(struct device *dev)
 	if (dev->iommu_group)
 		ops->remove_device(dev);
 
-	iommu_free_dev_param(dev);
+	if (dev->iommu_param) {
+		module_put(ops->owner);
+		iommu_free_dev_param(dev);
+	}
 }
 
 static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 29bac5345563..d9dab5a3e912 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -245,6 +245,7 @@ struct iommu_iotlb_gather {
  * @sva_get_pasid: Get PASID associated to a SVA handle
  * @page_response: handle page request response
  * @pgsize_bitmap: bitmap of all possible supported page sizes
+ * @owner: Driver module providing these ops
  */
 struct iommu_ops {
 	bool (*capable)(enum iommu_cap);
@@ -308,6 +309,7 @@ struct iommu_ops {
 			     struct iommu_page_response *msg);
 
 	unsigned long pgsize_bitmap;
+	struct module *owner;
 };
 
 /**
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 05/14] iommu/of: Take a ref to the IOMMU driver during ->of_xlate()

[Will Deacon]

Ensure that we hold a reference to the IOMMU driver module while calling
the '->of_xlate()' callback during early device probing.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/of_iommu.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 78faa9f73a91..25491403a0bd 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -8,6 +8,7 @@
 #include <linux/export.h>
 #include <linux/iommu.h>
 #include <linux/limits.h>
+#include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_iommu.h>
 #include <linux/of_pci.h>
@@ -89,16 +90,16 @@ static int of_iommu_xlate(struct device *dev,
 {
 	const struct iommu_ops *ops;
 	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
-	int err;
+	int ret;
 
 	ops = iommu_ops_from_fwnode(fwnode);
 	if ((ops && !ops->of_xlate) ||
 	    !of_device_is_available(iommu_spec->np))
 		return NO_IOMMU;
 
-	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
-	if (err)
-		return err;
+	ret = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (ret)
+		return ret;
 	/*
 	 * The otherwise-empty fwspec handily serves to indicate the specific
 	 * IOMMU device we're waiting for, which will be useful if we ever get
@@ -107,7 +108,12 @@ static int of_iommu_xlate(struct device *dev,
 	if (!ops)
 		return driver_deferred_probe_check_state(dev);
 
-	return ops->of_xlate(dev, iommu_spec);
+	if (!try_module_get(ops->owner))
+		return -ENODEV;
+
+	ret = ops->of_xlate(dev, iommu_spec);
+	module_put(ops->owner);
+	return ret;
 }
 
 struct of_pci_iommu_alias_info {
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 06/14] drivers/iommu: Allow IOMMU bus ops to be unregistered

[Will Deacon]

'bus_set_iommu()' allows IOMMU drivers to register their ops for a given
bus type. Unfortunately, it then doesn't allow them to be removed, which
is necessary for modular drivers to shutdown cleanly so that they can be
reloaded later on.

Allow 'bus_set_iommu()' to take a NULL 'ops' argument, which clear the
ops pointer for the selected bus_type.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/iommu.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 4bfecfbbe2cf..e99704c2e06b 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1558,6 +1558,11 @@ int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
 {
 	int err;
 
+	if (ops == NULL) {
+		bus->iommu_ops = NULL;
+		return 0;
+	}
+
 	if (bus->iommu_ops != NULL)
 		return -EBUSY;
 
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 07/14] Revert "iommu/arm-smmu: Make arm-smmu-v3 explicitly non-modular"

[Will Deacon]

This reverts commit c07b6426df922d21a13a959cf785d46e9c531941.

Let's get the SMMUv3 driver building as a module, which means putting
back some dead code that we used to carry.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu-v3.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 8da93e730d6f..2ad8e2ca0583 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -21,8 +21,7 @@
 #include <linux/io-pgtable.h>
 #include <linux/iommu.h>
 #include <linux/iopoll.h>
-#include <linux/init.h>
-#include <linux/moduleparam.h>
+#include <linux/module.h>
 #include <linux/msi.h>
 #include <linux/of.h>
 #include <linux/of_address.h>
@@ -384,10 +383,6 @@
 #define MSI_IOVA_BASE			0x8000000
 #define MSI_IOVA_LENGTH			0x100000
 
-/*
- * not really modular, but the easiest way to keep compat with existing
- * bootargs behaviour is to continue using module_param_named here.
- */
 static bool disable_bypass = 1;
 module_param_named(disable_bypass, disable_bypass, bool, S_IRUGO);
 MODULE_PARM_DESC(disable_bypass,
@@ -3683,25 +3678,37 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	return 0;
 }
 
-static void arm_smmu_device_shutdown(struct platform_device *pdev)
+static int arm_smmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
 
 	arm_smmu_device_disable(smmu);
+
+	return 0;
+}
+
+static void arm_smmu_device_shutdown(struct platform_device *pdev)
+{
+	arm_smmu_device_remove(pdev);
 }
 
 static const struct of_device_id arm_smmu_of_match[] = {
 	{ .compatible = "arm,smmu-v3", },
 	{ },
 };
+MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
 
 static struct platform_driver arm_smmu_driver = {
 	.driver	= {
 		.name		= "arm-smmu-v3",
 		.of_match_table	= of_match_ptr(arm_smmu_of_match),
-		.suppress_bind_attrs = true,
 	},
 	.probe	= arm_smmu_device_probe,
+	.remove	= arm_smmu_device_remove,
 	.shutdown = arm_smmu_device_shutdown,
 };
-builtin_platform_driver(arm_smmu_driver);
+module_platform_driver(arm_smmu_driver);
+
+MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
+MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
+MODULE_LICENSE("GPL v2");
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 08/14] Revert "iommu/arm-smmu: Make arm-smmu explicitly non-modular"

[Will Deacon]

This reverts commit addb672f200f4e99368270da205320b83efe01a0.

Let's get the SMMU driver building as a module, which means putting
back some dead code that we used to carry.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu.c | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 7c503a6bc585..53bbe0663b9e 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -27,8 +27,7 @@
 #include <linux/interrupt.h>
 #include <linux/io.h>
 #include <linux/iopoll.h>
-#include <linux/init.h>
-#include <linux/moduleparam.h>
+#include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
@@ -59,10 +58,6 @@
 #define MSI_IOVA_LENGTH			0x100000
 
 static int force_stage;
-/*
- * not really modular, but the easiest way to keep compat with existing
- * bootargs behaviour is to continue using module_param() here.
- */
 module_param(force_stage, int, S_IRUGO);
 MODULE_PARM_DESC(force_stage,
 	"Force SMMU mappings to be installed at a particular stage of translation. A value of '1' or '2' forces the corresponding stage. All other values are ignored (i.e. no stage is forced). Note that selecting a specific stage will disable support for nested translation.");
@@ -1878,6 +1873,7 @@ static const struct of_device_id arm_smmu_of_match[] = {
 	{ .compatible = "qcom,smmu-v2", .data = &qcom_smmuv2 },
 	{ },
 };
+MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
 
 #ifdef CONFIG_ACPI
 static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
@@ -2165,12 +2161,12 @@ static int arm_smmu_legacy_bus_init(void)
 }
 device_initcall_sync(arm_smmu_legacy_bus_init);
 
-static void arm_smmu_device_shutdown(struct platform_device *pdev)
+static int arm_smmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
 
 	if (!smmu)
-		return;
+		return -ENODEV;
 
 	if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
 		dev_err(&pdev->dev, "removing device with active domains!\n");
@@ -2186,6 +2182,12 @@ static void arm_smmu_device_shutdown(struct platform_device *pdev)
 		clk_bulk_disable(smmu->num_clks, smmu->clks);
 
 	clk_bulk_unprepare(smmu->num_clks, smmu->clks);
+	return 0;
+}
+
+static void arm_smmu_device_shutdown(struct platform_device *pdev)
+{
+	arm_smmu_device_remove(pdev);
 }
 
 static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
@@ -2235,12 +2237,16 @@ static const struct dev_pm_ops arm_smmu_pm_ops = {
 
 static struct platform_driver arm_smmu_driver = {
 	.driver	= {
-		.name			= "arm-smmu",
-		.of_match_table		= of_match_ptr(arm_smmu_of_match),
-		.pm			= &arm_smmu_pm_ops,
-		.suppress_bind_attrs	= true,
+		.name		= "arm-smmu",
+		.of_match_table	= of_match_ptr(arm_smmu_of_match),
+		.pm		= &arm_smmu_pm_ops,
 	},
 	.probe	= arm_smmu_device_probe,
+	.remove	= arm_smmu_device_remove,
 	.shutdown = arm_smmu_device_shutdown,
 };
-builtin_platform_driver(arm_smmu_driver);
+module_platform_driver(arm_smmu_driver);
+
+MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
+MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
+MODULE_LICENSE("GPL v2");
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 09/14] iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers

[Will Deacon]

Forcefully unbinding the Arm SMMU drivers is a pretty dangerous operation,
since it will likely lead to catastrophic failure for any DMA devices
mastering through the SMMU being unbound. When the driver then attempts
to "handle" the fatal faults, it's very easy to trip over dead data
structures, leading to use-after-free.

On John's machine, he reports that the machine was "unusable" due to
loss of the storage controller following a forced unbind of the SMMUv3
driver:

  | # cd ./bus/platform/drivers/arm-smmu-v3
  | # echo arm-smmu-v3.0.auto > unbind
  | hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
  | platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
  | [hwprod 0x00000146, hwcons 0x00000000]

Prevent this forced unbinding of the drivers by setting "suppress_bind_attrs"
to true.

Link: https://lore.kernel.org/lkml/06dfd385-1af0-3106-4cc5-6a5b8e864759@huawei.com
Reported-by: John Garry <john.garry@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu-v3.c | 5 +++--
 drivers/iommu/arm-smmu.c    | 7 ++++---
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 2ad8e2ca0583..3fd75abce3bb 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -3700,8 +3700,9 @@ MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
 
 static struct platform_driver arm_smmu_driver = {
 	.driver	= {
-		.name		= "arm-smmu-v3",
-		.of_match_table	= of_match_ptr(arm_smmu_of_match),
+		.name			= "arm-smmu-v3",
+		.of_match_table		= of_match_ptr(arm_smmu_of_match),
+		.suppress_bind_attrs	= true,
 	},
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 53bbe0663b9e..d6c83bd69555 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2237,9 +2237,10 @@ static const struct dev_pm_ops arm_smmu_pm_ops = {
 
 static struct platform_driver arm_smmu_driver = {
 	.driver	= {
-		.name		= "arm-smmu",
-		.of_match_table	= of_match_ptr(arm_smmu_of_match),
-		.pm		= &arm_smmu_pm_ops,
+		.name			= "arm-smmu",
+		.of_match_table		= of_match_ptr(arm_smmu_of_match),
+		.pm			= &arm_smmu_pm_ops,
+		.suppress_bind_attrs    = true,
 	},
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 09/14] iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers

[John Garry]

On 21/11/2019 11:49, Will Deacon wrote:
> Forcefully unbinding the Arm SMMU drivers is a pretty dangerous operation,
> since it will likely lead to catastrophic failure for any DMA devices
> mastering through the SMMU being unbound. When the driver then attempts
> to "handle" the fatal faults, it's very easy to trip over dead data
> structures, leading to use-after-free.
> 
> On John's machine, he reports that the machine was "unusable" due to
> loss of the storage controller following a forced unbind of the SMMUv3
> driver:
> 
>    | # cd ./bus/platform/drivers/arm-smmu-v3
>    | # echo arm-smmu-v3.0.auto > unbind
>    | hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
>    | platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
>    | [hwprod 0x00000146, hwcons 0x00000000]
> 
> Prevent this forced unbinding of the drivers by setting "suppress_bind_attrs"
> to true.

This seems a reasonable approach for now.

BTW, I'll give this series a spin this week, which again looks to be 
your iommu/module branch, excluding the new IORT patch.

Cheers,
John

> 
> Link: https://lore.kernel.org/lkml/06dfd385-1af0-3106-4cc5-6a5b8e864759@huawei.com
> Reported-by: John Garry <john.garry@huawei.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>   drivers/iommu/arm-smmu-v3.c | 5 +++--
>   drivers/iommu/arm-smmu.c    | 7 ++++---
>   2 files changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index 2ad8e2ca0583..3fd75abce3bb 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -3700,8 +3700,9 @@ MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>   
>   static struct platform_driver arm_smmu_driver = {
>   	.driver	= {
> -		.name		= "arm-smmu-v3",
> -		.of_match_table	= of_match_ptr(arm_smmu_of_match),
> +		.name			= "arm-smmu-v3",
> +		.of_match_table		= of_match_ptr(arm_smmu_of_match),
> +		.suppress_bind_attrs	= true,
>   	},
>   	.probe	= arm_smmu_device_probe,
>   	.remove	= arm_smmu_device_remove,
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 53bbe0663b9e..d6c83bd69555 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -2237,9 +2237,10 @@ static const struct dev_pm_ops arm_smmu_pm_ops = {
>   
>   static struct platform_driver arm_smmu_driver = {
>   	.driver	= {
> -		.name		= "arm-smmu",
> -		.of_match_table	= of_match_ptr(arm_smmu_of_match),
> -		.pm		= &arm_smmu_pm_ops,
> +		.name			= "arm-smmu",
> +		.of_match_table		= of_match_ptr(arm_smmu_of_match),
> +		.pm			= &arm_smmu_pm_ops,
> +		.suppress_bind_attrs    = true,
>   	},
>   	.probe	= arm_smmu_device_probe,
>   	.remove	= arm_smmu_device_remove,
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 09/14] iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers

[Saravana Kannan via iommu]

On Tue, Nov 26, 2019 at 1:13 AM John Garry <john.garry@huawei.com> wrote:
>
> On 21/11/2019 11:49, Will Deacon wrote:
> > Forcefully unbinding the Arm SMMU drivers is a pretty dangerous operation,
> > since it will likely lead to catastrophic failure for any DMA devices
> > mastering through the SMMU being unbound. When the driver then attempts
> > to "handle" the fatal faults, it's very easy to trip over dead data
> > structures, leading to use-after-free.
> >
> > On John's machine, he reports that the machine was "unusable" due to
> > loss of the storage controller following a forced unbind of the SMMUv3
> > driver:
> >
> >    | # cd ./bus/platform/drivers/arm-smmu-v3
> >    | # echo arm-smmu-v3.0.auto > unbind
> >    | hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
> >    | platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
> >    | [hwprod 0x00000146, hwcons 0x00000000]
> >
> > Prevent this forced unbinding of the drivers by setting "suppress_bind_attrs"
> > to true.
>
> This seems a reasonable approach for now.
>
> BTW, I'll give this series a spin this week, which again looks to be
> your iommu/module branch, excluding the new IORT patch.

Is this on a platform where of_devlink creates device links between
the iommu device and its suppliers? I'm guessing no? Because device
links should for unbinding of all the consumers before unbinding the
supplier.

Looks like it'll still allow the supplier to unbind if the consumers
don't allow unbinding. Is that the case here?

-Saravana
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 09/14] iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers

[John Garry]

On 26/11/2019 20:27, Saravana Kannan wrote:
> On Tue, Nov 26, 2019 at 1:13 AM John Garry <john.garry@huawei.com> wrote:
>>
>> On 21/11/2019 11:49, Will Deacon wrote:
>>> Forcefully unbinding the Arm SMMU drivers is a pretty dangerous operation,
>>> since it will likely lead to catastrophic failure for any DMA devices
>>> mastering through the SMMU being unbound. When the driver then attempts
>>> to "handle" the fatal faults, it's very easy to trip over dead data
>>> structures, leading to use-after-free.
>>>
>>> On John's machine, he reports that the machine was "unusable" due to
>>> loss of the storage controller following a forced unbind of the SMMUv3
>>> driver:
>>>
>>>     | # cd ./bus/platform/drivers/arm-smmu-v3
>>>     | # echo arm-smmu-v3.0.auto > unbind
>>>     | hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
>>>     | platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
>>>     | [hwprod 0x00000146, hwcons 0x00000000]
>>>
>>> Prevent this forced unbinding of the drivers by setting "suppress_bind_attrs"
>>> to true.
>>
>> This seems a reasonable approach for now.
>>
>> BTW, I'll give this series a spin this week, which again looks to be
>> your iommu/module branch, excluding the new IORT patch.
> 

Hi Saravana,

> Is this on a platform where of_devlink creates device links between
> the iommu device and its suppliers?I'm guessing no? Because device
> links should for unbinding of all the consumers before unbinding the
> supplier.

I'm only really interested in ACPI, TBH.

> 
> Looks like it'll still allow the supplier to unbind if the consumers
> don't allow unbinding. Is that the case here?

So just unbinding the driver from a device does not delete the device 
nor exit the device from it's IOMMU group - so we keep the reference to 
the SMMU ko. As such, I don't know how to realistically test unloading 
the SMMU ko when we have platform devices involved. Maybe someone can 
enlighten me...

Thanks,
John
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 09/14] iommu/arm-smmu: Prevent forced unbinding of Arm SMMU drivers

[John Garry]

On 27/11/2019 11:04, John Garry wrote:
> On 26/11/2019 20:27, Saravana Kannan wrote:
>> On Tue, Nov 26, 2019 at 1:13 AM John Garry <john.garry@huawei.com> wrote:
>>>
>>> On 21/11/2019 11:49, Will Deacon wrote:
>>>> Forcefully unbinding the Arm SMMU drivers is a pretty dangerous 
>>>> operation,
>>>> since it will likely lead to catastrophic failure for any DMA devices
>>>> mastering through the SMMU being unbound. When the driver then attempts
>>>> to "handle" the fatal faults, it's very easy to trip over dead data
>>>> structures, leading to use-after-free.
>>>>
>>>> On John's machine, he reports that the machine was "unusable" due to
>>>> loss of the storage controller following a forced unbind of the SMMUv3
>>>> driver:
>>>>
>>>>     | # cd ./bus/platform/drivers/arm-smmu-v3
>>>>     | # echo arm-smmu-v3.0.auto > unbind
>>>>     | hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
>>>>     | platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
>>>>     | [hwprod 0x00000146, hwcons 0x00000000]
>>>>
>>>> Prevent this forced unbinding of the drivers by setting 
>>>> "suppress_bind_attrs"
>>>> to true.
>>>
>>> This seems a reasonable approach for now.
>>>
>>> BTW, I'll give this series a spin this week, which again looks to be
>>> your iommu/module branch, excluding the new IORT patch.
>>
> 
> Hi Saravana,
> 
>> Is this on a platform where of_devlink creates device links between
>> the iommu device and its suppliers?I'm guessing no? Because device
>> links should for unbinding of all the consumers before unbinding the
>> supplier.
> 
> I'm only really interested in ACPI, TBH.
> 
>>
>> Looks like it'll still allow the supplier to unbind if the consumers
>> don't allow unbinding. Is that the case here?
> 
> So just unbinding the driver from a device does not delete the device 
> nor exit the device from it's IOMMU group - so we keep the reference to 
> the SMMU ko. As such, I don't know how to realistically test unloading 
> the SMMU ko when we have platform devices involved. Maybe someone can 
> enlighten me...

But I could do it on our D06 dev board, where all devices behind the 
SMMUs are PCI based:

--- Initial state ---

root@(none)$ dmesg | grep Adding
[   30.271801] pcieport 0000:00:00.0: Adding to iommu group 0
[   30.296088] pcieport 0000:00:04.0: Adding to iommu group 1
[   30.322234] pcieport 0000:00:08.0: Adding to iommu group 2
[   30.335641] pcieport 0000:00:0c.0: Adding to iommu group 3
[   30.343114] pcieport 0000:00:10.0: Adding to iommu group 4
[   30.355650] pcieport 0000:00:12.0: Adding to iommu group 5
[   30.366794] pcieport 0000:7c:00.0: Adding to iommu group 6
[   30.377993] hns3 0000:7d:00.0: Adding to iommu group 7
[   31.861957] hns3 0000:7d:00.1: Adding to iommu group 8
[   33.313967] hns3 0000:7d:00.2: Adding to iommu group 9
[   33.436029] hns3 0000:7d:00.3: Adding to iommu group 10
[   33.555935] hns3 0000:bd:00.0: Adding to iommu group 11
[   35.143851] pcieport 0000:74:00.0: Adding to iommu group 12
[   35.150736] pcieport 0000:80:00.0: Adding to iommu group 13
[   35.158910] pcieport 0000:80:08.0: Adding to iommu group 14
[   35.166860] pcieport 0000:80:0c.0: Adding to iommu group 15
[   35.174813] pcieport 0000:80:10.0: Adding to iommu group 16
[   35.182854] pcieport 0000:bc:00.0: Adding to iommu group 17
[   35.189702] pcieport 0000:b4:00.0: Adding to iommu group 18
[   35.196445] hisi_sas_v3_hw 0000:74:02.0: Adding to iommu group 19
[   39.410693] ahci 0000:74:03.0: Adding to iommu group 20
root@(none)$ lsmod
     Not tainted
arm_smmu_v3 40960 21 - Live 0xffff800008c60000

--- Start removing devices ---

root@(none)$  echo 1 > ./sys/devices/pci0000:00/0000:00:00.0/remove
[   55.567808] pci_bus 0000:01: busn_res: [bus 01] is released
[   55.573514] pci 0000:00:00.0: Removing from iommu group 0
root@(none)$  echo 1 > ./sys/devices/pci0000:00/0000:00:04.0/remove
[   61.767425] pci_bus 0000:02: busn_res: [bus 02] is released
[   61.773132] pci 0000:00:04.0: Removing from iommu group 1
root@(none)$ echo 1 > ./sys/devices/pci0000:00/0000:00:04.0/remove
sh: ./sys/devices/pci0000:00/0000:00:04.0/remove: No such file or directory
root@(none)$  echo 1 > ./sys/devices/pci0000:00/0000:00:08.0/remove
[   75.635417] pci_bus 0000:03: busn_res: [bus 03] is released
[   75.641124] pci 0000:00:08.0: Removing from iommu group 2
root@(none)$ echo 1 > ./sys/devices/pci0000:00/0000:00:0c.0/remove
[   81.587419] pci_bus 0000:04: busn_res: [bus 04] is released
[   81.593110] pci 0000:00:0c.0: Removing from iommu group 3
root@(none)$  echo 1 > ./sys/devices/pci0000:00/0000:00:10.0/remove
[   85.607605] pci_bus 0000:05: busn_res: [bus 05] is released
[   85.613300] pci 0000:00:10.0: Removing from iommu group 4
root@(none)$ echo 1 > ./sys/devices/pci0000:00/0000:00:12.0/remove
[   92.731421] pci_bus 0000:06: busn_res: [bus 06] is released
[   92.737125] pci 0000:00:12.0: Removing from iommu group 5
root@(none)$ echo 1 > ./sys/devices/pci0000:7c/0000:7c:00.0/remove
[  102.286726] pci 0000:7d:00.0: Removing from iommu group 7
[  102.294157] pci 0000:7d:00.1: Removing from iommu group 8
[  102.301634] pci 0000:7d:00.2: Removing from iommu group 9
[  102.308973] pci 0000:7d:00.3: Removing from iommu group 10
[  102.316578] pci_bus 0000:7d: busn_res: [bus 7d] is released
[  102.322282] pci 0000:7c:00.0: Removing from iommu group 6
root@(none)$ echo 1 > ./sys/devices/pci0000:74/0000:74:00.0/remove
[  108.047590] pci_bus 0000:75: busn_res: [bus 75] is released
[  108.053278] pci 0000:74:00.0: Removing from iommu group 12
root@(none)$ echo 1 > ./sys/devices/pci0000:80/0000:80:00.0/remove
[  112.283590] pci_bus 0000:81: busn_res: [bus 81] is released
[  112.289293] pci 0000:80:00.0: Removing from iommu group 13
root@(none)$ echo 1 > ./sys/devices/pci0000:80/0000:80:0c.0/remove
[  117.975427] pci_bus 0000:83: busn_res: [bus 83] is released
[  117.981126] pci 0000:80:0c.0: Removing from iommu group 15
root@(none)$ echo 1 > ./sys/devices/pci0000:80/0000:80:10.0/remove
[  123.935417] pci_bus 0000:84: busn_res: [bus 84] is released
[  123.941115] pci 0000:80:10.0: Removing from iommu group 16
root@(none)$ echo 1 > ./sys/devices/pci0000:bc/0000:bc:00.0/remove
[  129.855959] pci 0000:bd:00.0: Removing from iommu group 11
[  129.863316] pci_bus 0000:bd: busn_res: [bus bd] is released
[  129.869345] pci 0000:bc:00.0: Removing from iommu group 17
root@(none)$ echo 1 > ./sys/devices/pci0000:b4/0000:b4:00.0/remove
[  137.235535] pci_bus 0000:b5: busn_res: [bus b5] is released
[  137.241222] pci 0000:b4:00.0: Removing from iommu group 18
root@(none)$ echo 1 > ./sys/devices/pci0000:74/0000:74:02.0/remove
[  145.984795] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  145.991313] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  145.997853] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  146.004364] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  146.010921] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  146.017455] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  146.023921] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  146.029948] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] is gone
[  146.036768] sd 0:0:3:0: [sdd] Synchronizing SCSI cache
[  146.112158] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] is gone
[  146.118977] sd 0:0:2:0: [sdc] Synchronizing SCSI cache
[  146.124208] sd 0:0:2:0: [sdc] Stopping disk
[  146.660144] hisi_sas_v3_hw 0000:74:02.0: dev[4:5] is gone
[  146.666978] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[  146.744078] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] is gone
[  146.750762] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[  146.808170] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] is gone
[  146.816723] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] is gone
[  146.887746] pci 0000:74:02.0: Removing from iommu group 19
root@(none)$ echo 1 > ./sys/bus/pci/devices/0000:74:03.0/remove
[  156.443637] pci 0000:74:03.0: Removing from iommu group 20
root@(none)$ echo 1 > ./sys/bus/pci/devices/0000:80:08.0/remove
[  165.199447] pci_bus 0000:82: busn_res: [bus 82] is released
[  165.205166] pci 0000:80:08.0: Removing from iommu group 14

--- All devices removed, so can unload ko ---

root@(none)$ lsmod
     Not tainted
arm_smmu_v3 40960 0 - Live 0xffff800008c60000
root@(none)$ rmmod arm_smmu_v3
root@(none)$ lsmod
     Not tainted

--- Bring it back to life ---

root@(none)$ echo 1 > sys/bus/pci/rescan
[  257.639436] pci 0000:00:00.0: [19e5:a120] type 01 class 0x060400
[  257.645608] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.652419] pci 0000:00:04.0: [19e5:a120] type 01 class 0x060400
[  257.658505] pci 0000:00:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.665228] pci 0000:00:08.0: [19e5:a120] type 01 class 0x060400
[  257.671316] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.678030] pci 0000:00:0c.0: [19e5:a120] type 01 class 0x060400
[  257.684113] pci 0000:00:0c.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.690833] pci 0000:00:10.0: [19e5:a120] type 01 class 0x060400
[  257.696915] pci 0000:00:10.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.703623] pci 0000:00:12.0: [19e5:a120] type 01 class 0x060400
[  257.709704] pci 0000:00:12.0: PME# supported from D0 D1 D2 D3hot D3cold
[  257.716473] pci 0000:01:00.0: [8086:10fb] type 00 class 0x020000
[  257.722489] pci 0000:01:00.0: reg 0x10: [mem 
0x80000000000-0x8000007ffff 64bit pref]
[  257.730224] pci 0000:01:00.0: reg 0x18: [io  0x1000-0x101f]
[  257.735796] pci 0000:01:00.0: reg 0x20: [mem 
0x80000100000-0x80000103fff 64bit pref]
[  257.743531] pci 0000:01:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[  257.750311] pci 0000:01:00.0: PME# supported from D0 D3hot
[  257.755813] pci 0000:01:00.0: reg 0x184: [mem 
0x80000104000-0x80000107fff 64bit pref]
[  257.763636] pci 0000:01:00.0: VF(n) BAR0 space: [mem 
0x80000104000-0x80000203fff 64bit pref] (contains BAR0 for 64 VFs)
[  257.774416] pci 0000:01:00.0: reg 0x190: [mem 
0x80000204000-0x80000207fff 64bit pref]
[  257.782234] pci 0000:01:00.0: VF(n) BAR3 space: [mem 
0x80000204000-0x80000303fff 64bit pref] (contains BAR3 for 64 VFs)
[  257.793278] pci 0000:01:00.1: [8086:10fb] type 00 class 0x020000
[  257.799293] pci 0000:01:00.1: reg 0x10: [mem 
0x80000080000-0x800000fffff 64bit pref]
[  257.807028] pci 0000:01:00.1: reg 0x18: [io  0x1020-0x103f]
[  257.812600] pci 0000:01:00.1: reg 0x20: [mem 
0x80000304000-0x80000307fff 64bit pref]
[  257.820335] pci 0000:01:00.1: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[  257.827114] pci 0000:01:00.1: PME# supported from D0 D3hot
[  257.832612] pci 0000:01:00.1: reg 0x184: [mem 
0x80000308000-0x8000030bfff 64bit pref]
[  257.840431] pci 0000:01:00.1: VF(n) BAR0 space: [mem 
0x80000308000-0x80000407fff 64bit pref] (contains BAR0 for 64 VFs)
[  257.851211] pci 0000:01:00.1: reg 0x190: [mem 
0x80000408000-0x8000040bfff 64bit pref]
[  257.859030] pci 0000:01:00.1: VF(n) BAR3 space: [mem 
0x80000408000-0x80000507fff 64bit pref] (contains BAR3 for 64 VFs)
[  257.870270] pci 0000:05:00.0: [19e5:1711] type 00 class 0x030000
[  257.876296] pci 0000:05:00.0: reg 0x10: [mem 0xe0000000-0xe1ffffff pref]
[  257.882997] pci 0000:05:00.0: reg 0x14: [mem 0xe2000000-0xe21fffff]
[  257.889436] pci 0000:05:00.0: supports D1
[  257.893435] pci 0000:05:00.0: PME# supported from D0 D1 D3hot
[  257.899282] pci 0000:05:00.0: vgaarb: VGA device added: 
decodes=io+mem,owns=none,locks=none
[  257.907716] pci 0000:00:00.0: bridge window [mem 
0x00100000-0x002fffff 64bit pref] to [bus 01] add_size 400000 add_align 
100000
[  257.919192] pci 0000:00:10.0: BAR 14: assigned [mem 
0xe0000000-0xe2ffffff]
[  257.926056] pci 0000:00:00.0: BAR 14: assigned [mem 
0xe3000000-0xe30fffff]
[  257.932920] pci 0000:00:00.0: BAR 15: assigned [mem 
0x80000000000-0x800005fffff 64bit pref]
[  257.941259] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[  257.947438] pci 0000:01:00.0: BAR 0: assigned [mem 
0x80000000000-0x8000007ffff 64bit pref]
[  257.955697] pci 0000:01:00.0: BAR 6: assigned [mem 
0xe3000000-0xe307ffff pref]
[  257.962911] pci 0000:01:00.1: BAR 0: assigned [mem 
0x80000080000-0x800000fffff 64bit pref]
[  257.971168] pci 0000:01:00.1: BAR 6: assigned [mem 
0xe3080000-0xe30fffff pref]
[  257.978378] pci 0000:01:00.0: BAR 4: assigned [mem 
0x80000100000-0x80000103fff 64bit pref]
[  257.986636] pci 0000:01:00.0: BAR 7: assigned [mem 
0x80000104000-0x80000203fff 64bit pref]
[  257.994891] pci 0000:01:00.0: BAR 10: assigned [mem 
0x80000204000-0x80000303fff 64bit pref]
[  258.003232] pci 0000:01:00.1: BAR 4: assigned [mem 
0x80000304000-0x80000307fff 64bit pref]
[  258.011490] pci 0000:01:00.1: BAR 7: assigned [mem 
0x80000308000-0x80000407fff 64bit pref]
[  258.019744] pci 0000:01:00.1: BAR 10: assigned [mem 
0x80000408000-0x80000507fff 64bit pref]
[  258.028085] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[  258.034169] pci 0000:01:00.1: BAR 2: assigned [io  0x1020-0x103f]
[  258.040254] pci 0000:00:00.0: PCI bridge to [bus 01]
[  258.045208] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[  258.051291] pci 0000:00:00.0:   bridge window [mem 0xe3000000-0xe30fffff]
[  258.058068] pci 0000:00:00.0:   bridge window [mem 
0x80000000000-0x800005fffff 64bit pref]
[  258.066326] pci 0000:00:04.0: PCI bridge to [bus 02]
[  258.071284] pci 0000:00:08.0: PCI bridge to [bus 03]
[  258.076241] pci 0000:00:0c.0: PCI bridge to [bus 04]
[  258.081199] pci 0000:05:00.0: BAR 0: assigned [mem 
0xe0000000-0xe1ffffff pref]
[  258.088413] pci 0000:05:00.0: BAR 1: assigned [mem 0xe2000000-0xe21fffff]
[  258.095193] pci 0000:00:10.0: PCI bridge to [bus 05]
[  258.100148] pci 0000:00:10.0:   bridge window [mem 0xe0000000-0xe2ffffff]
[  258.106927] pci 0000:00:12.0: PCI bridge to [bus 06]
[  258.112271] pci 0000:7c:00.0: [19e5:a121] type 01 class 0x060400
[  258.118281] pci 0000:7c:00.0: enabling Extended Tags
[  258.123394] pci 0000:7d:00.0: [19e5:a222] type 00 class 0x020000
[  258.129397] pci 0000:7d:00.0: reg 0x10: [mem 0x122000000-0x12200ffff 
64bit pref]
[  258.136784] pci 0000:7d:00.0: reg 0x18: [mem 0x120000000-0x1200fffff 
64bit pref]
[  258.144202] pci 0000:7d:00.0: reg 0x224: [mem 0x122010000-0x12201ffff 
64bit pref]
[  258.151674] pci 0000:7d:00.0: VF(n) BAR0 space: [mem 
0x122010000-0x1220effff 64bit pref] (contains BAR0 for 14 VFs)
[  258.162100] pci 0000:7d:00.0: reg 0x22c: [mem 0x120100000-0x1201fffff 
64bit pref]
[  258.169575] pci 0000:7d:00.0: VF(n) BAR2 space: [mem 
0x120100000-0x120efffff 64bit pref] (contains BAR2 for 14 VFs)
[  258.180089] pci 0000:7d:00.1: [19e5:a222] type 00 class 0x020000
[  258.186091] pci 0000:7d:00.1: reg 0x10: [mem 0x1220f0000-0x1220fffff 
64bit pref]
[  258.193478] pci 0000:7d:00.1: reg 0x18: [mem 0x120f00000-0x120ffffff 
64bit pref]
[  258.200891] pci 0000:7d:00.1: reg 0x224: [mem 0x122100000-0x12210ffff 
64bit pref]
[  258.208362] pci 0000:7d:00.1: VF(n) BAR0 space: [mem 
0x122100000-0x1221dffff 64bit pref] (contains BAR0 for 14 VFs)
[  258.218787] pci 0000:7d:00.1: reg 0x22c: [mem 0x121000000-0x1210fffff 
64bit pref]
[  258.226257] pci 0000:7d:00.1: VF(n) BAR2 space: [mem 
0x121000000-0x121dfffff 64bit pref] (contains BAR2 for 14 VFs)
[  258.236777] pci 0000:7d:00.2: [19e5:a222] type 00 class 0x020000
[  258.242780] pci 0000:7d:00.2: reg 0x10: [mem 0x1221e0000-0x1221effff 
64bit pref]
[  258.250167] pci 0000:7d:00.2: reg 0x18: [mem 0x121e00000-0x121efffff 
64bit pref]
[  258.257638] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  258.263644] pci 0000:7d:00.3: reg 0x10: [mem 0x1221f0000-0x1221fffff 
64bit pref]
[  258.271031] pci 0000:7d:00.3: reg 0x18: [mem 0x121f00000-0x121ffffff 
64bit pref]
[  258.278506] pci 0000:7c:00.0: bridge window [mem 
0x00100000-0x005fffff 64bit pref] to [bus 7d] add_size 1d00000 add_align 
100000
[  258.290060] pci 0000:7c:00.0: BAR 15: assigned [mem 
0x120000000-0x1221fffff 64bit pref]
[  258.298062] pci 0000:7d:00.0: BAR 2: assigned [mem 
0x120000000-0x1200fffff 64bit pref]
[  258.305969] pci 0000:7d:00.0: BAR 9: assigned [mem 
0x120100000-0x120efffff 64bit pref]
[  258.313874] pci 0000:7d:00.1: BAR 2: assigned [mem 
0x120f00000-0x120ffffff 64bit pref]
[  258.321781] pci 0000:7d:00.1: BAR 9: assigned [mem 
0x121000000-0x121dfffff 64bit pref]
[  258.329687] pci 0000:7d:00.2: BAR 2: assigned [mem 
0x121e00000-0x121efffff 64bit pref]
[  258.337594] pci 0000:7d:00.3: BAR 2: assigned [mem 
0x121f00000-0x121ffffff 64bit pref]
[  258.345500] pci 0000:7d:00.0: BAR 0: assigned [mem 
0x122000000-0x12200ffff 64bit pref]
[  258.353407] pci 0000:7d:00.0: BAR 7: assigned [mem 
0x122010000-0x1220effff 64bit pref]
[  258.361313] pci 0000:7d:00.1: BAR 0: assigned [mem 
0x1220f0000-0x1220fffff 64bit pref]
[  258.369222] pci 0000:7d:00.1: BAR 7: assigned [mem 
0x122100000-0x1221dffff 64bit pref]
[  258.377128] pci 0000:7d:00.2: BAR 0: assigned [mem 
0x1221e0000-0x1221effff 64bit pref]
[  258.385035] pci 0000:7d:00.3: BAR 0: assigned [mem 
0x1221f0000-0x1221fffff 64bit pref]
[  258.392948] pci 0000:7c:00.0: PCI bridge to [bus 7d]
[  258.397903] pci 0000:7c:00.0:   bridge window [mem 
0x120000000-0x1221fffff 64bit pref]
[  258.405964] pci 0000:74:00.0: [19e5:a121] type 01 class 0x060400
[  258.411974] pci 0000:74:00.0: enabling Extended Tags
[  258.417017] pci 0000:74:02.0: [19e5:a230] type 00 class 0x010700
[  258.423029] pci 0000:74:02.0: reg 0x24: [mem 0xa2000000-0xa2007fff]
[  258.429477] pci 0000:74:03.0: [19e5:a235] type 00 class 0x010601
[  258.435489] pci 0000:74:03.0: reg 0x24: [mem 0xa2008000-0xa2008fff]
[  258.441881] pci 0000:75:00.0: [19e5:a250] type 00 class 0x120000
[  258.447887] pci 0000:75:00.0: reg 0x18: [mem 0x144000000-0x1443fffff 
64bit pref]
[  258.455307] pci 0000:75:00.0: reg 0x22c: [mem 0x144400000-0x14440ffff 
64bit pref]
[  258.462779] pci 0000:75:00.0: VF(n) BAR2 space: [mem 
0x144400000-0x1447effff 64bit pref] (contains BAR2 for 63 VFs)
[  258.473330] pci 0000:74:00.0: bridge window [mem 
0x00400000-0x007fffff 64bit pref] to [bus 75] add_size 400000 add_align 
400000
[  258.484800] pci 0000:74:00.0: BAR 15: assigned [mem 
0x144000000-0x1447fffff 64bit pref]
[  258.492795] pci 0000:74:02.0: BAR 5: assigned [mem 0xa2000000-0xa2007fff]
[  258.499576] pci 0000:74:03.0: BAR 5: assigned [mem 0xa2008000-0xa2008fff]
[  258.506356] pci 0000:75:00.0: BAR 2: assigned [mem 
0x144000000-0x1443fffff 64bit pref]
[  258.514263] pci 0000:75:00.0: BAR 9: assigned [mem 
0x144400000-0x1447effff 64bit pref]
[  258.522169] pci 0000:74:00.0: PCI bridge to [bus 75]
[  258.527124] pci 0000:74:00.0:   bridge window [mem 
0x144000000-0x1447fffff 64bit pref]
[  258.535182] pci 0000:80:00.0: [19e5:a120] type 01 class 0x060400
[  258.541278] pci 0000:80:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  258.547972] pci 0000:80:08.0: [19e5:a120] type 01 class 0x060400
[  258.554069] pci 0000:80:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[  258.560759] pci 0000:80:0c.0: [19e5:a120] type 01 class 0x060400
[  258.566858] pci 0000:80:0c.0: PME# supported from D0 D1 D2 D3hot D3cold
[  258.573549] pci 0000:80:10.0: [19e5:a120] type 01 class 0x060400
[  258.579644] pci 0000:80:10.0: PME# supported from D0 D1 D2 D3hot D3cold
[  258.586377] pci 0000:81:00.0: [19e5:0123] type 00 class 0x010802
[  258.592387] pci 0000:81:00.0: reg 0x10: [mem 0xf0000000-0xf003ffff 64bit]
[  258.599184] pci 0000:81:00.0: reg 0x30: [mem 0xfffe0000-0xffffffff pref]
[  258.605965] pc[  258.629823] pci 0000:81:00.0: BAR 6: assigned [mem 
0xf0040000-0xf005ffff pref]
[  258.637033] pci 0000:80:00.0: PCI bridge to [bus 81]
[  258.641988] pci 0000:80:00.0:   bridge window [mem 0xf0000000-0xf00fffff]
[  258.648766] pci 0000:80:08.0: PCI bridge to [bus 82]
[  258.653722] pci 0000:80:0c.0: PCI bridge to [bus 83]
[  258.658679] pci 0000:80:10.0: PCI bridge to [bus 84]
[  258.663865] pci 0000:bc:00.0: [19e5:a121] type 01 class 0x060400
[  258.669874] pci 0000:bc:00.0: enabling Extended Tags
[  258.674960] pci 0000:bd:00.0: [19e5:a226] type 00 class 0x020000
[  258.680962] pci 0000:bd:00.0: reg 0x10: [mem 
0x400122000000-0x40012200ffff 64bit pref]
[  258.688869] pci 0000:bd:00.0: reg 0x18: [mem 
0x400120000000-0x4001200fffff 64bit pref]
[  258.696799] pci 0000:bd:00.0: reg 0x224: [mem 
0x400122010000-0x40012201ffff 64bit pref]
[  258.704791] pci 0000:bd:00.0: VF(n) BAR0 space: [mem 
0x400122010000-0x4001221fffff 64bit pref] (contains BAR0 for 31 VFs)
[  258.715736] pci 0000:bd:00.0: reg 0x22c: [mem 
0x400120100000-0x4001201fffff 64bit pref]
[  258.723728] pci 0000:bd:00.0: VF(n) BAR2 space: [mem 
0x400120100000-0x400121ffffff 64bit pref] (contains BAR2 for 31 VFs)
[  258.734762] pci 0000:bc:00.0: bridge window [mem 
0x00100000-0x002fffff 64bit pref] to [bus bd] add_size 2000000 add_align 
100000
[  258.746317] pci 0000:bc:00.0: BAR 15: assigned [mem 
0x400120000000-0x4001221fffff 64bit pref]
[  258.754831] pci 0000:bd:00.0: BAR 2: assigned [mem 
0x400120000000-0x4001200fffff 64bit pref]
[  258.763258] pci 0000:bd:00.0: BAR 9: assigned [mem 
0x400120100000-0x400121ffffff 64bit pref]
[  258.771688] pci 0000:bd:00.0: BAR 0: assigned [mem 
0x400122000000-0x40012200ffff 64bit pref]
[  258.780115] pci 0000:bd:00.0: BAR 7: assigned [mem 
0x400122010000-0x4001221fffff 64bit pref]
[  258.788542] pci 0000:bc:00.0: PCI bridge to [bus bd]
[  258.793497] pci 0000:bc:00.0:   bridge window [mem 
0x400120000000-0x4001221fffff 64bit pref]
[  258.801992] pci 0000:b4:00.0: [19e5:a121] type 01 class 0x060400
[  258.807998] pci 0000:b4:00.0: enabling Extended Tags
[  258.813076] pci 0000:b5:00.0: [19e5:a250] type 00 class 0x120000
[  258.819080] pci 0000:b5:00.0: reg 0x18: [mem 
0x400144000000-0x4001443fffff 64bit pref]
[  258.827012] pci 0000:b5:00.0: reg 0x22c: [mem 
0x400144400000-0x40014440ffff 64bit pref]
[  258.835003] pci 0000:b5:00.0: VF(n) BAR2 space: [mem 
0x400144400000-0x4001447effff 64bit pref] (contains BAR2 for 63 VFs)
[  258.846053] pci 0000:b4:00.0: bridge window [mem 
0x00400000-0x007fffff 64bit pref] to [bus b5] add_size 400000 add_align 
400000
[  258.857521] pci 0000:b4:00.0: BAR 15: assigned [mem 
0x400144000000-0x4001447fffff 64bit pref]
[  258.866034] pci 0000:b5:00.0: BAR 2: assigned [mem 
0x400144000000-0x4001443fffff 64bit pref]
[  258.874466] pci 0000:b5:00.0: BAR 9: assigned [mem 
0x400144400000-0x4001447effff 64bit pref]

--- Insert ko again ---

root@(none)$ !1
insmod lib/modules/arm-smmu-v3.ko
[  282.751387] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0
[  282.756992] arm-smmu-v3 arm-smmu-v3.0.auto: IRQ combined not found
[  282.763163] arm-smmu-v3 arm-smmu-v3.0.auto: IRQ eventq not found
[  282.769161] arm-smmu-v3 arm-smmu-v3.0.auto: IRQ priq not found
[  282.774986] arm-smmu-v3 arm-smmu-v3.0.auto: IRQ gerror not found
[  282.780989] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  282.789378] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 65536 entries 
for cmdq
[  282.796648] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 32768 entries 
for evtq
[  282.804513] pcieport 0000:00:00.0: Adding to iommu group 0
[  282.807369] arm-smmu-v3 arm-smmu-v3.1.auto: option mask 0x0
[  282.812284] pcieport 0000:00:04.0: Adding to iommu group 1
[  282.815609] arm-smmu-v3 arm-smmu-v3.1.auto: IRQ combined not found
[  282.823242] pcieport 0000:00:08.0: Adding to iommu group 2
[  282.827210] arm-smmu-v3 arm-smmu-v3.1.auto: IRQ eventq not found
[  282.827212] arm-smmu-v3 arm-smmu-v3.1.auto: IRQ priq not found
[  282.827213] arm-smmu-v3 arm-smmu-v3.1.auto: IRQ gerror not found
[  282.827216] arm-smmu-v3 arm-smmu-v3.1.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  282.827364] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 65536 entries 
for cmdq
[  282.834808] pcieport 0000:00:0c.0: Adding to iommu group 3
[  282.838981] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 32768 entries 
for evtq
[  282.846646] pcieport 0000:00:10.0: Adding to iommu group 4
[  282.850726] arm-smmu-v3 arm-smmu-v3.2.auto: option mask 0x0
[  282.860722] pcieport 0000:00:12.0: Adding to iommu group 5
[  282.865657] arm-smmu-v3 arm-smmu-v3.2.auto: IRQ combined not found
[  282.873349] pcieport 0000:7c:00.0: Adding to iommu group 6
[  282.878160] arm-smmu-v3 arm-smmu-v3.2.auto: IRQ eventq not found
[  282.878162] arm-smmu-v3 arm-smmu-v3.2.auto: IRQ priq not found
[  282.878163] arm-smmu-v3 arm-smmu-v3.2.auto: IRQ gerror not found
[  282.878166] arm-smmu-v3 arm-smmu-v3.2.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  282.878290] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 65536 entries 
for cmdq
[  282.885133] hns3 0000:7d:00.0: Adding to iommu group 7
[  282.889379] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 32768 entries 
for evtq
[  282.895943] hns3 0000:7d:00.0: The firmware version is 176.98.1.48
[  282.901035] arm-smmu-v3 arm-smmu-v3.3.auto: option mask 0x0
[  282.910767] hns3 0000:7d:00.0: Firmware compatible features not 
enabled(-95).
[  282.912348] arm-smmu-v3 arm-smmu-v3.3.auto: IRQ combined not found
[  282.976427] arm-smmu-v3 arm-smmu-v3.3.auto: IRQ eventq not found
[  282.982421] arm-smmu-v3 arm-smmu-v3.3.auto: IRQ priq not found
[  282.988241] arm-smmu-v3 arm-smmu-v3.3.auto: IRQ gerror not found
[  282.994236] arm-smmu-v3 arm-smmu-v3.3.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  283.002437] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 65536 entries 
for cmdq
[  283.009756] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 32768 entries 
for evtq
[  283.016982] arm-smmu-v3 arm-smmu-v3.4.auto: option mask 0x0
[  283.022559] arm-smmu-v3 arm-smmu-v3.4.auto: IRQ combined not found
[  283.028728] arm-smmu-v3 arm-smmu-v3.4.auto: IRQ eventq not found
[  283.034722] arm-smmu-v3 arm-smmu-v3.4.auto: IRQ priq not found
[  283.040542] arm-smmu-v3 arm-smmu-v3.4.auto: IRQ gerror not found
[  283.046537] arm-smmu-v3 arm-smmu-v3.4.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  283.054732] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 65536 entries 
for cmdq
[  283.061936] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 32768 entries 
for evtq
[  283.069129] arm-smmu-v3 arm-smmu-v3.5.auto: option mask 0x0
[  283.074703] arm-smmu-v3 arm-smmu-v3.5.auto: IRQ combined not found
[  283.080871] arm-smmu-v3 arm-smmu-v3.5.auto: IRQ eventq not found
[  283.086867] arm-smmu-v3 arm-smmu-v3.5.auto: IRQ priq not found
[  283.092687] arm-smmu-v3 arm-smmu-v3.5.auto: IRQ gerror not found
[  283.098682] arm-smmu-v3 arm-smmu-v3.5.auto: ias 48-bit, oas 48-bit 
(features 0x00000fef)
[  283.106878] arm-smmu-v3 arm-smmu-v3.5.auto: allocated 65536 entries 
for cmdq
[  283.114191] arm-smmu-v3 arm-smmu-v3.5.auto: allocated 32768 entries 
for evtq
root@(none)$ [  284.294657] hns3 0000:7d:00.0: hclge driver 
initialization finished.
[  284.326277] hns3 0000:7d:00.1: Adding to iommu group 8
[  284.332691] hns3 0000:7d:00.1: The firmware version is 176.98.1.48
[  284.343297] hns3 0000:7d:00.1: Firmware compatible features not 
enabled(-95).
[  285.706654] hns3 0000:7d:00.1: hclge driver initialization finished.
[  285.737930] hns3 0000:7d:00.2: Adding to iommu group 9
[  285.744325] hns3 0000:7d:00.2: The firmware version is 176.98.1.48
[  285.754931] hns3 0000:7d:00.2: Firmware compatible features not 
enabled(-95).
[  285.765990] libphy: hisilicon MII bus: probed
[  285.835882] hns3 0000:7d:00.2: hclge driver initialization finished.
[  285.848083] Generic PHY mii-0000:7d:00.2:00: attached PHY driver 
[Generic PHY] (mii_bus:phy_addr=mii-0000:7d:00.2:00, irq=POLL)
[  285.860042] hns3 0000:7d:00.3: Adding to iommu group 10
[  285.866529] hns3 0000:7d:00.3: The firmware version is 176.98.1.48
[  285.877130] hns3 0000:7d:00.3: Firmware compatible features not 
enabled(-95).
[  285.886386] libphy: hisilicon MII bus: probed
[  285.955696] hns3 0000:7d:00.3: hclge driver initialization finished.
[  285.967919] Generic PHY mii-0000:7d:00.3:01: attached PHY driver 
[Generic PHY] (mii_bus:phy_addr=mii-0000:7d:00.3:01, irq=POLL)
[  285.979982] pcieport 0000:74:00.0: Adding to iommu group 11
[  285.986863] hisi_sas_v3_hw 0000:74:02.0: Adding to iommu group 12
[  286.008195] scsi host0: hisi_sas_v3_hw
[  287.298893] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11
[  287.304978] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11
[  287.311061] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11
[  287.311066] hisi_sas_v3_hw 0000:74:02.0: phyup: phy3 link_rate=11
[  287.323224] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11
[  287.323230] hisi_sas_v3_hw 0000:74:02.0: phyup: phy5 link_rate=11
[  287.323235] hisi_sas_v3_hw 0000:74:02.0: phyup: phy6 link_rate=11
[  287.323268] hisi_sas_v3_hw 0000:74:02.0: phyup: phy7 link_rate=11
[  287.329634] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] found
[  287.356200] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] found
[  287.361593] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] found
[  287.367180] hisi_sas_v3_hw 0000:74:02.0: dev[4:5] found
[  287.532058] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] found
[  287.537456] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] found
[  287.542868] sas: Enter sas_scsi_recover_host busy: 0 failed: 0
[  289.729042] ata4.00: ATA-8: SAMSUNG HM320JI, 2SS00_01, max UDMA7
[  289.735039] ata4.00: 625142448 sectors, multi 0: LBA48 NCQ (depth 32)
[  289.747065] ata4.00: configured for UDMA/133
[  289.751338] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 
tries: 1
[  289.762761] scsi 0:0:0:0: Direct-Access     SEAGATE  ST2000NM0045 
N004 PQ: 0 ANSI: 6
[  289.774757] scsi 0:0:1:0: Direct-Access     SEAGATE  ST2000NM0045 
N004 PQ: 0 ANSI: 6
[  289.775005] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: 
(2.00 TB/1.82 TiB)
[  289.785688] scsi 0:0:2:0: Direct-Access     ATA      SAMSUNG HM320JI 
0_01 PQ: 0 ANSI: 5
[  289.786940] sd 0:0:1:0: [sdb] 3907029168 512-byte logical blocks: 
(2.00 TB/1.82 TiB)
[  289.787612] sd 0:0:1:0: [sdb] Write Protect is off
[  289.788816] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[  289.791242] sd 0:0:0:0: [sda] Write Protect is off
[  289.799026] sd 0:0:2:0: [sdc] 625142448 512-byte logical blocks: (320 
GB/298 GiB)
[  289.799865] scsi 0:0:3:0: Direct-Access     SEAGATE  ST1000NM0023 
0006 PQ: 0 ANSI: 6
[  289.807642] sd 0:0:0:0: [sda] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[  289.811219] sd 0:0:2:0: [sdc] Write Protect is off
[  289.828112]  sdb: sdb1
[  289.831969] sd 0:0:2:0: [sdc] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[  289.832627] scsi 0:0:4:0: Enclosure         HUAWEI   Expander 12Gx16 
128  PQ: 0 ANSI: 6
[  289.832952] sd 0:0:3:0: [sdd] 1953525168 512-byte logical blocks: 
(1.00 TB/932 GiB)
[  289.833296] sd 0:0:3:0: [sdd] Write Protect is off
[  289.833942] sd 0:0:3:0: [sdd] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[  289.834261] sas: ex 500e004aaaaaaa1f phy08 change count has changed
[  289.837361] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.838882] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.840403] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.845988] sd 0:0:1:0: [sdb] Attached SCSI disk
[  289.922155] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.929063] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.935167]  sdd: sdd1 sdd2
[  289.936001] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.937042]  sda: sda1
[  289.942356] sd 0:0:3:0: [sdd] Attached SCSI disk
[  289.943134] sd 0:0:0:0: [sda] Attached SCSI disk
[  289.956420] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.963228] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.970139] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.977068] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.983892] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.990796] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  289.997676] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.004530] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.011413] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.018326] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.025153] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.032032] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.038952] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.045032] random: crng init done
[  290.049356] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.056228] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.063116] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.070028] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.076864] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.083766] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.090642] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.097469] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.104334] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.111231] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.118077] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.124972] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.131892] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.138701] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.145628] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.152511] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.159352] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.166244] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.173134] sas: ex 500e004aaaaaaa1f phys DID NOT change
[  290.178616] ahci 0000:74:03.0: Adding to iommu group 13
[  290.185204] ahci 0000:74:03.0: SSS flag set, parallel bus scan disabled
[  290.191824] ahci 0000:74:03.0: AHCI 0001.0300 32 slots 2 ports 6 Gbps 
0x3 impl SATA mode
[  290.199903] ahci 0000:74:03.0: flags: 64bit ncq sntf stag pm led clo 
only pmp fbs slum part ccc sxs boh
[  290.209768] scsi host1: ahci
[  290.212787] scsi host2: ahci
[  290.215729] ata5: SATA max UDMA/133 abar m4096@0xa2008000 port 
0xa2008100 irq 660
[  290.223201] ata6: SATA max UDMA/133 abar m4096@0xa2008000 port 
0xa2008180 irq 661
[  290.230933] pcieport 0000:80:00.0: Adding to iommu group 14
[  290.239292] pcieport 0000:80:08.0: Adding to iommu group 15
[  290.247396] pcieport 0000:80:0c.0: Adding to iommu group 16
[  290.255453] pcieport 0000:80:10.0: Adding to iommu group 17
[  290.263638] pcieport 0000:bc:00.0: Adding to iommu group 18
[  290.270632] hns3 0000:bd:00.0: Adding to iommu group 19
[  290.277182] hns3 0000:bd:00.0: The firmware version is 176.98.1.48
[  290.287847] hns3 0000:bd:00.0: Firmware compatible features not 
enabled(-95).
[  290.545421] ata5: SATA link down (SStatus 0 SControl 300)
[  291.699895] hns3 0000:bd:00.0: hclge driver initialization finished.
[  291.736307] pcieport 0000:b4:00.0: Adding to iommu group 20
[  292.155588]  sdc: sdc1 sdc2 sdc3
[  292.160569] sd 0:0:2:0: [sdc] Attached SCSI disk
[  292.477420] ata6: SATA link down (SStatus 0 SControl 300)


John
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 10/14] iommu/arm-smmu-v3: Unregister IOMMU and bus ops on device removal

[Will Deacon]

When removing the SMMUv3 driver, we need to clear any state that we
registered during probe. This includes our bus ops, sysfs entries and
the IOMMU device registered for early firmware probing of masters.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu-v3.c | 64 +++++++++++++++++++++++++------------
 1 file changed, 43 insertions(+), 21 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 3fd75abce3bb..0e7a135efdfe 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -3565,6 +3565,45 @@ static unsigned long arm_smmu_resource_size(struct arm_smmu_device *smmu)
 		return SZ_128K;
 }
 
+static int arm_smmu_set_bus_ops(struct iommu_ops *ops)
+{
+	int err;
+
+#ifdef CONFIG_PCI
+	if (pci_bus_type.iommu_ops != ops) {
+		if (ops)
+			pci_request_acs();
+		err = bus_set_iommu(&pci_bus_type, ops);
+		if (err)
+			return err;
+	}
+#endif
+#ifdef CONFIG_ARM_AMBA
+	if (amba_bustype.iommu_ops != ops) {
+		err = bus_set_iommu(&amba_bustype, ops);
+		if (err)
+			goto err_reset_pci_ops;
+	}
+#endif
+	if (platform_bus_type.iommu_ops != ops) {
+		err = bus_set_iommu(&platform_bus_type, ops);
+		if (err)
+			goto err_reset_amba_ops;
+	}
+
+	return 0;
+
+err_reset_amba_ops:
+#ifdef CONFIG_ARM_AMBA
+	bus_set_iommu(&amba_bustype, NULL);
+#endif
+err_reset_pci_ops: __maybe_unused;
+#ifdef CONFIG_PCI
+	bus_set_iommu(&pci_bus_type, NULL);
+#endif
+	return err;
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	int irq, ret;
@@ -3655,33 +3694,16 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		return ret;
 	}
 
-#ifdef CONFIG_PCI
-	if (pci_bus_type.iommu_ops != &arm_smmu_ops) {
-		pci_request_acs();
-		ret = bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-		if (ret)
-			return ret;
-	}
-#endif
-#ifdef CONFIG_ARM_AMBA
-	if (amba_bustype.iommu_ops != &arm_smmu_ops) {
-		ret = bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-		if (ret)
-			return ret;
-	}
-#endif
-	if (platform_bus_type.iommu_ops != &arm_smmu_ops) {
-		ret = bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-		if (ret)
-			return ret;
-	}
-	return 0;
+	return arm_smmu_set_bus_ops(&arm_smmu_ops);
 }
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
 
+	arm_smmu_set_bus_ops(NULL);
+	iommu_device_unregister(&smmu->iommu);
+	iommu_device_sysfs_remove(&smmu->iommu);
 	arm_smmu_device_disable(smmu);
 
 	return 0;
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 11/14] iommu/arm-smmu-v3: Allow building as a module

[Will Deacon]

By removing the redundant call to 'pci_request_acs()' we can allow the
ARM SMMUv3 driver to be built as a module.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/Kconfig       | 2 +-
 drivers/iommu/arm-smmu-v3.c | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index e3842eabcfdd..7583d47fc4d5 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -388,7 +388,7 @@ config ARM_SMMU_DISABLE_BYPASS_BY_DEFAULT
 	  config.
 
 config ARM_SMMU_V3
-	bool "ARM Ltd. System MMU Version 3 (SMMUv3) Support"
+	tristate "ARM Ltd. System MMU Version 3 (SMMUv3) Support"
 	depends on ARM64
 	select IOMMU_API
 	select IOMMU_IO_PGTABLE_LPAE
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 0e7a135efdfe..f79b14f75107 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2733,6 +2733,7 @@ static struct iommu_ops arm_smmu_ops = {
 	.get_resv_regions	= arm_smmu_get_resv_regions,
 	.put_resv_regions	= arm_smmu_put_resv_regions,
 	.pgsize_bitmap		= -1UL, /* Restricted during device attach */
+	.owner			= THIS_MODULE,
 };
 
 /* Probing and initialisation functions */
@@ -3571,8 +3572,6 @@ static int arm_smmu_set_bus_ops(struct iommu_ops *ops)
 
 #ifdef CONFIG_PCI
 	if (pci_bus_type.iommu_ops != ops) {
-		if (ops)
-			pci_request_acs();
 		err = bus_set_iommu(&pci_bus_type, ops);
 		if (err)
 			return err;
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 12/14] iommu/arm-smmu: Unregister IOMMU and bus ops on device removal

[Will Deacon]

When removing the SMMU driver, we need to clear any state that we
registered during probe. This includes our bus ops, sysfs entries and
the IOMMU device registered for early firmware probing of masters.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu.c | 50 ++++++++++++++++++++++++++++++++--------
 1 file changed, 40 insertions(+), 10 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index d6c83bd69555..307026fb58b3 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1976,25 +1976,51 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
-static void arm_smmu_bus_init(void)
+static int arm_smmu_bus_init(struct iommu_ops *ops)
 {
+	int err;
+
 	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+	if (!iommu_present(&platform_bus_type)) {
+		err = bus_set_iommu(&platform_bus_type, ops);
+		if (err)
+			return err;
+	}
 #ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+	if (!iommu_present(&amba_bustype)) {
+		err = bus_set_iommu(&amba_bustype, ops);
+		if (err)
+			goto err_reset_platform_ops;
+	}
 #endif
 #ifdef CONFIG_PCI
 	if (!iommu_present(&pci_bus_type)) {
 		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+		err = bus_set_iommu(&pci_bus_type, ops);
+		if (err)
+			goto err_reset_amba_ops;
 	}
 #endif
 #ifdef CONFIG_FSL_MC_BUS
-	if (!iommu_present(&fsl_mc_bus_type))
-		bus_set_iommu(&fsl_mc_bus_type, &arm_smmu_ops);
+	if (!iommu_present(&fsl_mc_bus_type)) {
+		err = bus_set_iommu(&fsl_mc_bus_type, ops);
+		if (err)
+			goto err_reset_pci_ops;
+	}
+#endif
+	return 0;
+
+err_reset_pci_ops: __maybe_unused;
+#ifdef CONFIG_PCI
+	bus_set_iommu(&pci_bus_type, NULL);
 #endif
+err_reset_amba_ops: __maybe_unused;
+#ifdef CONFIG_ARM_AMBA
+	bus_set_iommu(&amba_bustype, NULL);
+#endif
+err_reset_platform_ops: __maybe_unused;
+	bus_set_iommu(&platform_bus_type, NULL);
+	return err;
 }
 
 static int arm_smmu_device_probe(struct platform_device *pdev)
@@ -2142,7 +2168,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	 * ready to handle default domain setup as soon as any SMMU exists.
 	 */
 	if (!using_legacy_binding)
-		arm_smmu_bus_init();
+		return arm_smmu_bus_init(&arm_smmu_ops);
 
 	return 0;
 }
@@ -2156,7 +2182,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 static int arm_smmu_legacy_bus_init(void)
 {
 	if (using_legacy_binding)
-		arm_smmu_bus_init();
+		return arm_smmu_bus_init(&arm_smmu_ops);
 	return 0;
 }
 device_initcall_sync(arm_smmu_legacy_bus_init);
@@ -2171,6 +2197,10 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
 		dev_err(&pdev->dev, "removing device with active domains!\n");
 
+	arm_smmu_bus_init(NULL);
+	iommu_device_unregister(&smmu->iommu);
+	iommu_device_sysfs_remove(&smmu->iommu);
+
 	arm_smmu_rpm_get(smmu);
 	/* Turn the thing off */
 	arm_smmu_gr0_write(smmu, ARM_SMMU_GR0_sCR0, sCR0_CLIENTPD);
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 13/14] iommu/arm-smmu: Allow building as a module

[Will Deacon]

By conditionally dropping support for the legacy binding and exporting
the newly introduced 'arm_smmu_impl_init()' function we can allow the
ARM SMMU driver to be built as a module.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/Kconfig    | 14 ++++++++++-
 drivers/iommu/Makefile   |  3 ++-
 drivers/iommu/arm-smmu.c | 52 ++++++++++++++++++++++++----------------
 3 files changed, 47 insertions(+), 22 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 7583d47fc4d5..fc55f7ba0d18 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -350,7 +350,7 @@ config SPAPR_TCE_IOMMU
 
 # ARM IOMMU support
 config ARM_SMMU
-	bool "ARM Ltd. System MMU (SMMU) Support"
+	tristate "ARM Ltd. System MMU (SMMU) Support"
 	depends on (ARM64 || ARM) && MMU
 	select IOMMU_API
 	select IOMMU_IO_PGTABLE_LPAE
@@ -362,6 +362,18 @@ config ARM_SMMU
 	  Say Y here if your SoC includes an IOMMU device implementing
 	  the ARM SMMU architecture.
 
+config ARM_SMMU_LEGACY_DT_BINDINGS
+	bool "Support the legacy \"mmu-masters\" devicetree bindings"
+	depends on ARM_SMMU=y && OF
+	help
+	  Support for the badly designed and deprecated "mmu-masters"
+	  devicetree bindings. This allows some DMA masters to attach
+	  to the SMMU but does not provide any support via the DMA API.
+	  If you're lucky, you might be able to get VFIO up and running.
+
+	  If you say Y here then you'll make me very sad. Instead, say N
+	  and move your firmware to the utopian future that was 2016.
+
 config ARM_SMMU_DISABLE_BYPASS_BY_DEFAULT
 	bool "Default to disabling bypass on ARM SMMU v1 and v2"
 	depends on ARM_SMMU
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 4f405f926e73..b52a03d87fc3 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -13,7 +13,8 @@ obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o
 obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o amd_iommu_quirks.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += amd_iommu_debugfs.o
 obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
-obj-$(CONFIG_ARM_SMMU) += arm-smmu.o arm-smmu-impl.o
+obj-$(CONFIG_ARM_SMMU) += arm-smmu-mod.o
+arm-smmu-mod-objs += arm-smmu.o arm-smmu-impl.o
 obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
 obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 307026fb58b3..100ab5b9c255 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -125,6 +125,12 @@ static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
 	return container_of(dom, struct arm_smmu_domain, domain);
 }
 
+static struct platform_driver arm_smmu_driver;
+static struct iommu_ops arm_smmu_ops;
+
+#ifdef CONFIG_ARM_SMMU_LEGACY_DT_BINDINGS
+static int arm_smmu_bus_init(struct iommu_ops *ops);
+
 static struct device_node *dev_get_dev_node(struct device *dev)
 {
 	if (dev_is_pci(dev)) {
@@ -160,9 +166,6 @@ static int __find_legacy_master_phandle(struct device *dev, void *data)
 	return err == -ENOENT ? 0 : err;
 }
 
-static struct platform_driver arm_smmu_driver;
-static struct iommu_ops arm_smmu_ops;
-
 static int arm_smmu_register_legacy_master(struct device *dev,
 					   struct arm_smmu_device **smmu)
 {
@@ -214,6 +217,27 @@ static int arm_smmu_register_legacy_master(struct device *dev,
 	return err;
 }
 
+/*
+ * With the legacy DT binding in play, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		return arm_smmu_bus_init(&arm_smmu_ops);
+	return 0;
+}
+device_initcall_sync(arm_smmu_legacy_bus_init);
+#else
+static int arm_smmu_register_legacy_master(struct device *dev,
+					   struct arm_smmu_device **smmu)
+{
+	return -ENODEV;
+}
+#endif /* CONFIG_ARM_SMMU_LEGACY_DT_BINDINGS */
+
 static int __arm_smmu_alloc_bitmap(unsigned long *map, int start, int end)
 {
 	int idx;
@@ -1566,6 +1590,7 @@ static struct iommu_ops arm_smmu_ops = {
 	.get_resv_regions	= arm_smmu_get_resv_regions,
 	.put_resv_regions	= arm_smmu_put_resv_regions,
 	.pgsize_bitmap		= -1UL, /* Restricted during device attach */
+	.owner			= THIS_MODULE,
 };
 
 static void arm_smmu_device_reset(struct arm_smmu_device *smmu)
@@ -1960,8 +1985,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 
 	legacy_binding = of_find_property(dev->of_node, "mmu-masters", NULL);
 	if (legacy_binding && !using_generic_binding) {
-		if (!using_legacy_binding)
-			pr_notice("deprecated \"mmu-masters\" DT property in use; DMA API support unavailable\n");
+		if (!using_legacy_binding) {
+			pr_notice("deprecated \"mmu-masters\" DT property in use; %s support unavailable\n",
+				  IS_ENABLED(CONFIG_ARM_SMMU_LEGACY_DT_BINDINGS) ? "DMA API" : "SMMU");
+		}
 		using_legacy_binding = true;
 	} else if (!legacy_binding && !using_legacy_binding) {
 		using_generic_binding = true;
@@ -1995,7 +2022,6 @@ static int arm_smmu_bus_init(struct iommu_ops *ops)
 #endif
 #ifdef CONFIG_PCI
 	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
 		err = bus_set_iommu(&pci_bus_type, ops);
 		if (err)
 			goto err_reset_amba_ops;
@@ -2173,20 +2199,6 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	return 0;
 }
 
-/*
- * With the legacy DT binding in play, though, we have no guarantees about
- * probe order, but then we're also not doing default domains, so we can
- * delay setting bus ops until we're sure every possible SMMU is ready,
- * and that way ensure that no add_device() calls get missed.
- */
-static int arm_smmu_legacy_bus_init(void)
-{
-	if (using_legacy_binding)
-		return arm_smmu_bus_init(&arm_smmu_ops);
-	return 0;
-}
-device_initcall_sync(arm_smmu_legacy_bus_init);
-
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v3 14/14] iommu/arm-smmu: Update my email address in MODULE_AUTHOR()

[Will Deacon]

I no longer work for Arm, so update the stale reference to my old email
address.

Signed-off-by: Will Deacon <will@kernel.org>
---
 drivers/iommu/arm-smmu-v3.c | 2 +-
 drivers/iommu/arm-smmu.c    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index f79b14f75107..7669beafc493 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -3732,5 +3732,5 @@ static struct platform_driver arm_smmu_driver = {
 module_platform_driver(arm_smmu_driver);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
-MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
+MODULE_AUTHOR("Will Deacon <will@kernel.org>");
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 100ab5b9c255..d55acc48aee3 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2291,5 +2291,5 @@ static struct platform_driver arm_smmu_driver = {
 module_platform_driver(arm_smmu_driver);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
-MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
+MODULE_AUTHOR("Will Deacon <will@kernel.org>");
 MODULE_LICENSE("GPL v2");
-- 
2.24.0.432.g9d3f5f5b63-goog

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH] iommu/arm-smmu: support SMMU module probing from the IORT

[Ard Biesheuvel]

Add support for SMMU drivers built as modules to the ACPI/IORT device
probing path, by deferring the probe of the master if the SMMU driver is
known to exist but has not been loaded yet. Given that the IORT code
registers a platform device for each SMMU that it discovers, we can
easily trigger the udev based autoloading of the SMMU drivers by making
the platform device identifier part of the module alias.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 drivers/acpi/arm64/iort.c   | 4 ++--
 drivers/iommu/arm-smmu-v3.c | 1 +
 drivers/iommu/arm-smmu.c    | 1 +
 3 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 5a7551d060f2..a696457a9b11 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -850,9 +850,9 @@ static inline bool iort_iommu_driver_enabled(u8 type)
 {
 	switch (type) {
 	case ACPI_IORT_NODE_SMMU_V3:
-		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+		return IS_ENABLED(CONFIG_ARM_SMMU_V3);
 	case ACPI_IORT_NODE_SMMU:
-		return IS_BUILTIN(CONFIG_ARM_SMMU);
+		return IS_ENABLED(CONFIG_ARM_SMMU);
 	default:
 		pr_warn("IORT node type %u does not describe an SMMU\n", type);
 		return false;
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 7669beafc493..bf6a1e8eb9b0 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -3733,4 +3733,5 @@ module_platform_driver(arm_smmu_driver);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will@kernel.org>");
+MODULE_ALIAS("platform:arm-smmu-v3");
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index d55acc48aee3..db5106b0955b 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2292,4 +2292,5 @@ module_platform_driver(arm_smmu_driver);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will@kernel.org>");
+MODULE_ALIAS("platform:arm-smmu");
 MODULE_LICENSE("GPL v2");
-- 
2.20.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] iommu/arm-smmu: support SMMU module probing from the IORT

[Robin Murphy]

On 22/11/2019 5:41 pm, Ard Biesheuvel wrote:
> Add support for SMMU drivers built as modules to the ACPI/IORT device
> probing path, by deferring the probe of the master if the SMMU driver is
> known to exist but has not been loaded yet. Given that the IORT code
> registers a platform device for each SMMU that it discovers, we can
> easily trigger the udev based autoloading of the SMMU drivers by making
> the platform device identifier part of the module alias.

Thanks Ard, I was just gearing up to check the ACPI fallout myself :)

AFAICS this looks sufficient to avoid any unexpected behaviour if users 
start playing with the rest of the series on ACPI systems, so we can 
investigate 'proper' device links for IORT at some point in future.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>   drivers/acpi/arm64/iort.c   | 4 ++--
>   drivers/iommu/arm-smmu-v3.c | 1 +
>   drivers/iommu/arm-smmu.c    | 1 +
>   3 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 5a7551d060f2..a696457a9b11 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -850,9 +850,9 @@ static inline bool iort_iommu_driver_enabled(u8 type)
>   {
>   	switch (type) {
>   	case ACPI_IORT_NODE_SMMU_V3:
> -		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +		return IS_ENABLED(CONFIG_ARM_SMMU_V3);
>   	case ACPI_IORT_NODE_SMMU:
> -		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +		return IS_ENABLED(CONFIG_ARM_SMMU);
>   	default:
>   		pr_warn("IORT node type %u does not describe an SMMU\n", type);
>   		return false;
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index 7669beafc493..bf6a1e8eb9b0 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -3733,4 +3733,5 @@ module_platform_driver(arm_smmu_driver);
>   
>   MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
>   MODULE_AUTHOR("Will Deacon <will@kernel.org>");
> +MODULE_ALIAS("platform:arm-smmu-v3");
>   MODULE_LICENSE("GPL v2");
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index d55acc48aee3..db5106b0955b 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -2292,4 +2292,5 @@ module_platform_driver(arm_smmu_driver);
>   
>   MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
>   MODULE_AUTHOR("Will Deacon <will@kernel.org>");
> +MODULE_ALIAS("platform:arm-smmu");
>   MODULE_LICENSE("GPL v2");
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] iommu/arm-smmu: support SMMU module probing from the IORT

[Lorenzo Pieralisi]

On Fri, Nov 22, 2019 at 06:41:25PM +0100, Ard Biesheuvel wrote:
> Add support for SMMU drivers built as modules to the ACPI/IORT device
> probing path, by deferring the probe of the master if the SMMU driver is
> known to exist but has not been loaded yet. Given that the IORT code
> registers a platform device for each SMMU that it discovers, we can
> easily trigger the udev based autoloading of the SMMU drivers by making
> the platform device identifier part of the module alias.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  drivers/acpi/arm64/iort.c   | 4 ++--
>  drivers/iommu/arm-smmu-v3.c | 1 +
>  drivers/iommu/arm-smmu.c    | 1 +
>  3 files changed, 4 insertions(+), 2 deletions(-)

I think it is best if Will picks this up and add it to the
series that modularize the SMMU drivers:

Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 5a7551d060f2..a696457a9b11 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -850,9 +850,9 @@ static inline bool iort_iommu_driver_enabled(u8 type)
>  {
>  	switch (type) {
>  	case ACPI_IORT_NODE_SMMU_V3:
> -		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +		return IS_ENABLED(CONFIG_ARM_SMMU_V3);
>  	case ACPI_IORT_NODE_SMMU:
> -		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +		return IS_ENABLED(CONFIG_ARM_SMMU);
>  	default:
>  		pr_warn("IORT node type %u does not describe an SMMU\n", type);
>  		return false;
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index 7669beafc493..bf6a1e8eb9b0 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -3733,4 +3733,5 @@ module_platform_driver(arm_smmu_driver);
>  
>  MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
>  MODULE_AUTHOR("Will Deacon <will@kernel.org>");
> +MODULE_ALIAS("platform:arm-smmu-v3");
>  MODULE_LICENSE("GPL v2");
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index d55acc48aee3..db5106b0955b 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -2292,4 +2292,5 @@ module_platform_driver(arm_smmu_driver);
>  
>  MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
>  MODULE_AUTHOR("Will Deacon <will@kernel.org>");
> +MODULE_ALIAS("platform:arm-smmu");
>  MODULE_LICENSE("GPL v2");
> -- 
> 2.20.1
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] iommu/arm-smmu: support SMMU module probing from the IORT

[John Garry]

On 25/11/2019 16:04, Lorenzo Pieralisi wrote:
> On Fri, Nov 22, 2019 at 06:41:25PM +0100, Ard Biesheuvel wrote:
>> Add support for SMMU drivers built as modules to the ACPI/IORT device
>> probing path, by deferring the probe of the master if the SMMU driver is
>> known to exist but has not been loaded yet. Given that the IORT code
>> registers a platform device for each SMMU that it discovers, we can
>> easily trigger the udev based autoloading of the SMMU drivers by making
>> the platform device identifier part of the module alias.
>>
>> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
>> ---
>>   drivers/acpi/arm64/iort.c   | 4 ++--
>>   drivers/iommu/arm-smmu-v3.c | 1 +
>>   drivers/iommu/arm-smmu.c    | 1 +
>>   3 files changed, 4 insertions(+), 2 deletions(-)
> 
> I think it is best if Will picks this up and add it to the
> series that modularize the SMMU drivers:
> 
> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Tested-by: John Garry <john.garry@huawei.com> # only manual smmu ko loading

> 
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index 5a7551d060f2..a696457a9b11 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -850,9 +850,9 @@ static inline bool iort_iommu_driver_enabled(u8 type)
>>   {
>>   	switch (type) {
>>   	case ACPI_IORT_NODE_SMMU_V3:
>> -		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
>> +		return IS_ENABLED(CONFIG_ARM_SMMU_V3);
>>   	case ACPI_IORT_NODE_SMMU:
>> -		return IS_BUILTIN(CONFIG_ARM_SMMU);
>> +		return IS_ENABLED(CONFIG_ARM_SMMU);
>>   	default:
>>   		pr_warn("IORT node type %u does not describe an SMMU\n", type);
>>   		return false;
>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
>> index 7669beafc493..bf6a1e8eb9b0 100644
>> --- a/drivers/iommu/arm-smmu-v3.c
>> +++ b/drivers/iommu/arm-smmu-v3.c
>> @@ -3733,4 +3733,5 @@ module_platform_driver(arm_smmu_driver);
>>   
>>   MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
>>   MODULE_AUTHOR("Will Deacon <will@kernel.org>");
>> +MODULE_ALIAS("platform:arm-smmu-v3");
>>   MODULE_LICENSE("GPL v2");
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index d55acc48aee3..db5106b0955b 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -2292,4 +2292,5 @@ module_platform_driver(arm_smmu_driver);
>>   
>>   MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
>>   MODULE_AUTHOR("Will Deacon <will@kernel.org>");
>> +MODULE_ALIAS("platform:arm-smmu");
>>   MODULE_LICENSE("GPL v2");
>> -- 
>> 2.20.1
>>
> .
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu