linux-arm-msm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v7 0/5] Define EINVAL as device/domain incompatibility
@ 2022-10-17 23:00 Nicolin Chen
  2022-10-17 23:01 ` [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs Nicolin Chen
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Nicolin Chen @ 2022-10-17 23:00 UTC (permalink / raw)
  To: joro, will, robin.murphy
  Cc: jgg, kevin.tian, quic_jjohnson, suravee.suthikulpanit, robdclark,
	dwmw2, baolu.lu, yong.wu, matthias.bgg, orsonzhai, baolin.wang,
	zhang.lyra, thierry.reding, vdumpa, jonathanh, jean-philippe,
	tglx, shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

This series is to replace the previous EMEDIUMTYPE patch in a VFIO series:
https://lore.kernel.org/kvm/Yxnt9uQTmbqul5lf@8bytes.org/

The purpose is to regulate all existing ->attach_dev callback functions to
use EINVAL exclusively for an incompatibility error between a device and a
domain. This allows VFIO and IOMMUFD to detect such a soft error, and then
try a different domain with the same device.

Among all the patches, the first two are preparatory changes. And then one
patch to update kdocs and another three patches for the enforcement effort.

Although it might be ideal to merge the previous VFIO series together with
this series, given the number of new changes, the review in the IOMMU list
might need a couple of rounds to finalize. Also, considering that v6.0 is
at rc5 now, perhaps we could merge this IOMMU series and the VFIO one in
different cycles to avoid merge conflicts. If there's less concern for it,
I can respin the finalized version of this series with the previous VFIO
one to merge together into the VFIO tree.

This series is also available on Github:
https://github.com/nicolinc/iommufd/commits/iommu_attach_dev-v7

Changelog
v7/resend:
 * Rebased on v6.1-rc1
v6: https://lore.kernel.org/linux-iommu/cover.1663899032.git.nicolinc@nvidia.com/
 * Added "Reviewed-by" from Jason to all the changes
 * Added "Reviewed-by" from Yong to the mtk_iommu change
 * Dropped the msm patch as it needs a bigger change to fix
v5: https://lore.kernel.org/linux-iommu/cover.1663836372.git.nicolinc@nvidia.com/
 * Updated kdocs to correct "attach" narratives
 * Updated kdocs to be more concise and accurate
 * Added "Reviewed-by" from Kevin to most of changes
 * Added "Reviewed-by" from Baolu to the intel_iommu changes
 * Added "Reviewed-by" from Jean to the virtio-iommu changes
v4: https://lore.kernel.org/linux-iommu/cover.1663744983.git.nicolinc@nvidia.com/
 * Refined kdocs with Kevin's input
 * Fixed an EINVAL conversion in the intel_iommu driver
 * Added missing error-out routines in the msm_iommu driver
 * Added a missing EINVAL conversion in the virtio-iommu driver
 * Updated commit message and added "Reviewed-by" from Kevin to the last patch
v3: https://lore.kernel.org/linux-iommu/cover.1663227492.git.nicolinc@nvidia.com/
 * Added "Reviewed-by" from Vasant to the AMD patch
 * Dropped all unnecessary errno enforcement patches
 * Updated kdocs and brought back the kdocs for the helpers
 * Added a separate patch to propagate "ret" for potential EINVALs
 * Converted to ENODEV those existing EINVAL places that are device-specific
v2: https://lore.kernel.org/linux-iommu/20220914051234.10006-1-nicolinc@nvidia.com/
 * Fixed kdocs format
 * Grouped with the virtio patch from Jean (with a small change)
 * Separated previous ENODEV and EINVAL changes to per-driver ones
 * Redone some of the changes to have explicit return values in the
   ->attach_dev() callback functions or their direct sub-functions.
v1: https://lore.kernel.org/linux-iommu/20220913082448.31120-1-nicolinc@nvidia.com/

Thanks!

Nicolin Chen (5):
  iommu/amd: Drop unnecessary checks in amd_iommu_attach_device()
  iommu: Add return value rules to attach_dev op and APIs
  iommu: Regulate EINVAL in ->attach_dev callback functions
  iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  iommu: Propagate return value in ->attach_dev callback functions

 drivers/iommu/amd/iommu.c                   | 12 ++---------
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 +---------
 drivers/iommu/arm/arm-smmu/arm-smmu.c       |  3 ---
 drivers/iommu/arm/arm-smmu/qcom_iommu.c     |  7 +-----
 drivers/iommu/fsl_pamu.c                    |  2 +-
 drivers/iommu/fsl_pamu_domain.c             |  4 ++--
 drivers/iommu/intel/iommu.c                 | 10 +++------
 drivers/iommu/intel/pasid.c                 |  6 ++++--
 drivers/iommu/iommu.c                       | 24 +++++++++++++++++++++
 drivers/iommu/ipmmu-vmsa.c                  |  2 --
 drivers/iommu/mtk_iommu.c                   |  4 ++--
 drivers/iommu/omap-iommu.c                  |  6 +++---
 drivers/iommu/sprd-iommu.c                  |  4 +---
 drivers/iommu/tegra-gart.c                  |  2 +-
 drivers/iommu/virtio-iommu.c                |  7 +++---
 include/linux/iommu.h                       | 12 +++++++++++
 16 files changed, 60 insertions(+), 56 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs
  2022-10-17 23:00 [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Nicolin Chen
@ 2022-10-17 23:01 ` Nicolin Chen
  2022-10-18  1:20   ` Baolu Lu
  2022-10-17 23:02 ` [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev Nicolin Chen
  2022-10-17 23:54 ` [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Jason Gunthorpe
  2 siblings, 1 reply; 9+ messages in thread
From: Nicolin Chen @ 2022-10-17 23:01 UTC (permalink / raw)
  To: joro, will, robin.murphy
  Cc: jgg, kevin.tian, quic_jjohnson, suravee.suthikulpanit, robdclark,
	dwmw2, baolu.lu, yong.wu, matthias.bgg, orsonzhai, baolin.wang,
	zhang.lyra, thierry.reding, vdumpa, jonathanh, jean-philippe,
	tglx, shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

Cases like VFIO wish to attach a device to an existing domain that was
not allocated specifically from the device. This raises a condition
where the IOMMU driver can fail the domain attach because the domain and
device are incompatible with each other.

This is a soft failure that can be resolved by using a different domain.

Provide a dedicated errno EINVAL from the IOMMU driver during attach that
the reason why the attach failed is because of domain incompatibility.

VFIO can use this to know that the attach is a soft failure and it should
continue searching. Otherwise, the attach will be a hard failure and VFIO
will return the code to userspace.

Update kdocs to add rules of return value to the attach_dev op and APIs.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/iommu.c | 24 ++++++++++++++++++++++++
 include/linux/iommu.h | 12 ++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 4893c2429ca5..3c0c5d64bb50 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1949,6 +1949,18 @@ static int __iommu_attach_device(struct iommu_domain *domain,
 	return ret;
 }
 
+/**
+ * iommu_attach_device - Attach an IOMMU domain to a device
+ * @domain: IOMMU domain to attach
+ * @dev: Device that will be attached
+ *
+ * Returns 0 on success and error code on failure
+ *
+ * Note that EINVAL can be treated as a soft failure, indicating
+ * that certain configuration of the domain is incompatible with
+ * the device. In this case attaching a different domain to the
+ * device may succeed.
+ */
 int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 {
 	struct iommu_group *group;
@@ -2075,6 +2087,18 @@ static int __iommu_attach_group(struct iommu_domain *domain,
 	return ret;
 }
 
+/**
+ * iommu_attach_group - Attach an IOMMU domain to an IOMMU group
+ * @domain: IOMMU domain to attach
+ * @group: IOMMU group that will be attached
+ *
+ * Returns 0 on success and error code on failure
+ *
+ * Note that EINVAL can be treated as a soft failure, indicating
+ * that certain configuration of the domain is incompatible with
+ * the group. In this case attaching a different domain to the
+ * group may succeed.
+ */
 int iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group)
 {
 	int ret;
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index a325532aeab5..6f3569340e8a 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -266,6 +266,18 @@ struct iommu_ops {
 /**
  * struct iommu_domain_ops - domain specific operations
  * @attach_dev: attach an iommu domain to a device
+ *  Return:
+ * * 0		- success
+ * * EINVAL	- can indicate that device and domain are incompatible due to
+ *		  some previous configuration of the domain, in which case the
+ *		  driver shouldn't log an error, since it is legitimate for a
+ *		  caller to test reuse of existing domains. Otherwise, it may
+ *		  still represent some other fundamental problem
+ * * ENOMEM	- out of memory
+ * * ENOSPC	- non-ENOMEM type of resource allocation failures
+ * * EBUSY	- device is attached to a domain and cannot be changed
+ * * ENODEV	- device specific errors, not able to be attached
+ * * <others>	- treated as ENODEV by the caller. Use is discouraged
  * @detach_dev: detach an iommu domain from a device
  * @map: map a physically contiguous memory region to an iommu domain
  * @map_pages: map a physically contiguous set of pages of the same size to
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  2022-10-17 23:00 [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Nicolin Chen
  2022-10-17 23:01 ` [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs Nicolin Chen
@ 2022-10-17 23:02 ` Nicolin Chen
  2022-11-07 15:26   ` Will Deacon
  2022-10-17 23:54 ` [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Jason Gunthorpe
  2 siblings, 1 reply; 9+ messages in thread
From: Nicolin Chen @ 2022-10-17 23:02 UTC (permalink / raw)
  To: joro, will, robin.murphy
  Cc: jgg, kevin.tian, quic_jjohnson, suravee.suthikulpanit, robdclark,
	dwmw2, baolu.lu, yong.wu, matthias.bgg, orsonzhai, baolin.wang,
	zhang.lyra, thierry.reding, vdumpa, jonathanh, jean-philippe,
	tglx, shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

Following the new rules in include/linux/iommu.h kdocs, update all drivers
->attach_dev callback functions to return EINVAL in the failure paths that
are related to domain incompatibility.

Also, drop adjacent error prints to prevent a kernel log spam.

Reviewed-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 +----------
 drivers/iommu/arm/arm-smmu/arm-smmu.c       |  3 ---
 drivers/iommu/arm/arm-smmu/qcom_iommu.c     |  7 +------
 drivers/iommu/intel/iommu.c                 | 10 +++-------
 drivers/iommu/ipmmu-vmsa.c                  |  2 --
 drivers/iommu/omap-iommu.c                  |  2 +-
 drivers/iommu/sprd-iommu.c                  |  4 +---
 drivers/iommu/tegra-gart.c                  |  2 +-
 drivers/iommu/virtio-iommu.c                |  3 +--
 9 files changed, 9 insertions(+), 35 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index ba47c73f5b8c..01fd7df16cb9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 			goto out_unlock;
 		}
 	} else if (smmu_domain->smmu != smmu) {
-		dev_err(dev,
-			"cannot attach to SMMU %s (upstream of %s)\n",
-			dev_name(smmu_domain->smmu->dev),
-			dev_name(smmu->dev));
-		ret = -ENXIO;
+		ret = -EINVAL;
 		goto out_unlock;
 	} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
 		   master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
-		dev_err(dev,
-			"cannot attach to incompatible domain (%u SSID bits != %u)\n",
-			smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
 		ret = -EINVAL;
 		goto out_unlock;
 	} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
 		   smmu_domain->stall_enabled != master->stall_enabled) {
-		dev_err(dev, "cannot attach to stall-%s domain\n",
-			smmu_domain->stall_enabled ? "enabled" : "disabled");
 		ret = -EINVAL;
 		goto out_unlock;
 	}
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
index 6c1114a4d6cc..3bca3cc1ceae 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
@@ -1150,9 +1150,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 	 * different SMMUs.
 	 */
 	if (smmu_domain->smmu != smmu) {
-		dev_err(dev,
-			"cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n",
-			dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev));
 		ret = -EINVAL;
 		goto rpm_put;
 	}
diff --git a/drivers/iommu/arm/arm-smmu/qcom_iommu.c b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
index 3869c3ecda8c..bfd7b51eb5db 100644
--- a/drivers/iommu/arm/arm-smmu/qcom_iommu.c
+++ b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
@@ -381,13 +381,8 @@ static int qcom_iommu_attach_dev(struct iommu_domain *domain, struct device *dev
 	 * Sanity check the domain. We don't support domains across
 	 * different IOMMUs.
 	 */
-	if (qcom_domain->iommu != qcom_iommu) {
-		dev_err(dev, "cannot attach to IOMMU %s while already "
-			"attached to domain on IOMMU %s\n",
-			dev_name(qcom_domain->iommu->dev),
-			dev_name(qcom_iommu->dev));
+	if (qcom_domain->iommu != qcom_iommu)
 		return -EINVAL;
-	}
 
 	return 0;
 }
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index a8b36c3fddf1..1ac94f7fc8be 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -4189,19 +4189,15 @@ static int prepare_domain_attach_device(struct iommu_domain *domain,
 		return -ENODEV;
 
 	if (dmar_domain->force_snooping && !ecap_sc_support(iommu->ecap))
-		return -EOPNOTSUPP;
+		return -EINVAL;
 
 	/* check if this iommu agaw is sufficient for max mapped address */
 	addr_width = agaw_to_width(iommu->agaw);
 	if (addr_width > cap_mgaw(iommu->cap))
 		addr_width = cap_mgaw(iommu->cap);
 
-	if (dmar_domain->max_addr > (1LL << addr_width)) {
-		dev_err(dev, "%s: iommu width (%d) is not "
-		        "sufficient for the mapped address (%llx)\n",
-		        __func__, addr_width, dmar_domain->max_addr);
-		return -EFAULT;
-	}
+	if (dmar_domain->max_addr > (1LL << addr_width))
+		return -EINVAL;
 	dmar_domain->gaw = addr_width;
 
 	/*
diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c
index 3b30c0752274..22230cc15dcd 100644
--- a/drivers/iommu/ipmmu-vmsa.c
+++ b/drivers/iommu/ipmmu-vmsa.c
@@ -628,8 +628,6 @@ static int ipmmu_attach_device(struct iommu_domain *io_domain,
 		 * Something is wrong, we can't attach two devices using
 		 * different IOMMUs to the same domain.
 		 */
-		dev_err(dev, "Can't attach IPMMU %s to domain on IPMMU %s\n",
-			dev_name(mmu->dev), dev_name(domain->mmu->dev));
 		ret = -EINVAL;
 	} else
 		dev_info(dev, "Reusing IPMMU context %u\n", domain->context_id);
diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c
index 3f153f9e0ac5..2fd7702c6709 100644
--- a/drivers/iommu/omap-iommu.c
+++ b/drivers/iommu/omap-iommu.c
@@ -1472,7 +1472,7 @@ omap_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
 	/* only a single client device can be attached to a domain */
 	if (omap_domain->dev) {
 		dev_err(dev, "iommu domain is already attached\n");
-		ret = -EBUSY;
+		ret = -EINVAL;
 		goto out;
 	}
 
diff --git a/drivers/iommu/sprd-iommu.c b/drivers/iommu/sprd-iommu.c
index fadd2c907222..e02793375598 100644
--- a/drivers/iommu/sprd-iommu.c
+++ b/drivers/iommu/sprd-iommu.c
@@ -237,10 +237,8 @@ static int sprd_iommu_attach_device(struct iommu_domain *domain,
 	struct sprd_iommu_domain *dom = to_sprd_domain(domain);
 	size_t pgt_size = sprd_iommu_pgt_size(domain);
 
-	if (dom->sdev) {
-		pr_err("There's already a device attached to this domain.\n");
+	if (dom->sdev)
 		return -EINVAL;
-	}
 
 	dom->pgt_va = dma_alloc_coherent(sdev->dev, pgt_size, &dom->pgt_pa, GFP_KERNEL);
 	if (!dom->pgt_va)
diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c
index e5ca3cf1a949..ed53279d1106 100644
--- a/drivers/iommu/tegra-gart.c
+++ b/drivers/iommu/tegra-gart.c
@@ -112,7 +112,7 @@ static int gart_iommu_attach_dev(struct iommu_domain *domain,
 	spin_lock(&gart->dom_lock);
 
 	if (gart->active_domain && gart->active_domain != domain) {
-		ret = -EBUSY;
+		ret = -EINVAL;
 	} else if (dev_iommu_priv_get(dev) != domain) {
 		dev_iommu_priv_set(dev, domain);
 		gart->active_domain = domain;
diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c
index f0caf9a4f6f8..9e0ca757fa42 100644
--- a/drivers/iommu/virtio-iommu.c
+++ b/drivers/iommu/virtio-iommu.c
@@ -732,8 +732,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		 */
 		ret = viommu_domain_finalise(vdev, domain);
 	} else if (vdomain->viommu != vdev->viommu) {
-		dev_err(dev, "cannot attach to foreign vIOMMU\n");
-		ret = -EXDEV;
+		ret = -EINVAL;
 	}
 	mutex_unlock(&vdomain->mutex);
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 0/5] Define EINVAL as device/domain incompatibility
  2022-10-17 23:00 [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Nicolin Chen
  2022-10-17 23:01 ` [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs Nicolin Chen
  2022-10-17 23:02 ` [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev Nicolin Chen
@ 2022-10-17 23:54 ` Jason Gunthorpe
  2 siblings, 0 replies; 9+ messages in thread
From: Jason Gunthorpe @ 2022-10-17 23:54 UTC (permalink / raw)
  To: Nicolin Chen, joro
  Cc: will, robin.murphy, kevin.tian, quic_jjohnson,
	suravee.suthikulpanit, robdclark, dwmw2, baolu.lu, yong.wu,
	matthias.bgg, orsonzhai, baolin.wang, zhang.lyra, thierry.reding,
	vdumpa, jonathanh, jean-philippe, tglx,
	shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On Mon, Oct 17, 2022 at 04:00:29PM -0700, Nicolin Chen wrote:
> This series is to replace the previous EMEDIUMTYPE patch in a VFIO series:
> https://lore.kernel.org/kvm/Yxnt9uQTmbqul5lf@8bytes.org/
> 
> The purpose is to regulate all existing ->attach_dev callback functions to
> use EINVAL exclusively for an incompatibility error between a device and a
> domain. This allows VFIO and IOMMUFD to detect such a soft error, and then
> try a different domain with the same device.
> 
> Among all the patches, the first two are preparatory changes. And then one
> patch to update kdocs and another three patches for the enforcement effort.
> 
> Although it might be ideal to merge the previous VFIO series together with
> this series, given the number of new changes, the review in the IOMMU list
> might need a couple of rounds to finalize. Also, considering that v6.0 is
> at rc5 now, perhaps we could merge this IOMMU series and the VFIO one in
> different cycles to avoid merge conflicts. If there's less concern for it,
> I can respin the finalized version of this series with the previous VFIO
> one to merge together into the VFIO tree.
> 
> This series is also available on Github:
> https://github.com/nicolinc/iommufd/commits/iommu_attach_dev-v7

Since it didn't make v6.1-rc1, I'd like this on a PR as we have two
trees that will need it now.

Joerg I can make this into a formal signed PR if that is how you'd
like things?

Thanks,
Jason

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs
  2022-10-17 23:01 ` [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs Nicolin Chen
@ 2022-10-18  1:20   ` Baolu Lu
  0 siblings, 0 replies; 9+ messages in thread
From: Baolu Lu @ 2022-10-18  1:20 UTC (permalink / raw)
  To: Nicolin Chen, joro, will, robin.murphy
  Cc: baolu.lu, jgg, kevin.tian, quic_jjohnson, suravee.suthikulpanit,
	robdclark, dwmw2, yong.wu, matthias.bgg, orsonzhai, baolin.wang,
	zhang.lyra, thierry.reding, vdumpa, jonathanh, jean-philippe,
	tglx, shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On 2022/10/18 7:01, Nicolin Chen wrote:
> Cases like VFIO wish to attach a device to an existing domain that was
> not allocated specifically from the device. This raises a condition
> where the IOMMU driver can fail the domain attach because the domain and
> device are incompatible with each other.
> 
> This is a soft failure that can be resolved by using a different domain.
> 
> Provide a dedicated errno EINVAL from the IOMMU driver during attach that
> the reason why the attach failed is because of domain incompatibility.
> 
> VFIO can use this to know that the attach is a soft failure and it should
> continue searching. Otherwise, the attach will be a hard failure and VFIO
> will return the code to userspace.
> 
> Update kdocs to add rules of return value to the attach_dev op and APIs.
> 
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>   drivers/iommu/iommu.c | 24 ++++++++++++++++++++++++
>   include/linux/iommu.h | 12 ++++++++++++
>   2 files changed, 36 insertions(+)
> 
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index 4893c2429ca5..3c0c5d64bb50 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1949,6 +1949,18 @@ static int __iommu_attach_device(struct iommu_domain *domain,
>   	return ret;
>   }
>   
> +/**
> + * iommu_attach_device - Attach an IOMMU domain to a device
> + * @domain: IOMMU domain to attach
> + * @dev: Device that will be attached
> + *
> + * Returns 0 on success and error code on failure
> + *
> + * Note that EINVAL can be treated as a soft failure, indicating
> + * that certain configuration of the domain is incompatible with
> + * the device. In this case attaching a different domain to the
> + * device may succeed.
> + */
>   int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
>   {
>   	struct iommu_group *group;
> @@ -2075,6 +2087,18 @@ static int __iommu_attach_group(struct iommu_domain *domain,
>   	return ret;
>   }
>   
> +/**
> + * iommu_attach_group - Attach an IOMMU domain to an IOMMU group
> + * @domain: IOMMU domain to attach
> + * @group: IOMMU group that will be attached
> + *
> + * Returns 0 on success and error code on failure
> + *
> + * Note that EINVAL can be treated as a soft failure, indicating
> + * that certain configuration of the domain is incompatible with
> + * the group. In this case attaching a different domain to the
> + * group may succeed.
> + */
>   int iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group)
>   {
>   	int ret;
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index a325532aeab5..6f3569340e8a 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -266,6 +266,18 @@ struct iommu_ops {
>   /**
>    * struct iommu_domain_ops - domain specific operations
>    * @attach_dev: attach an iommu domain to a device
> + *  Return:
> + * * 0		- success
> + * * EINVAL	- can indicate that device and domain are incompatible due to
> + *		  some previous configuration of the domain, in which case the
> + *		  driver shouldn't log an error, since it is legitimate for a
> + *		  caller to test reuse of existing domains. Otherwise, it may
> + *		  still represent some other fundamental problem
> + * * ENOMEM	- out of memory
> + * * ENOSPC	- non-ENOMEM type of resource allocation failures
> + * * EBUSY	- device is attached to a domain and cannot be changed
> + * * ENODEV	- device specific errors, not able to be attached
> + * * <others>	- treated as ENODEV by the caller. Use is discouraged
>    * @detach_dev: detach an iommu domain from a device
>    * @map: map a physically contiguous memory region to an iommu domain
>    * @map_pages: map a physically contiguous set of pages of the same size to

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

Best regards,
baolu

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  2022-10-17 23:02 ` [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev Nicolin Chen
@ 2022-11-07 15:26   ` Will Deacon
  2022-11-08  0:14     ` Nicolin Chen
  0 siblings, 1 reply; 9+ messages in thread
From: Will Deacon @ 2022-11-07 15:26 UTC (permalink / raw)
  To: Nicolin Chen
  Cc: joro, robin.murphy, jgg, kevin.tian, quic_jjohnson,
	suravee.suthikulpanit, robdclark, dwmw2, baolu.lu, yong.wu,
	matthias.bgg, orsonzhai, baolin.wang, zhang.lyra, thierry.reding,
	vdumpa, jonathanh, jean-philippe, tglx,
	shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On Mon, Oct 17, 2022 at 04:02:21PM -0700, Nicolin Chen wrote:
> Following the new rules in include/linux/iommu.h kdocs, update all drivers
> ->attach_dev callback functions to return EINVAL in the failure paths that
> are related to domain incompatibility.
> 
> Also, drop adjacent error prints to prevent a kernel log spam.
> 
> Reviewed-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 +----------
>  drivers/iommu/arm/arm-smmu/arm-smmu.c       |  3 ---
>  drivers/iommu/arm/arm-smmu/qcom_iommu.c     |  7 +------
>  drivers/iommu/intel/iommu.c                 | 10 +++-------
>  drivers/iommu/ipmmu-vmsa.c                  |  2 --
>  drivers/iommu/omap-iommu.c                  |  2 +-
>  drivers/iommu/sprd-iommu.c                  |  4 +---
>  drivers/iommu/tegra-gart.c                  |  2 +-
>  drivers/iommu/virtio-iommu.c                |  3 +--
>  9 files changed, 9 insertions(+), 35 deletions(-)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index ba47c73f5b8c..01fd7df16cb9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
>  			goto out_unlock;
>  		}
>  	} else if (smmu_domain->smmu != smmu) {
> -		dev_err(dev,
> -			"cannot attach to SMMU %s (upstream of %s)\n",
> -			dev_name(smmu_domain->smmu->dev),
> -			dev_name(smmu->dev));
> -		ret = -ENXIO;
> +		ret = -EINVAL;
>  		goto out_unlock;
>  	} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
>  		   master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
> -		dev_err(dev,
> -			"cannot attach to incompatible domain (%u SSID bits != %u)\n",
> -			smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
>  		ret = -EINVAL;
>  		goto out_unlock;
>  	} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
>  		   smmu_domain->stall_enabled != master->stall_enabled) {
> -		dev_err(dev, "cannot attach to stall-%s domain\n",
> -			smmu_domain->stall_enabled ? "enabled" : "disabled");
>  		ret = -EINVAL;
>  		goto out_unlock;
>  	}

I think it would be helpful to preserve these messages using
dev_err_ratelimited() so that attach failure can be diagnosed without
having to hack the messages back into the driver.

With that:

Acked-by: Will Deacon <will@kernel.org>

Will

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  2022-11-07 15:26   ` Will Deacon
@ 2022-11-08  0:14     ` Nicolin Chen
  2022-11-08 13:20       ` Will Deacon
  0 siblings, 1 reply; 9+ messages in thread
From: Nicolin Chen @ 2022-11-08  0:14 UTC (permalink / raw)
  To: Will Deacon
  Cc: joro, robin.murphy, jgg, kevin.tian, quic_jjohnson,
	suravee.suthikulpanit, robdclark, dwmw2, baolu.lu, yong.wu,
	matthias.bgg, orsonzhai, baolin.wang, zhang.lyra, thierry.reding,
	vdumpa, jonathanh, jean-philippe, tglx,
	shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On Mon, Nov 07, 2022 at 03:26:45PM +0000, Will Deacon wrote:

> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index ba47c73f5b8c..01fd7df16cb9 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> >                       goto out_unlock;
> >               }
> >       } else if (smmu_domain->smmu != smmu) {
> > -             dev_err(dev,
> > -                     "cannot attach to SMMU %s (upstream of %s)\n",
> > -                     dev_name(smmu_domain->smmu->dev),
> > -                     dev_name(smmu->dev));
> > -             ret = -ENXIO;
> > +             ret = -EINVAL;
> >               goto out_unlock;
> >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> >                  master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
> > -             dev_err(dev,
> > -                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
> > -                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
> >               ret = -EINVAL;
> >               goto out_unlock;
> >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> >                  smmu_domain->stall_enabled != master->stall_enabled) {
> > -             dev_err(dev, "cannot attach to stall-%s domain\n",
> > -                     smmu_domain->stall_enabled ? "enabled" : "disabled");
> >               ret = -EINVAL;
> >               goto out_unlock;
> >       }

> I think it would be helpful to preserve these messages using
> dev_err_ratelimited() so that attach failure can be diagnosed without
> having to hack the messages back into the driver.

Thank you for the review.

The change is already picked up last week. Yet, I can add prints
back with a followup patch, if no one has a problem with that.

Also, I am not quite sure what the use case would be to have an
error print. Perhaps dev_dbg() would be more fitting if it is
just for diagnosis?

Thanks
Nic

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  2022-11-08  0:14     ` Nicolin Chen
@ 2022-11-08 13:20       ` Will Deacon
  2022-11-08 13:23         ` Jason Gunthorpe
  0 siblings, 1 reply; 9+ messages in thread
From: Will Deacon @ 2022-11-08 13:20 UTC (permalink / raw)
  To: Nicolin Chen
  Cc: joro, robin.murphy, jgg, kevin.tian, quic_jjohnson,
	suravee.suthikulpanit, robdclark, dwmw2, baolu.lu, yong.wu,
	matthias.bgg, orsonzhai, baolin.wang, zhang.lyra, thierry.reding,
	vdumpa, jonathanh, jean-philippe, tglx,
	shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On Mon, Nov 07, 2022 at 04:14:32PM -0800, Nicolin Chen wrote:
> On Mon, Nov 07, 2022 at 03:26:45PM +0000, Will Deacon wrote:
> 
> > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > index ba47c73f5b8c..01fd7df16cb9 100644
> > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > @@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> > >                       goto out_unlock;
> > >               }
> > >       } else if (smmu_domain->smmu != smmu) {
> > > -             dev_err(dev,
> > > -                     "cannot attach to SMMU %s (upstream of %s)\n",
> > > -                     dev_name(smmu_domain->smmu->dev),
> > > -                     dev_name(smmu->dev));
> > > -             ret = -ENXIO;
> > > +             ret = -EINVAL;
> > >               goto out_unlock;
> > >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> > >                  master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
> > > -             dev_err(dev,
> > > -                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
> > > -                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
> > >               ret = -EINVAL;
> > >               goto out_unlock;
> > >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> > >                  smmu_domain->stall_enabled != master->stall_enabled) {
> > > -             dev_err(dev, "cannot attach to stall-%s domain\n",
> > > -                     smmu_domain->stall_enabled ? "enabled" : "disabled");
> > >               ret = -EINVAL;
> > >               goto out_unlock;
> > >       }
> 
> > I think it would be helpful to preserve these messages using
> > dev_err_ratelimited() so that attach failure can be diagnosed without
> > having to hack the messages back into the driver.
> 
> Thank you for the review.
> 
> The change is already picked up last week. Yet, I can add prints
> back with a followup patch, if no one has a problem with that.

Sorry, I fell behind with upstream so I got to this late. A patch on top
would be fantastic!

> Also, I am not quite sure what the use case would be to have an
> error print. Perhaps dev_dbg() would be more fitting if it is
> just for diagnosis?

Sure, that works for me. I think the messages are useful for folks
triggering this path e.g. via sysfs but if they're limited to debug I think
that's better than removing them altogether.

Cheers,

Will

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev
  2022-11-08 13:20       ` Will Deacon
@ 2022-11-08 13:23         ` Jason Gunthorpe
  0 siblings, 0 replies; 9+ messages in thread
From: Jason Gunthorpe @ 2022-11-08 13:23 UTC (permalink / raw)
  To: Will Deacon
  Cc: Nicolin Chen, joro, robin.murphy, kevin.tian, quic_jjohnson,
	suravee.suthikulpanit, robdclark, dwmw2, baolu.lu, yong.wu,
	matthias.bgg, orsonzhai, baolin.wang, zhang.lyra, thierry.reding,
	vdumpa, jonathanh, jean-philippe, tglx,
	shameerali.kolothum.thodi, christophe.jaillet, yangyicong,
	yangyingliang, quic_saipraka, jon, iommu, linux-kernel,
	linux-arm-kernel, linux-arm-msm, linux-mediatek, linux-tegra,
	virtualization

On Tue, Nov 08, 2022 at 01:20:42PM +0000, Will Deacon wrote:
> On Mon, Nov 07, 2022 at 04:14:32PM -0800, Nicolin Chen wrote:
> > On Mon, Nov 07, 2022 at 03:26:45PM +0000, Will Deacon wrote:
> > 
> > > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > index ba47c73f5b8c..01fd7df16cb9 100644
> > > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > @@ -2430,23 +2430,14 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> > > >                       goto out_unlock;
> > > >               }
> > > >       } else if (smmu_domain->smmu != smmu) {
> > > > -             dev_err(dev,
> > > > -                     "cannot attach to SMMU %s (upstream of %s)\n",
> > > > -                     dev_name(smmu_domain->smmu->dev),
> > > > -                     dev_name(smmu->dev));
> > > > -             ret = -ENXIO;
> > > > +             ret = -EINVAL;
> > > >               goto out_unlock;
> > > >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> > > >                  master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) {
> > > > -             dev_err(dev,
> > > > -                     "cannot attach to incompatible domain (%u SSID bits != %u)\n",
> > > > -                     smmu_domain->s1_cfg.s1cdmax, master->ssid_bits);
> > > >               ret = -EINVAL;
> > > >               goto out_unlock;
> > > >       } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 &&
> > > >                  smmu_domain->stall_enabled != master->stall_enabled) {
> > > > -             dev_err(dev, "cannot attach to stall-%s domain\n",
> > > > -                     smmu_domain->stall_enabled ? "enabled" : "disabled");
> > > >               ret = -EINVAL;
> > > >               goto out_unlock;
> > > >       }
> > 
> > > I think it would be helpful to preserve these messages using
> > > dev_err_ratelimited() so that attach failure can be diagnosed without
> > > having to hack the messages back into the driver.
> > 
> > Thank you for the review.
> > 
> > The change is already picked up last week. Yet, I can add prints
> > back with a followup patch, if no one has a problem with that.
> 
> Sorry, I fell behind with upstream so I got to this late. A patch on top
> would be fantastic!
> 
> > Also, I am not quite sure what the use case would be to have an
> > error print. Perhaps dev_dbg() would be more fitting if it is
> > just for diagnosis?
> 
> Sure, that works for me. I think the messages are useful for folks
> triggering this path e.g. via sysfs but if they're limited to debug I think
> that's better than removing them altogether.

I suspsect it has to be dbg - vfio/iommufd will probably trigger these
messages as it probes for domains that are compatible - eg certainly
the first one. Even if it is a "once" it would still emit a confusing
message for a normal occurance.

This is why they were removed in the first place..

Jason

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-11-08 13:23 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-17 23:00 [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Nicolin Chen
2022-10-17 23:01 ` [PATCH v7 2/5] iommu: Add return value rules to attach_dev op and APIs Nicolin Chen
2022-10-18  1:20   ` Baolu Lu
2022-10-17 23:02 ` [PATCH v7 4/5] iommu: Use EINVAL for incompatible device/domain in ->attach_dev Nicolin Chen
2022-11-07 15:26   ` Will Deacon
2022-11-08  0:14     ` Nicolin Chen
2022-11-08 13:20       ` Will Deacon
2022-11-08 13:23         ` Jason Gunthorpe
2022-10-17 23:54 ` [PATCH v7 0/5] Define EINVAL as device/domain incompatibility Jason Gunthorpe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).