[PATCH v2 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[PATCH v2 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[shameer]

On certain HiSilicon platforms (Hip06/Hip07) the GIC ITS and
PCIe RC deviates from the standard implementation and this breaks
PCIe MSI functionality when SMMU is enabled.

The HiSilicon erratum 161010801 describes this limitation of certain
HiSilicon platforms to support the SMMU mappings for MSI transactions.
On these platforms GICv3 ITS translator is presented with the deviceID
by extending the MSI payload data to 64 bits to include the deviceID.
Hence, the PCIe controller on this platforms has to differentiate the
MSI payload against other DMA payload and has to modify the MSI payload.
This basically makes it difficult for this platforms to have a SMMU
translation for MSI.

This patch implements a ACPI table based quirk to reserve the hw msi
regions in the smmu-v3 driver which means these address regions will
not be translated and will be excluded from iova allocations.

To implement this quirk, the following changes are incorporated:

1. Added a generic helper function to IORT code to retrieve and
   reserve the HW ITS address regions.
2. Added quirk to SMMUv3 to reserve HW ITS address regions based
   on IORT SMMUv3 model.

This is based on the following patches:
1. https://patchwork.kernel.org/patch/9740733/
2. https://patchwork.kernel.org/patch/9730491/

Thanks,
Shameer

Changelog:

v1 --> v2

-patch 2/2: Invoke iort helper fn based on fwnode type(acpi).

RFCv2 -->PATCH

-Incorporated Lorenzo's review comments.

RFC v1 --> RFC v2

Based on Robin's review comments,
-Removed  the generic erratum framework.
-Using IORT/MADT tables to retrieve the ITS base addr instead
 of vendor specific CSRT table.

shameer (2):
  acpi:iort: Add an IORT helper function to reserve HW ITS address
    regions for IOMMU drivers
  iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

 drivers/acpi/arm64/iort.c        | 91 ++++++++++++++++++++++++++++++++++++++--
 drivers/iommu/arm-smmu-v3.c      | 29 ++++++++++---
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 +++-
 4 files changed, 120 insertions(+), 10 deletions(-)

-- 
1.9.1

[PATCH v2 1/2] acpi:iort: Add an IORT helper function to reserve HW ITS address regions for IOMMU drivers

[shameer]

The helper function retrieves ITS address regions through IORT
device <-> ITS mappings and reserves it so that these regions
will not be translated by IOMMU and will be excluded from IOVA
allocations. IOMMU drivers can use this to implement their
.get_resv_regions callback.

Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
---
 drivers/acpi/arm64/iort.c        | 91 ++++++++++++++++++++++++++++++++++++++--
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 +++-
 3 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index c5fecf9..4810dbe 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -34,6 +34,7 @@
 struct iort_its_msi_chip {
 	struct list_head	list;
 	struct fwnode_handle	*fw_node;
+	phys_addr_t		base_addr;
 	u32			translation_id;
 };
 
@@ -131,14 +132,16 @@ typedef acpi_status (*iort_find_node_callback)
 static DEFINE_SPINLOCK(iort_msi_chip_lock);
 
 /**
- * iort_register_domain_token() - register domain token and related ITS ID
- * to the list from where we can get it back later on.
+ * iort_register_domain_token() - register domain token along with related
+ * ITS ID and base address to the list from where we can get it back later on.
  * @trans_id: ITS ID.
+ * @base: ITS base address.
  * @fw_node: Domain token.
  *
  * Returns: 0 on success, -ENOMEM if no memory when allocating list element
  */
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node)
 {
 	struct iort_its_msi_chip *its_msi_chip;
 
@@ -148,6 +151,7 @@ int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
 
 	its_msi_chip->fw_node = fw_node;
 	its_msi_chip->translation_id = trans_id;
+	its_msi_chip->base_addr = base;
 
 	spin_lock(&iort_msi_chip_lock);
 	list_add(&its_msi_chip->list, &iort_msi_chip_list);
@@ -491,6 +495,24 @@ int iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id)
 	return -ENODEV;
 }
 
+static int iort_find_its_base(u32 its_id, phys_addr_t *base)
+{
+	struct iort_its_msi_chip *its_msi_chip;
+	bool match = false;
+
+	spin_lock(&iort_msi_chip_lock);
+	list_for_each_entry(its_msi_chip, &iort_msi_chip_list, list) {
+		if (its_msi_chip->translation_id == its_id) {
+			*base = its_msi_chip->base_addr;
+			match = true;
+			break;
+		}
+	}
+	spin_unlock(&iort_msi_chip_lock);
+
+	return match ? 0 : -ENODEV;
+}
+
 /**
  * iort_dev_find_its_id() - Find the ITS identifier for a device
  * @dev: The device.
@@ -649,6 +671,67 @@ int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 
 	return err;
 }
+
+/**
+ * iort_iommu_its_get_resv_regions - Reserved region driver helper
+ * @dev: Device from iommu_get_resv_regions()
+ * @list: Reserved region list from iommu_get_resv_regions()
+ *
+ * Returns: 0 on at least one ITS address region reservation,
+ *          appropriate error value otherwise.
+ *
+ * IOMMU drivers can use this to implement their .get_resv_regions callback
+ * for reserving the HW ITS address regions.
+ */
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{
+	int i;
+	struct acpi_iort_its_group *its;
+	struct acpi_iort_node *node, *its_node = NULL;
+	bool resv = false;
+
+	node = iort_find_dev_node(dev);
+	if (!node)
+		return -ENODEV;
+
+	if (dev_is_pci(dev)) {
+		u32 rid;
+
+		pci_for_each_dma_alias(to_pci_dev(dev), __get_pci_rid, &rid);
+		its_node = iort_node_map_id(node, rid, NULL, IORT_MSI_TYPE);
+	} else {
+		for (i = 0; i < node->mapping_count; i++) {
+			its_node = iort_node_map_platform_id(node, NULL,
+							 IORT_MSI_TYPE, i);
+			if (its_node)
+				break;
+		}
+	}
+
+	if (!its_node)
+		return -ENODEV;
+
+	/* Move to ITS specific data */
+	its = (struct acpi_iort_its_group *)its_node->node_data;
+
+	for (i = 0; i < its->its_count; i++) {
+		phys_addr_t base;
+
+		if (!iort_find_its_base(its->identifiers[i], &base)) {
+			int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+			struct iommu_resv_region *region;
+
+			region = iommu_alloc_resv_region(base, SZ_128K, prot,
+							 IOMMU_RESV_MSI);
+			if (region) {
+				list_add_tail(&region->list, head);
+				resv = true;
+			}
+		}
+	}
+
+	return resv ? 0 : -ENODEV;
+}
 #else
 static inline
 const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
@@ -656,6 +739,8 @@ const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
 static inline
 int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 { return 0; }
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 45ea1933..c45a2ad 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1854,7 +1854,8 @@ static int __init gic_acpi_parse_madt_its(struct acpi_subtable_header *header,
 		return -ENOMEM;
 	}
 
-	err = iort_register_domain_token(its_entry->translation_id, dom_handle);
+	err = iort_register_domain_token(its_entry->translation_id, res.start,
+					 dom_handle);
 	if (err) {
 		pr_err("ITS@%pa: Unable to register GICv3 ITS domain token (ITS ID %d) to IORT\n",
 		       &res.start, its_entry->translation_id);
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 3ff9ace..35cf45c 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -26,7 +26,8 @@
 #define IORT_IRQ_MASK(irq)		(irq & 0xffffffffULL)
 #define IORT_IRQ_TRIGGER_MASK(irq)	((irq >> 32) & 0xffffffffULL)
 
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node);
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node);
 void iort_deregister_domain_token(int trans_id);
 struct fwnode_handle *iort_find_domain_token(int trans_id);
 #ifdef CONFIG_ACPI_IORT
@@ -39,6 +40,7 @@
 /* IOMMU interface */
 void iort_set_dma_mask(struct device *dev);
 const struct iommu_ops *iort_iommu_configure(struct device *dev);
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head);
 #else
 static inline void acpi_iort_init(void) { }
 static inline bool iort_node_match(u8 type) { return false; }
@@ -53,6 +55,9 @@ static inline void iort_set_dma_mask(struct device *dev) { }
 static inline
 const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
+static inline
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 #endif /* __ACPI_IORT_H__ */
-- 
1.9.1

[PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[shameer]

The HiSilicon erratum 161010801 describes the limitation of HiSilicon
platforms Hip06/Hip07 to support the SMMU mappings for MSI transactions.

On these platforms GICv3 ITS translator is presented with the deviceID
by extending the MSI payload data to 64 bits to include the deviceID.
Hence, the PCIe controller on this platforms has to differentiate the
MSI payload against other DMA payload and has to modify the MSI payload.
This basically makes it difficult for this platforms to have a SMMU
translation for MSI.

This patch implements a ACPI table based quirk to reserve the hw msi
regions in the smmu-v3 driver which means these address regions will
not be translated and will be excluded from iova allocations.

Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
---
 drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index abe4b88..f03c63b 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -597,6 +597,7 @@ struct arm_smmu_device {
 	u32				features;
 
 #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
+#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
 	u32				options;
 
 	struct arm_smmu_cmdq		cmdq;
@@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct device *dev,
 				      struct list_head *head)
 {
 	struct iommu_resv_region *region;
+	struct arm_smmu_device *smmu;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
 
-	region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
-					 prot, IOMMU_RESV_SW_MSI);
-	if (!region)
-		return;
+	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
+
+	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
+		      dev_is_pci(dev)) {
+		int ret = -EINVAL;
+
+		if (!is_of_node(smmu->dev->fwnode))
+			ret = iort_iommu_its_get_resv_regions(dev, head);
 
-	list_add_tail(&region->list, head);
+		if (ret) {
+			dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
+			return;
+		}
+	} else {
+		region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
+						 prot, IOMMU_RESV_SW_MSI);
+		if (!region)
+			return;
+
+		list_add_tail(&region->list, head);
+	}
 
 	iommu_dma_get_resv_regions(dev, head);
 }
@@ -2611,6 +2629,7 @@ static void parse_driver_acpi_options(struct acpi_iort_smmu_v3 *iort_smmu,
 	switch (iort_smmu->model) {
 	case ACPI_IORT_SMMU_HISILICON_HI161X:
 		smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
+		smmu->options |= ARM_SMMU_OPT_RESV_HW_MSI;
 		break;
 	default:
 		break;
-- 
1.9.1

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Robin Murphy]

On 19/06/17 16:45, shameer wrote:
> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> platforms Hip06/Hip07 to support the SMMU mappings for MSI transactions.
> 
> On these platforms GICv3 ITS translator is presented with the deviceID
> by extending the MSI payload data to 64 bits to include the deviceID.
> Hence, the PCIe controller on this platforms has to differentiate the
> MSI payload against other DMA payload and has to modify the MSI payload.
> This basically makes it difficult for this platforms to have a SMMU
> translation for MSI.
> 
> This patch implements a ACPI table based quirk to reserve the hw msi
> regions in the smmu-v3 driver which means these address regions will
> not be translated and will be excluded from iova allocations.
> 
> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
>  1 file changed, 24 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index abe4b88..f03c63b 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -597,6 +597,7 @@ struct arm_smmu_device {
>  	u32				features;
>  
>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
>  	u32				options;
>  
>  	struct arm_smmu_cmdq		cmdq;
> @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct device *dev,
>  				      struct list_head *head)
>  {
>  	struct iommu_resv_region *region;
> +	struct arm_smmu_device *smmu;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
>  
> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> -					 prot, IOMMU_RESV_SW_MSI);
> -	if (!region)
> -		return;
> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> +
> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&

AFAICS, iommu_get_resv_regions is only ever called on devices which are
at least already part of an iommu_group, so smmu should never
legitimately be NULL. I'd say if you really want to be robust against
flagrant API misuse, at least WARN_ON and bail out immediately.

Robin.

> +		      dev_is_pci(dev)) {
> +		int ret = -EINVAL;
> +
> +		if (!is_of_node(smmu->dev->fwnode))
> +			ret = iort_iommu_its_get_resv_regions(dev, head);
>  
> -	list_add_tail(&region->list, head);
> +		if (ret) {
> +			dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
> +			return;
> +		}
> +	} else {
> +		region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> +						 prot, IOMMU_RESV_SW_MSI);
> +		if (!region)
> +			return;
> +
> +		list_add_tail(&region->list, head);
> +	}
>  
>  	iommu_dma_get_resv_regions(dev, head);
>  }
> @@ -2611,6 +2629,7 @@ static void parse_driver_acpi_options(struct acpi_iort_smmu_v3 *iort_smmu,
>  	switch (iort_smmu->model) {
>  	case ACPI_IORT_SMMU_HISILICON_HI161X:
>  		smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
> +		smmu->options |= ARM_SMMU_OPT_RESV_HW_MSI;
>  		break;
>  	default:
>  		break;
> 

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Lorenzo Pieralisi]

Hi Shameer,

On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> platforms Hip06/Hip07 to support the SMMU mappings for MSI transactions.
> 
> On these platforms GICv3 ITS translator is presented with the deviceID
> by extending the MSI payload data to 64 bits to include the deviceID.
> Hence, the PCIe controller on this platforms has to differentiate the
> MSI payload against other DMA payload and has to modify the MSI payload.
> This basically makes it difficult for this platforms to have a SMMU
> translation for MSI.
> 
> This patch implements a ACPI table based quirk to reserve the hw msi
> regions in the smmu-v3 driver which means these address regions will
> not be translated and will be excluded from iova allocations.
> 
> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
>  1 file changed, 24 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index abe4b88..f03c63b 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -597,6 +597,7 @@ struct arm_smmu_device {
>  	u32				features;
>  
>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
>  	u32				options;
>  
>  	struct arm_smmu_cmdq		cmdq;
> @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct device *dev,
>  				      struct list_head *head)
>  {
>  	struct iommu_resv_region *region;
> +	struct arm_smmu_device *smmu;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
>  
> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> -					 prot, IOMMU_RESV_SW_MSI);
> -	if (!region)
> -		return;
> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> +
> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
> +		      dev_is_pci(dev)) {

IORT changes are fine to me, I am still no big fan of this supposedly
generic option that is _really_ platform specific (in particular as I
said before the quirk depends on the PCI host bridge but in this
patchset I see no such dependency. In short - the quirk is hooked off
the SMMUv3 model which implicitly implies a PCI host bridge
configuration IIUC). It is Will and Robin decision though, I am not sure
you can make it any tidier (given that on ACPI you may not even have
a PCI host bridge specific _HID to base your check above on).

Thanks,
Lorenzo

> +		int ret = -EINVAL;
> +
> +		if (!is_of_node(smmu->dev->fwnode))
> +			ret = iort_iommu_its_get_resv_regions(dev, head);
>  
> -	list_add_tail(&region->list, head);
> +		if (ret) {
> +			dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
> +			return;
> +		}
> +	} else {
> +		region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> +						 prot, IOMMU_RESV_SW_MSI);
> +		if (!region)
> +			return;
> +
> +		list_add_tail(&region->list, head);
> +	}
>  
>  	iommu_dma_get_resv_regions(dev, head);
>  }
> @@ -2611,6 +2629,7 @@ static void parse_driver_acpi_options(struct acpi_iort_smmu_v3 *iort_smmu,
>  	switch (iort_smmu->model) {
>  	case ACPI_IORT_SMMU_HISILICON_HI161X:
>  		smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
> +		smmu->options |= ARM_SMMU_OPT_RESV_HW_MSI;
>  		break;
>  	default:
>  		break;
> -- 
> 1.9.1
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
> Sent: Monday, June 19, 2017 6:42 PM
> To: Shameerali Kolothum Thodi; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org;
> marc.zyngier-5wv7dgnIgG8@public.gmane.org; sudeep.holla-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org;
> hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
> Cc: Gabriele Paoloni; John Garry; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-
> arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
> devel-E0kO6a4B6psdnm+yROfE0A@public.gmane.org; Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> HiSilicon erratum 161010801
> 
> On 19/06/17 16:45, shameer wrote:
> > The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> > platforms Hip06/Hip07 to support the SMMU mappings for MSI
> transactions.
> >
> > On these platforms GICv3 ITS translator is presented with the deviceID
> > by extending the MSI payload data to 64 bits to include the deviceID.
> > Hence, the PCIe controller on this platforms has to differentiate the
> > MSI payload against other DMA payload and has to modify the MSI
> payload.
> > This basically makes it difficult for this platforms to have a SMMU
> > translation for MSI.
> >
> > This patch implements a ACPI table based quirk to reserve the hw msi
> > regions in the smmu-v3 driver which means these address regions will
> > not be translated and will be excluded from iova allocations.
> >
> > Signed-off-by: shameer <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
> > ---
> >  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
> >  1 file changed, 24 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> v3.c
> > index abe4b88..f03c63b 100644
> > --- a/drivers/iommu/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm-smmu-v3.c
> > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> >  	u32				features;
> >
> >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> >  	u32				options;
> >
> >  	struct arm_smmu_cmdq		cmdq;
> > @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct
> device *dev,
> >  				      struct list_head *head)
> >  {
> >  	struct iommu_resv_region *region;
> > +	struct arm_smmu_device *smmu;
> > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> >
> > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> MSI_IOVA_LENGTH,
> > -					 prot, IOMMU_RESV_SW_MSI);
> > -	if (!region)
> > -		return;
> > +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> > +
> > +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> &&
> 
> AFAICS, iommu_get_resv_regions is only ever called on devices which are
> at least already part of an iommu_group, so smmu should never
> legitimately be NULL. I'd say if you really want to be robust against
> flagrant API misuse, at least WARN_ON and bail out immediately.

Ok.  I will address this in next revision.

Thanks,
Shameer

RE: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org]
> Sent: Tuesday, June 20, 2017 11:29 AM
> To: Shameerali Kolothum Thodi
> Cc: marc.zyngier-5wv7dgnIgG8@public.gmane.org; sudeep.holla-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org;
> robin.murphy-5wv7dgnIgG8@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; Gabriele Paoloni; John
> Garry; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
> kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; devel-E0kO6a4B6psdnm+yROfE0A@public.gmane.org;
> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> HiSilicon erratum 161010801
> 
> Hi Shameer,
> 
> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
> > The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> > platforms Hip06/Hip07 to support the SMMU mappings for MSI
> transactions.
> >
> > On these platforms GICv3 ITS translator is presented with the deviceID
> > by extending the MSI payload data to 64 bits to include the deviceID.
> > Hence, the PCIe controller on this platforms has to differentiate the
> > MSI payload against other DMA payload and has to modify the MSI
> payload.
> > This basically makes it difficult for this platforms to have a SMMU
> > translation for MSI.
> >
> > This patch implements a ACPI table based quirk to reserve the hw msi
> > regions in the smmu-v3 driver which means these address regions will
> > not be translated and will be excluded from iova allocations.
> >
> > Signed-off-by: shameer <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
> > ---
> >  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
> >  1 file changed, 24 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> v3.c
> > index abe4b88..f03c63b 100644
> > --- a/drivers/iommu/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm-smmu-v3.c
> > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> >  	u32				features;
> >
> >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> >  	u32				options;
> >
> >  	struct arm_smmu_cmdq		cmdq;
> > @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct
> device *dev,
> >  				      struct list_head *head)
> >  {
> >  	struct iommu_resv_region *region;
> > +	struct arm_smmu_device *smmu;
> > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> >
> > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> MSI_IOVA_LENGTH,
> > -					 prot, IOMMU_RESV_SW_MSI);
> > -	if (!region)
> > -		return;
> > +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> > +
> > +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> &&
> > +		      dev_is_pci(dev)) {
> 
> IORT changes are fine to me, I am still no big fan of this supposedly
> generic option that is _really_ platform specific (in particular as I
> said before the quirk depends on the PCI host bridge but in this
> patchset I see no such dependency. In short - the quirk is hooked off
> the SMMUv3 model which implicitly implies a PCI host bridge
> configuration IIUC). It is Will and Robin decision though, I am not sure
> you can make it any tidier (given that on ACPI you may not even have
> a PCI host bridge specific _HID to base your check above on).

Thanks Lorenzo. I understand your point. As far as our platform is 
concerned, I think It is ok to remove the dev_is_pci() check and make
it generic, if that helps.  I don't think it will harm us other than couple of
 "HW MSI region resv failed: " logs  for our platform devices.

Hi Will/Robin,
Please let me know your thoughts.

Thanks,
Shameer

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Robin Murphy]

On 20/06/17 15:07, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org]
>> Sent: Tuesday, June 20, 2017 11:29 AM
>> To: Shameerali Kolothum Thodi
>> Cc: marc.zyngier-5wv7dgnIgG8@public.gmane.org; sudeep.holla-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org;
>> robin.murphy-5wv7dgnIgG8@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; Gabriele Paoloni; John
>> Garry; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
>> kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; devel-E0kO6a4B6psdnm+yROfE0A@public.gmane.org;
>> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
>> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
>> HiSilicon erratum 161010801
>>
>> Hi Shameer,
>>
>> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
>>> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
>>> platforms Hip06/Hip07 to support the SMMU mappings for MSI
>> transactions.
>>>
>>> On these platforms GICv3 ITS translator is presented with the deviceID
>>> by extending the MSI payload data to 64 bits to include the deviceID.
>>> Hence, the PCIe controller on this platforms has to differentiate the
>>> MSI payload against other DMA payload and has to modify the MSI
>> payload.
>>> This basically makes it difficult for this platforms to have a SMMU
>>> translation for MSI.
>>>
>>> This patch implements a ACPI table based quirk to reserve the hw msi
>>> regions in the smmu-v3 driver which means these address regions will
>>> not be translated and will be excluded from iova allocations.
>>>
>>> Signed-off-by: shameer <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
>>> ---
>>>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
>>>  1 file changed, 24 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
>> v3.c
>>> index abe4b88..f03c63b 100644
>>> --- a/drivers/iommu/arm-smmu-v3.c
>>> +++ b/drivers/iommu/arm-smmu-v3.c
>>> @@ -597,6 +597,7 @@ struct arm_smmu_device {
>>>  	u32				features;
>>>
>>>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
>>> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
>>>  	u32				options;
>>>
>>>  	struct arm_smmu_cmdq		cmdq;
>>> @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct
>> device *dev,
>>>  				      struct list_head *head)
>>>  {
>>>  	struct iommu_resv_region *region;
>>> +	struct arm_smmu_device *smmu;
>>> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>>>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
>>>
>>> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
>> MSI_IOVA_LENGTH,
>>> -					 prot, IOMMU_RESV_SW_MSI);
>>> -	if (!region)
>>> -		return;
>>> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
>>> +
>>> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
>> &&
>>> +		      dev_is_pci(dev)) {
>>
>> IORT changes are fine to me, I am still no big fan of this supposedly
>> generic option that is _really_ platform specific (in particular as I
>> said before the quirk depends on the PCI host bridge but in this
>> patchset I see no such dependency. In short - the quirk is hooked off
>> the SMMUv3 model which implicitly implies a PCI host bridge
>> configuration IIUC). It is Will and Robin decision though, I am not sure
>> you can make it any tidier (given that on ACPI you may not even have
>> a PCI host bridge specific _HID to base your check above on).
> 
> Thanks Lorenzo. I understand your point. As far as our platform is 
> concerned, I think It is ok to remove the dev_is_pci() check and make
> it generic, if that helps.  I don't think it will harm us other than couple of
>  "HW MSI region resv failed: " logs  for our platform devices.

I think the answer there is that iort_iommu_its_get_resv_regions()
really should distinguish between "this device just doesn't have an ITS
mapping" and "something actually went wrong", such that you don't treat
the former as an error. That's almost certainly cleaner than e.g. trying
to check if dev has an associated MSI domain here before calling it.

Robin.

> 
> Hi Will/Robin,
> Please let me know your thoughts.
> 
> Thanks,
> Shameer
> 
> 

RE: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Robin Murphy [mailto:robin.murphy@arm.com]
> Sent: Tuesday, June 20, 2017 4:16 PM
> To: Shameerali Kolothum Thodi; Lorenzo Pieralisi
> Cc: marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> hanjun.guo@linaro.org; Gabriele Paoloni; John Garry; iommu@lists.linux-
> foundation.org; linux-arm-kernel@lists.infradead.org; linux-
> acpi@vger.kernel.org; devel@acpica.org; Linuxarm; Wangzhou (B);
> Guohanjun (Hanjun Guo)
> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> HiSilicon erratum 161010801
> 
> On 20/06/17 15:07, Shameerali Kolothum Thodi wrote:
> >
> >
> >> -----Original Message-----
> >> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> >> Sent: Tuesday, June 20, 2017 11:29 AM
> >> To: Shameerali Kolothum Thodi
> >> Cc: marc.zyngier@arm.com; sudeep.holla@arm.com;
> will.deacon@arm.com;
> >> robin.murphy@arm.com; hanjun.guo@linaro.org; Gabriele Paoloni; John
> >> Garry; iommu@lists.linux-foundation.org; linux-arm-
> >> kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> devel@acpica.org;
> >> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> >> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> >> HiSilicon erratum 161010801
> >>
> >> Hi Shameer,
> >>
> >> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
> >>> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> >>> platforms Hip06/Hip07 to support the SMMU mappings for MSI
> >> transactions.
> >>>
> >>> On these platforms GICv3 ITS translator is presented with the deviceID
> >>> by extending the MSI payload data to 64 bits to include the deviceID.
> >>> Hence, the PCIe controller on this platforms has to differentiate the
> >>> MSI payload against other DMA payload and has to modify the MSI
> >> payload.
> >>> This basically makes it difficult for this platforms to have a SMMU
> >>> translation for MSI.
> >>>
> >>> This patch implements a ACPI table based quirk to reserve the hw msi
> >>> regions in the smmu-v3 driver which means these address regions will
> >>> not be translated and will be excluded from iova allocations.
> >>>
> >>> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
> >>> ---
> >>>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
> >>>  1 file changed, 24 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-
> smmu-
> >> v3.c
> >>> index abe4b88..f03c63b 100644
> >>> --- a/drivers/iommu/arm-smmu-v3.c
> >>> +++ b/drivers/iommu/arm-smmu-v3.c
> >>> @@ -597,6 +597,7 @@ struct arm_smmu_device {
> >>>  	u32				features;
> >>>
> >>>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> >>> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> >>>  	u32				options;
> >>>
> >>>  	struct arm_smmu_cmdq		cmdq;
> >>> @@ -1904,14 +1905,31 @@ static void
> arm_smmu_get_resv_regions(struct
> >> device *dev,
> >>>  				      struct list_head *head)
> >>>  {
> >>>  	struct iommu_resv_region *region;
> >>> +	struct arm_smmu_device *smmu;
> >>> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> >>>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> >>>
> >>> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> >> MSI_IOVA_LENGTH,
> >>> -					 prot, IOMMU_RESV_SW_MSI);
> >>> -	if (!region)
> >>> -		return;
> >>> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> >>> +
> >>> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> >> &&
> >>> +		      dev_is_pci(dev)) {
> >>
> >> IORT changes are fine to me, I am still no big fan of this supposedly
> >> generic option that is _really_ platform specific (in particular as I
> >> said before the quirk depends on the PCI host bridge but in this
> >> patchset I see no such dependency. In short - the quirk is hooked off
> >> the SMMUv3 model which implicitly implies a PCI host bridge
> >> configuration IIUC). It is Will and Robin decision though, I am not sure
> >> you can make it any tidier (given that on ACPI you may not even have
> >> a PCI host bridge specific _HID to base your check above on).
> >
> > Thanks Lorenzo. I understand your point. As far as our platform is
> > concerned, I think It is ok to remove the dev_is_pci() check and make
> > it generic, if that helps.  I don't think it will harm us other than couple of
> >  "HW MSI region resv failed: " logs  for our platform devices.
> 
> I think the answer there is that iort_iommu_its_get_resv_regions()
> really should distinguish between "this device just doesn't have an ITS
> mapping" and "something actually went wrong", such that you don't treat
> the former as an error. That's almost certainly cleaner than e.g. trying
> to check if dev has an associated MSI domain here before calling it.

If I understood that correctly, the suggestion is to treat cases where device
doesn’t have any ITS node associated with it as "success".

+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{
[....]
+	if (!its_node)
+		return -ENODEV;

ie, return success above from patch 1/2.

Lorenzo, 

Please let me know if that’s a correct thing to do as I am not sure about all the error
scenarios here.

Thanks,
Shameer

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Lorenzo Pieralisi]

On Tue, Jun 20, 2017 at 04:16:18PM +0100, Robin Murphy wrote:
> On 20/06/17 15:07, Shameerali Kolothum Thodi wrote:
> > 
> > 
> >> -----Original Message-----
> >> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> >> Sent: Tuesday, June 20, 2017 11:29 AM
> >> To: Shameerali Kolothum Thodi
> >> Cc: marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> >> robin.murphy@arm.com; hanjun.guo@linaro.org; Gabriele Paoloni; John
> >> Garry; iommu@lists.linux-foundation.org; linux-arm-
> >> kernel@lists.infradead.org; linux-acpi@vger.kernel.org; devel@acpica.org;
> >> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> >> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> >> HiSilicon erratum 161010801
> >>
> >> Hi Shameer,
> >>
> >> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
> >>> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> >>> platforms Hip06/Hip07 to support the SMMU mappings for MSI
> >> transactions.
> >>>
> >>> On these platforms GICv3 ITS translator is presented with the deviceID
> >>> by extending the MSI payload data to 64 bits to include the deviceID.
> >>> Hence, the PCIe controller on this platforms has to differentiate the
> >>> MSI payload against other DMA payload and has to modify the MSI
> >> payload.
> >>> This basically makes it difficult for this platforms to have a SMMU
> >>> translation for MSI.
> >>>
> >>> This patch implements a ACPI table based quirk to reserve the hw msi
> >>> regions in the smmu-v3 driver which means these address regions will
> >>> not be translated and will be excluded from iova allocations.
> >>>
> >>> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
> >>> ---
> >>>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
> >>>  1 file changed, 24 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> >> v3.c
> >>> index abe4b88..f03c63b 100644
> >>> --- a/drivers/iommu/arm-smmu-v3.c
> >>> +++ b/drivers/iommu/arm-smmu-v3.c
> >>> @@ -597,6 +597,7 @@ struct arm_smmu_device {
> >>>  	u32				features;
> >>>
> >>>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> >>> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> >>>  	u32				options;
> >>>
> >>>  	struct arm_smmu_cmdq		cmdq;
> >>> @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct
> >> device *dev,
> >>>  				      struct list_head *head)
> >>>  {
> >>>  	struct iommu_resv_region *region;
> >>> +	struct arm_smmu_device *smmu;
> >>> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> >>>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> >>>
> >>> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> >> MSI_IOVA_LENGTH,
> >>> -					 prot, IOMMU_RESV_SW_MSI);
> >>> -	if (!region)
> >>> -		return;
> >>> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> >>> +
> >>> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> >> &&
> >>> +		      dev_is_pci(dev)) {
> >>
> >> IORT changes are fine to me, I am still no big fan of this supposedly
> >> generic option that is _really_ platform specific (in particular as I
> >> said before the quirk depends on the PCI host bridge but in this
> >> patchset I see no such dependency. In short - the quirk is hooked off
> >> the SMMUv3 model which implicitly implies a PCI host bridge
> >> configuration IIUC). It is Will and Robin decision though, I am not sure
> >> you can make it any tidier (given that on ACPI you may not even have
> >> a PCI host bridge specific _HID to base your check above on).
> > 
> > Thanks Lorenzo. I understand your point. As far as our platform is 
> > concerned, I think It is ok to remove the dev_is_pci() check and make
> > it generic, if that helps.  I don't think it will harm us other than couple of
> >  "HW MSI region resv failed: " logs  for our platform devices.
> 
> I think the answer there is that iort_iommu_its_get_resv_regions()
> really should distinguish between "this device just doesn't have an ITS
> mapping" and "something actually went wrong", such that you don't treat
> the former as an error. That's almost certainly cleaner than e.g. trying
> to check if dev has an associated MSI domain here before calling it.

I would agree, even though this way we would end up reserving the ITS
address regions for all devices mapping to an ITS even though they may
or may not be affected by the quirk (because IIUC that's just a PCI
bridge problem), which is what my point is all about.

It does not seem to be clean-cut, however we slice it.

Thanks,
Lorenzo

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Robin Murphy]

On 20/06/17 16:51, Lorenzo Pieralisi wrote:
> On Tue, Jun 20, 2017 at 04:16:18PM +0100, Robin Murphy wrote:
>> On 20/06/17 15:07, Shameerali Kolothum Thodi wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
>>>> Sent: Tuesday, June 20, 2017 11:29 AM
>>>> To: Shameerali Kolothum Thodi
>>>> Cc: marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
>>>> robin.murphy@arm.com; hanjun.guo@linaro.org; Gabriele Paoloni; John
>>>> Garry; iommu@lists.linux-foundation.org; linux-arm-
>>>> kernel@lists.infradead.org; linux-acpi@vger.kernel.org; devel@acpica.org;
>>>> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
>>>> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
>>>> HiSilicon erratum 161010801
>>>>
>>>> Hi Shameer,
>>>>
>>>> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
>>>>> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
>>>>> platforms Hip06/Hip07 to support the SMMU mappings for MSI
>>>> transactions.
>>>>>
>>>>> On these platforms GICv3 ITS translator is presented with the deviceID
>>>>> by extending the MSI payload data to 64 bits to include the deviceID.
>>>>> Hence, the PCIe controller on this platforms has to differentiate the
>>>>> MSI payload against other DMA payload and has to modify the MSI
>>>> payload.
>>>>> This basically makes it difficult for this platforms to have a SMMU
>>>>> translation for MSI.
>>>>>
>>>>> This patch implements a ACPI table based quirk to reserve the hw msi
>>>>> regions in the smmu-v3 driver which means these address regions will
>>>>> not be translated and will be excluded from iova allocations.
>>>>>
>>>>> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
>>>>> ---
>>>>>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
>>>>>  1 file changed, 24 insertions(+), 5 deletions(-)
>>>>>
>>>>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
>>>> v3.c
>>>>> index abe4b88..f03c63b 100644
>>>>> --- a/drivers/iommu/arm-smmu-v3.c
>>>>> +++ b/drivers/iommu/arm-smmu-v3.c
>>>>> @@ -597,6 +597,7 @@ struct arm_smmu_device {
>>>>>  	u32				features;
>>>>>
>>>>>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
>>>>> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
>>>>>  	u32				options;
>>>>>
>>>>>  	struct arm_smmu_cmdq		cmdq;
>>>>> @@ -1904,14 +1905,31 @@ static void arm_smmu_get_resv_regions(struct
>>>> device *dev,
>>>>>  				      struct list_head *head)
>>>>>  {
>>>>>  	struct iommu_resv_region *region;
>>>>> +	struct arm_smmu_device *smmu;
>>>>> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>>>>>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
>>>>>
>>>>> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
>>>> MSI_IOVA_LENGTH,
>>>>> -					 prot, IOMMU_RESV_SW_MSI);
>>>>> -	if (!region)
>>>>> -		return;
>>>>> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
>>>>> +
>>>>> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
>>>> &&
>>>>> +		      dev_is_pci(dev)) {
>>>>
>>>> IORT changes are fine to me, I am still no big fan of this supposedly
>>>> generic option that is _really_ platform specific (in particular as I
>>>> said before the quirk depends on the PCI host bridge but in this
>>>> patchset I see no such dependency. In short - the quirk is hooked off
>>>> the SMMUv3 model which implicitly implies a PCI host bridge
>>>> configuration IIUC). It is Will and Robin decision though, I am not sure
>>>> you can make it any tidier (given that on ACPI you may not even have
>>>> a PCI host bridge specific _HID to base your check above on).
>>>
>>> Thanks Lorenzo. I understand your point. As far as our platform is 
>>> concerned, I think It is ok to remove the dev_is_pci() check and make
>>> it generic, if that helps.  I don't think it will harm us other than couple of
>>>  "HW MSI region resv failed: " logs  for our platform devices.
>>
>> I think the answer there is that iort_iommu_its_get_resv_regions()
>> really should distinguish between "this device just doesn't have an ITS
>> mapping" and "something actually went wrong", such that you don't treat
>> the former as an error. That's almost certainly cleaner than e.g. trying
>> to check if dev has an associated MSI domain here before calling it.
> 
> I would agree, even though this way we would end up reserving the ITS
> address regions for all devices mapping to an ITS even though they may
> or may not be affected by the quirk (because IIUC that's just a PCI
> bridge problem), which is what my point is all about.

As I'm sure I've said before, there's no foreseeable issue with that.
For DMA, all it means is that we'll preallocate an identity mapping of
the whole ITS rather than dynamically mapping pages of it as devices
request MSIs; no functional change - if anything, it's slightly more
efficient. For VFIO, it just means that the groups for platform devices
will expose an MSI reservation at GITS_TRANSLATER instead of the
arbitrary MSI_IOVA_BASE - again, arguably that's actually desirable
because having all IOMMU groups be consistent with each other makes
userspace's life that little bit simpler.

Robin.

> It does not seem to be clean-cut, however we slice it.
> 
> Thanks,
> Lorenzo
> 

Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Lorenzo Pieralisi]

On Tue, Jun 20, 2017 at 03:39:30PM +0000, Shameerali Kolothum Thodi wrote:
> 
> 
> > -----Original Message-----
> > From: Robin Murphy [mailto:robin.murphy@arm.com]
> > Sent: Tuesday, June 20, 2017 4:16 PM
> > To: Shameerali Kolothum Thodi; Lorenzo Pieralisi
> > Cc: marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> > hanjun.guo@linaro.org; Gabriele Paoloni; John Garry; iommu@lists.linux-
> > foundation.org; linux-arm-kernel@lists.infradead.org; linux-
> > acpi@vger.kernel.org; devel@acpica.org; Linuxarm; Wangzhou (B);
> > Guohanjun (Hanjun Guo)
> > Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> > HiSilicon erratum 161010801
> > 
> > On 20/06/17 15:07, Shameerali Kolothum Thodi wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> > >> Sent: Tuesday, June 20, 2017 11:29 AM
> > >> To: Shameerali Kolothum Thodi
> > >> Cc: marc.zyngier@arm.com; sudeep.holla@arm.com;
> > will.deacon@arm.com;
> > >> robin.murphy@arm.com; hanjun.guo@linaro.org; Gabriele Paoloni; John
> > >> Garry; iommu@lists.linux-foundation.org; linux-arm-
> > >> kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> > devel@acpica.org;
> > >> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> > >> Subject: Re: [PATCH v2 2/2] iommu/arm-smmu-v3:Enable ACPI based
> > >> HiSilicon erratum 161010801
> > >>
> > >> Hi Shameer,
> > >>
> > >> On Mon, Jun 19, 2017 at 04:45:00PM +0100, shameer wrote:
> > >>> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> > >>> platforms Hip06/Hip07 to support the SMMU mappings for MSI
> > >> transactions.
> > >>>
> > >>> On these platforms GICv3 ITS translator is presented with the deviceID
> > >>> by extending the MSI payload data to 64 bits to include the deviceID.
> > >>> Hence, the PCIe controller on this platforms has to differentiate the
> > >>> MSI payload against other DMA payload and has to modify the MSI
> > >> payload.
> > >>> This basically makes it difficult for this platforms to have a SMMU
> > >>> translation for MSI.
> > >>>
> > >>> This patch implements a ACPI table based quirk to reserve the hw msi
> > >>> regions in the smmu-v3 driver which means these address regions will
> > >>> not be translated and will be excluded from iova allocations.
> > >>>
> > >>> Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
> > >>> ---
> > >>>  drivers/iommu/arm-smmu-v3.c | 29 ++++++++++++++++++++++++-----
> > >>>  1 file changed, 24 insertions(+), 5 deletions(-)
> > >>>
> > >>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-
> > smmu-
> > >> v3.c
> > >>> index abe4b88..f03c63b 100644
> > >>> --- a/drivers/iommu/arm-smmu-v3.c
> > >>> +++ b/drivers/iommu/arm-smmu-v3.c
> > >>> @@ -597,6 +597,7 @@ struct arm_smmu_device {
> > >>>  	u32				features;
> > >>>
> > >>>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > >>> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> > >>>  	u32				options;
> > >>>
> > >>>  	struct arm_smmu_cmdq		cmdq;
> > >>> @@ -1904,14 +1905,31 @@ static void
> > arm_smmu_get_resv_regions(struct
> > >> device *dev,
> > >>>  				      struct list_head *head)
> > >>>  {
> > >>>  	struct iommu_resv_region *region;
> > >>> +	struct arm_smmu_device *smmu;
> > >>> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> > >>>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> > >>>
> > >>> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> > >> MSI_IOVA_LENGTH,
> > >>> -					 prot, IOMMU_RESV_SW_MSI);
> > >>> -	if (!region)
> > >>> -		return;
> > >>> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> > >>> +
> > >>> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> > >> &&
> > >>> +		      dev_is_pci(dev)) {
> > >>
> > >> IORT changes are fine to me, I am still no big fan of this supposedly
> > >> generic option that is _really_ platform specific (in particular as I
> > >> said before the quirk depends on the PCI host bridge but in this
> > >> patchset I see no such dependency. In short - the quirk is hooked off
> > >> the SMMUv3 model which implicitly implies a PCI host bridge
> > >> configuration IIUC). It is Will and Robin decision though, I am not sure
> > >> you can make it any tidier (given that on ACPI you may not even have
> > >> a PCI host bridge specific _HID to base your check above on).
> > >
> > > Thanks Lorenzo. I understand your point. As far as our platform is
> > > concerned, I think It is ok to remove the dev_is_pci() check and make
> > > it generic, if that helps.  I don't think it will harm us other than couple of
> > >  "HW MSI region resv failed: " logs  for our platform devices.
> > 
> > I think the answer there is that iort_iommu_its_get_resv_regions()
> > really should distinguish between "this device just doesn't have an ITS
> > mapping" and "something actually went wrong", such that you don't treat
> > the former as an error. That's almost certainly cleaner than e.g. trying
> > to check if dev has an associated MSI domain here before calling it.
> 
> If I understood that correctly, the suggestion is to treat cases where device
> doesn’t have any ITS node associated with it as "success".
> 
> +int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
> +{
> [....]
> +	if (!its_node)
> +		return -ENODEV;
> 
> ie, return success above from patch 1/2.
> 
> Lorenzo, 
> 
> Please let me know if that’s a correct thing to do as I am not sure
> about all the error scenarios here.

Yes it is, you need to add specific return values that you can then
handle in the SMMU callback accordingly (reverting to SW_MSI in case the
device does not map to an ITS).

Thanks,
Lorenzo