[PATCH 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[PATCH 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[shameer]

On certain HiSilicon platforms (Hip06/Hip07) the GIC ITS and
PCIe RC deviates from the standard implementation and this breaks
PCIe MSI functionality when SMMU is enabled.

The HiSilicon erratum 161010801 describes this limitation of certain
HiSilicon platforms to support the SMMU mappings for MSI transactions.
On these platforms GICv3 ITS translator is presented with the deviceID
by extending the MSI payload data to 64 bits to include the deviceID.
Hence, the PCIe controller on this platforms has to differentiate the
MSI payload against other DMA payload and has to modify the MSI payload.
This basically makes it difficult for this platforms to have a SMMU
translation for MSI.

This patch implements a ACPI table based quirk to reserve the hw msi
regions in the smmu-v3 driver which means these address regions will
not be translated and will be excluded from iova allocations.

To implement this quirk, the following changes are incorporated:

1. Added a generic helper function to IORT code to retrieve and
   reserve the HW ITS address regions.
2. Added quirk to SMMUv3 to reserve HW ITS address regions based
   on IORT SMMUv3 model.

This is based on the following patches:
1. https://patchwork.kernel.org/patch/9740733/
2. https://patchwork.kernel.org/patch/9730491/

Thanks,
Shameer

RFCv2 -->PATCH

Incorporated Lorenzo's review comments.

RFC v1 -->v2

Based on Robin's review comments,
-Removed  the generic erratum framework.
-Using IORT/MADT tables to retrieve the ITS base addr instead
 of vendor specific CSRT table.

shameer (2):
  acpi:iort: Add an IORT helper function to reserve HW ITS address
    regions for IOMMU drivers
  iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

 drivers/acpi/arm64/iort.c        | 92 ++++++++++++++++++++++++++++++++++++++--
 drivers/iommu/arm-smmu-v3.c      | 27 +++++++++---
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 ++-
 4 files changed, 119 insertions(+), 10 deletions(-)

-- 
1.9.1

[PATCH 1/2] acpi:iort: Add an IORT helper function to reserve HW ITS address regions for IOMMU drivers

[shameer]

The helper function retrieves ITS address regions through IORT
device <-> ITS mappings and reserves it so that these regions
will not be translated by IOMMU and will be excluded from IOVA
allocations. IOMMU drivers can use this to implement their
.get_resv_regions callback.

Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
---
 drivers/acpi/arm64/iort.c        | 92 ++++++++++++++++++++++++++++++++++++++--
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 ++-
 3 files changed, 97 insertions(+), 5 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index c5fecf9..0ee9965 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -34,6 +34,7 @@
 struct iort_its_msi_chip {
 	struct list_head	list;
 	struct fwnode_handle	*fw_node;
+	phys_addr_t		base_addr;
 	u32			translation_id;
 };
 
@@ -131,14 +132,16 @@ typedef acpi_status (*iort_find_node_callback)
 static DEFINE_SPINLOCK(iort_msi_chip_lock);
 
 /**
- * iort_register_domain_token() - register domain token and related ITS ID
- * to the list from where we can get it back later on.
+ * iort_register_domain_token() - register domain token along with related
+ * ITS ID and base address to the list from where we can get it back later on.
  * @trans_id: ITS ID.
+ * @base: ITS base address.
  * @fw_node: Domain token.
  *
  * Returns: 0 on success, -ENOMEM if no memory when allocating list element
  */
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node)
 {
 	struct iort_its_msi_chip *its_msi_chip;
 
@@ -148,6 +151,7 @@ int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
 
 	its_msi_chip->fw_node = fw_node;
 	its_msi_chip->translation_id = trans_id;
+	its_msi_chip->base_addr = base;
 
 	spin_lock(&iort_msi_chip_lock);
 	list_add(&its_msi_chip->list, &iort_msi_chip_list);
@@ -491,6 +495,24 @@ int iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id)
 	return -ENODEV;
 }
 
+static int iort_find_its_base(u32 its_id, phys_addr_t *base)
+{
+	struct iort_its_msi_chip *its_msi_chip;
+	bool match = false;
+
+	spin_lock(&iort_msi_chip_lock);
+	list_for_each_entry(its_msi_chip, &iort_msi_chip_list, list) {
+		if (its_msi_chip->translation_id == its_id) {
+			*base = its_msi_chip->base_addr;
+			match = true;
+			break;
+		}
+	}
+	spin_unlock(&iort_msi_chip_lock);
+
+	return match ? 0 : -ENODEV;
+}
+
 /**
  * iort_dev_find_its_id() - Find the ITS identifier for a device
  * @dev: The device.
@@ -649,6 +671,68 @@ int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 
 	return err;
 }
+
+/**
+ * iort_iommu_its_get_resv_regions - Reserved region driver helper
+ * @dev: Device from iommu_get_resv_regions()
+ * @list: Reserved region list from iommu_get_resv_regions()
+ *
+ * Returns: 0 on at least one ITS address region reservation,
+ *          appropriate error value otherwise.
+ *
+ * IOMMU drivers can use this to implement their .get_resv_regions callback
+ * for reserving the HW ITS address regions.
+ */
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{
+	int i;
+	struct acpi_iort_node *node;
+	struct acpi_iort_its_group *its;
+	bool resv = false;
+
+	node = iort_find_dev_node(dev);
+	if (!node)
+		return -ENODEV;
+
+	if (dev_is_pci(dev)) {
+		u32 rid;
+
+		pci_for_each_dma_alias(to_pci_dev(dev), __get_pci_rid, &rid);
+		node = iort_node_map_id(node, rid, NULL, IORT_MSI_TYPE);
+
+	} else {
+		for (i = 0; i < node->mapping_count; i++) {
+			node = iort_node_map_platform_id(node, NULL,
+							 IORT_MSI_TYPE, i);
+			if (node)
+				break;
+		}
+	}
+
+	if (!node)
+		return -ENODEV;
+
+	/* Move to ITS specific data */
+	its = (struct acpi_iort_its_group *)node->node_data;
+
+	for (i = 0; i < its->its_count; i++) {
+		phys_addr_t base;
+
+		if (!iort_find_its_base(its->identifiers[i], &base)) {
+			int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+			struct iommu_resv_region *region;
+
+			region = iommu_alloc_resv_region(base, SZ_128K, prot,
+							 IOMMU_RESV_MSI);
+			if (region) {
+				list_add_tail(&region->list, head);
+				resv = true;
+			}
+		}
+	}
+
+	return resv ? 0 : -ENODEV;
+}
 #else
 static inline
 const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
@@ -656,6 +740,8 @@ const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
 static inline
 int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 { return 0; }
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 45ea1933..c45a2ad 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1854,7 +1854,8 @@ static int __init gic_acpi_parse_madt_its(struct acpi_subtable_header *header,
 		return -ENOMEM;
 	}
 
-	err = iort_register_domain_token(its_entry->translation_id, dom_handle);
+	err = iort_register_domain_token(its_entry->translation_id, res.start,
+					 dom_handle);
 	if (err) {
 		pr_err("ITS@%pa: Unable to register GICv3 ITS domain token (ITS ID %d) to IORT\n",
 		       &res.start, its_entry->translation_id);
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 3ff9ace..35cf45c 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -26,7 +26,8 @@
 #define IORT_IRQ_MASK(irq)		(irq & 0xffffffffULL)
 #define IORT_IRQ_TRIGGER_MASK(irq)	((irq >> 32) & 0xffffffffULL)
 
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node);
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node);
 void iort_deregister_domain_token(int trans_id);
 struct fwnode_handle *iort_find_domain_token(int trans_id);
 #ifdef CONFIG_ACPI_IORT
@@ -39,6 +40,7 @@
 /* IOMMU interface */
 void iort_set_dma_mask(struct device *dev);
 const struct iommu_ops *iort_iommu_configure(struct device *dev);
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head);
 #else
 static inline void acpi_iort_init(void) { }
 static inline bool iort_node_match(u8 type) { return false; }
@@ -53,6 +55,9 @@ static inline void iort_set_dma_mask(struct device *dev) { }
 static inline
 const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
+static inline
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 #endif /* __ACPI_IORT_H__ */
-- 
1.9.1

[PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[shameer]

The HiSilicon erratum 161010801 describes the limitation of HiSilicon
platforms Hip06/Hip07 to support the SMMU mappings for MSI transactions.

On these platforms GICv3 ITS translator is presented with the deviceID
by extending the MSI payload data to 64 bits to include the deviceID.
Hence, the PCIe controller on this platforms has to differentiate the
MSI payload against other DMA payload and has to modify the MSI payload.
This basically makes it difficult for this platforms to have a SMMU
translation for MSI.

This patch implements a ACPI table based quirk to reserve the hw msi
regions in the smmu-v3 driver which means these address regions will
not be translated and will be excluded from iova allocations.

Signed-off-by: shameer <shameerali.kolothum.thodi@huawei.com>
---
 drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index abe4b88..2636c85 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -597,6 +597,7 @@ struct arm_smmu_device {
 	u32				features;
 
 #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
+#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
 	u32				options;
 
 	struct arm_smmu_cmdq		cmdq;
@@ -1904,14 +1905,29 @@ static void arm_smmu_get_resv_regions(struct device *dev,
 				      struct list_head *head)
 {
 	struct iommu_resv_region *region;
+	struct arm_smmu_device *smmu;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
 
-	region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
-					 prot, IOMMU_RESV_SW_MSI);
-	if (!region)
-		return;
+	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
 
-	list_add_tail(&region->list, head);
+	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
+		      dev_is_pci(dev)) {
+		int ret;
+
+		ret = iort_iommu_its_get_resv_regions(dev, head);
+		if (ret) {
+			dev_warn(dev, "HW MSI region reserve failed\n");
+			return;
+		}
+	} else {
+		region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
+						 prot, IOMMU_RESV_SW_MSI);
+		if (!region)
+			return;
+
+		list_add_tail(&region->list, head);
+	}
 
 	iommu_dma_get_resv_regions(dev, head);
 }
@@ -2611,6 +2627,7 @@ static void parse_driver_acpi_options(struct acpi_iort_smmu_v3 *iort_smmu,
 	switch (iort_smmu->model) {
 	case ACPI_IORT_SMMU_HISILICON_HI161X:
 		smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
+		smmu->options |= ARM_SMMU_OPT_RESV_HW_MSI;
 		break;
 	default:
 		break;
-- 
1.9.1

Re: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Lorenzo Pieralisi]

On Tue, Jun 13, 2017 at 12:48:29PM +0100, shameer wrote:
> The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> platforms Hip06/Hip07 to support the SMMU mappings for MSI transactions.
> 
> On these platforms GICv3 ITS translator is presented with the deviceID
> by extending the MSI payload data to 64 bits to include the deviceID.
> Hence, the PCIe controller on this platforms has to differentiate the
> MSI payload against other DMA payload and has to modify the MSI payload.
> This basically makes it difficult for this platforms to have a SMMU
> translation for MSI.
> 
> This patch implements a ACPI table based quirk to reserve the hw msi
> regions in the smmu-v3 driver which means these address regions will
> not be translated and will be excluded from iova allocations.
> 
> Signed-off-by: shameer <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
> ---
>  drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++-----
>  1 file changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index abe4b88..2636c85 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -597,6 +597,7 @@ struct arm_smmu_device {
>  	u32				features;
>  
>  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
>  	u32				options;
>  
>  	struct arm_smmu_cmdq		cmdq;
> @@ -1904,14 +1905,29 @@ static void arm_smmu_get_resv_regions(struct device *dev,
>  				      struct list_head *head)
>  {
>  	struct iommu_resv_region *region;
> +	struct arm_smmu_device *smmu;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
>  
> -	region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> -					 prot, IOMMU_RESV_SW_MSI);
> -	if (!region)
> -		return;
> +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
>  
> -	list_add_tail(&region->list, head);
> +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
> +		      dev_is_pci(dev)) {
> +		int ret;
> +
> +		ret = iort_iommu_its_get_resv_regions(dev, head);

This should be made fwnode dependent, it makes precious little
sense to call IORT to reserve regions on a DT based platforms
(I know the ARM_SMMU_OPT_RESV_HW_MSI option is only selected
in ACPI (?) but comment applies regardless - have you prototyped
a DT version too ?).

Lorenzo

> +		if (ret) {
> +			dev_warn(dev, "HW MSI region reserve failed\n");
> +			return;
> +		}
> +	} else {
> +		region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
> +						 prot, IOMMU_RESV_SW_MSI);
> +		if (!region)
> +			return;
> +
> +		list_add_tail(&region->list, head);
> +	}
>  
>  	iommu_dma_get_resv_regions(dev, head);
>  }
> @@ -2611,6 +2627,7 @@ static void parse_driver_acpi_options(struct acpi_iort_smmu_v3 *iort_smmu,
>  	switch (iort_smmu->model) {
>  	case ACPI_IORT_SMMU_HISILICON_HI161X:
>  		smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
> +		smmu->options |= ARM_SMMU_OPT_RESV_HW_MSI;
>  		break;
>  	default:
>  		break;
> -- 
> 1.9.1
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org]
> Sent: Tuesday, June 13, 2017 2:13 PM
> To: Shameerali Kolothum Thodi
> Cc: marc.zyngier-5wv7dgnIgG8@public.gmane.org; sudeep.holla-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org;
> robin.murphy-5wv7dgnIgG8@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; Gabriele Paoloni; John
> Garry; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
> kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; devel-E0kO6a4B6psdnm+yROfE0A@public.gmane.org;
> Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> Subject: Re: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon
> erratum 161010801
> 
> On Tue, Jun 13, 2017 at 12:48:29PM +0100, shameer wrote:
> > The HiSilicon erratum 161010801 describes the limitation of HiSilicon
> > platforms Hip06/Hip07 to support the SMMU mappings for MSI
> transactions.
> >
> > On these platforms GICv3 ITS translator is presented with the deviceID
> > by extending the MSI payload data to 64 bits to include the deviceID.
> > Hence, the PCIe controller on this platforms has to differentiate the
> > MSI payload against other DMA payload and has to modify the MSI
> payload.
> > This basically makes it difficult for this platforms to have a SMMU
> > translation for MSI.
> >
> > This patch implements a ACPI table based quirk to reserve the hw msi
> > regions in the smmu-v3 driver which means these address regions will
> > not be translated and will be excluded from iova allocations.
> >
> > Signed-off-by: shameer <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
> > ---
> >  drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++-----
> >  1 file changed, 22 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> v3.c
> > index abe4b88..2636c85 100644
> > --- a/drivers/iommu/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm-smmu-v3.c
> > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> >  	u32				features;
> >
> >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> >  	u32				options;
> >
> >  	struct arm_smmu_cmdq		cmdq;
> > @@ -1904,14 +1905,29 @@ static void arm_smmu_get_resv_regions(struct
> device *dev,
> >  				      struct list_head *head)
> >  {
> >  	struct iommu_resv_region *region;
> > +	struct arm_smmu_device *smmu;
> > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> >
> > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> MSI_IOVA_LENGTH,
> > -					 prot, IOMMU_RESV_SW_MSI);
> > -	if (!region)
> > -		return;
> > +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> >
> > -	list_add_tail(&region->list, head);
> > +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> &&
> > +		      dev_is_pci(dev)) {
> > +		int ret;
> > +
> > +		ret = iort_iommu_its_get_resv_regions(dev, head);
> 
> This should be made fwnode dependent, it makes precious little sense to call
> IORT to reserve regions on a DT based platforms (I know the
> ARM_SMMU_OPT_RESV_HW_MSI option is only selected in ACPI (?) but
> comment applies regardless - have you prototyped a DT version too ?).

Ok. I will add a check here.  I don't have a DT version for now as ACPI is our top
priority at the moment.

Thanks,
Shameer

RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

Hi Lorenzo,

> -----Original Message-----
> From: linuxarm-bounces-hv44wF8Li93QT0dZR+AlfA@public.gmane.org [mailto:linuxarm-
> bounces-hv44wF8Li93QT0dZR+AlfA@public.gmane.org] On Behalf Of Shameerali Kolothum Thodi
> Sent: Tuesday, June 13, 2017 3:53 PM
> To: Lorenzo Pieralisi
> Cc: marc.zyngier-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org; Linuxarm; linux-
> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org;
> hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; sudeep.holla-5wv7dgnIgG8@public.gmane.org; robin.murphy-5wv7dgnIgG8@public.gmane.org;
> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; devel-E0kO6a4B6psdnm+yROfE0A@public.gmane.org
> Subject: RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon
> erratum 161010801

[...]
 > > > ---
> > >  drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++-----
> > >  1 file changed, 22 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> > v3.c
> > > index abe4b88..2636c85 100644
> > > --- a/drivers/iommu/arm-smmu-v3.c
> > > +++ b/drivers/iommu/arm-smmu-v3.c
> > > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> > >  	u32				features;
> > >
> > >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> > >  	u32				options;
> > >
> > >  	struct arm_smmu_cmdq		cmdq;
> > > @@ -1904,14 +1905,29 @@ static void
> arm_smmu_get_resv_regions(struct
> > device *dev,
> > >  				      struct list_head *head)
> > >  {
> > >  	struct iommu_resv_region *region;
> > > +	struct arm_smmu_device *smmu;
> > > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> > >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> > >
> > > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> > MSI_IOVA_LENGTH,
> > > -					 prot, IOMMU_RESV_SW_MSI);
> > > -	if (!region)
> > > -		return;
> > > +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> > >
> > > -	list_add_tail(&region->list, head);
> > > +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> > &&
> > > +		      dev_is_pci(dev)) {
> > > +		int ret;
> > > +
> > > +		ret = iort_iommu_its_get_resv_regions(dev, head);
> >
> > This should be made fwnode dependent, it makes precious little sense to
> call
> > IORT to reserve regions on a DT based platforms (I know the
> > ARM_SMMU_OPT_RESV_HW_MSI option is only selected in ACPI (?) but
> > comment applies regardless - have you prototyped a DT version too ?).
> 
> Ok. I will add a check here. 

This is what I have in mind. Please take a look and let me know. I will send out
a v2 of this series soon.
....
if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
			dev_is_pci(dev)) {
	int ret = -EINVAL;

	if (!is_of_node(smmu->dev->fwnode))
		ret = iort_iommu_its_get_resv_regions(dev, head);

	if (ret) {
		dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
		return;
	}
} else {
---
Many thanks,
Shameer

Re: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Lorenzo Pieralisi]

On Fri, Jun 16, 2017 at 09:43:52AM +0000, Shameerali Kolothum Thodi wrote:
> Hi Lorenzo,
> 
> > -----Original Message-----
> > From: linuxarm-bounces@huawei.com [mailto:linuxarm-
> > bounces@huawei.com] On Behalf Of Shameerali Kolothum Thodi
> > Sent: Tuesday, June 13, 2017 3:53 PM
> > To: Lorenzo Pieralisi
> > Cc: marc.zyngier@arm.com; will.deacon@arm.com; Linuxarm; linux-
> > acpi@vger.kernel.org; iommu@lists.linux-foundation.org;
> > hanjun.guo@linaro.org; sudeep.holla@arm.com; robin.murphy@arm.com;
> > linux-arm-kernel@lists.infradead.org; devel@acpica.org
> > Subject: RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon
> > erratum 161010801
> 
> [...]
>  > > > ---
> > > >  drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++-----
> > > >  1 file changed, 22 insertions(+), 5 deletions(-)
> > > >
> > > > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-
> > > v3.c
> > > > index abe4b88..2636c85 100644
> > > > --- a/drivers/iommu/arm-smmu-v3.c
> > > > +++ b/drivers/iommu/arm-smmu-v3.c
> > > > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> > > >  	u32				features;
> > > >
> > > >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > > > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> > > >  	u32				options;
> > > >
> > > >  	struct arm_smmu_cmdq		cmdq;
> > > > @@ -1904,14 +1905,29 @@ static void
> > arm_smmu_get_resv_regions(struct
> > > device *dev,
> > > >  				      struct list_head *head)
> > > >  {
> > > >  	struct iommu_resv_region *region;
> > > > +	struct arm_smmu_device *smmu;
> > > > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> > > >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> > > >
> > > > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> > > MSI_IOVA_LENGTH,
> > > > -					 prot, IOMMU_RESV_SW_MSI);
> > > > -	if (!region)
> > > > -		return;
> > > > +	smmu = arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
> > > >
> > > > -	list_add_tail(&region->list, head);
> > > > +	if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI)
> > > &&
> > > > +		      dev_is_pci(dev)) {
> > > > +		int ret;
> > > > +
> > > > +		ret = iort_iommu_its_get_resv_regions(dev, head);
> > >
> > > This should be made fwnode dependent, it makes precious little sense to
> > call
> > > IORT to reserve regions on a DT based platforms (I know the
> > > ARM_SMMU_OPT_RESV_HW_MSI option is only selected in ACPI (?) but
> > > comment applies regardless - have you prototyped a DT version too ?).
> > 
> > Ok. I will add a check here. 
> 
> This is what I have in mind. Please take a look and let me know. I will send out
> a v2 of this series soon.
> ....
> if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
> 			dev_is_pci(dev)) {
> 	int ret = -EINVAL;
> 
> 	if (!is_of_node(smmu->dev->fwnode))
> 		ret = iort_iommu_its_get_resv_regions(dev, head);
> 
> 	if (ret) {
> 		dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
> 		return;
> 	}
> } else {

The fwnode handling is fine, I do not like much the:

dev_is_pci()

check because it relies on implicit knowledge of the platform and
the quirks you need (ie you know that it is _just_ a PCI RC quirk
implicitly), the logic behind reserving the regions is a bit
convoluted and not easy to understand at all.

Let me try to rephrase it: you know, through an SMMU model number,
that your PCI RC handles MSI in a specific way, but by reading the
code above this is not clear at all, at least to me. This is a PCI
RC quirk but it does not depend on any PCI RC specific firmware binding
whatsoever, that's what is a bit hard to understand.

Anyway you can post the patches and we will take it from there to
see if there is a way to improve it.

Thanks,
Lorenzo

RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon erratum 161010801

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> Sent: Friday, June 16, 2017 12:17 PM
> To: Shameerali Kolothum Thodi
> Cc: marc.zyngier@arm.com; will.deacon@arm.com; Linuxarm; linux-
> acpi@vger.kernel.org; iommu@lists.linux-foundation.org;
> hanjun.guo@linaro.org; sudeep.holla@arm.com; robin.murphy@arm.com;
> linux-arm-kernel@lists.infradead.org; devel@acpica.org
> Subject: Re: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based HiSilicon
> erratum 161010801
> 
> On Fri, Jun 16, 2017 at 09:43:52AM +0000, Shameerali Kolothum Thodi wrote:
> > Hi Lorenzo,
> >
> > > -----Original Message-----
> > > From: linuxarm-bounces@huawei.com [mailto:linuxarm-
> > > bounces@huawei.com] On Behalf Of Shameerali Kolothum Thodi
> > > Sent: Tuesday, June 13, 2017 3:53 PM
> > > To: Lorenzo Pieralisi
> > > Cc: marc.zyngier@arm.com; will.deacon@arm.com; Linuxarm; linux-
> > > acpi@vger.kernel.org; iommu@lists.linux-foundation.org;
> > > hanjun.guo@linaro.org; sudeep.holla@arm.com;
> robin.murphy@arm.com;
> > > linux-arm-kernel@lists.infradead.org; devel@acpica.org
> > > Subject: RE: [PATCH 2/2] iommu/arm-smmu-v3:Enable ACPI based
> HiSilicon
> > > erratum 161010801
> >
> > [...]
> >  > > > ---
> > > > >  drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++++++----
> -
> > > > >  1 file changed, 22 insertions(+), 5 deletions(-)
> > > > >
> > > > > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-
> smmu-
> > > > v3.c
> > > > > index abe4b88..2636c85 100644
> > > > > --- a/drivers/iommu/arm-smmu-v3.c
> > > > > +++ b/drivers/iommu/arm-smmu-v3.c
> > > > > @@ -597,6 +597,7 @@ struct arm_smmu_device {
> > > > >  	u32				features;
> > > > >
> > > > >  #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
> > > > > +#define ARM_SMMU_OPT_RESV_HW_MSI	(1 << 1)
> > > > >  	u32				options;
> > > > >
> > > > >  	struct arm_smmu_cmdq		cmdq;
> > > > > @@ -1904,14 +1905,29 @@ static void
> > > arm_smmu_get_resv_regions(struct
> > > > device *dev,
> > > > >  				      struct list_head *head)
> > > > >  {
> > > > >  	struct iommu_resv_region *region;
> > > > > +	struct arm_smmu_device *smmu;
> > > > > +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> > > > >  	int prot = IOMMU_WRITE | IOMMU_NOEXEC |
> IOMMU_MMIO;
> > > > >
> > > > > -	region = iommu_alloc_resv_region(MSI_IOVA_BASE,
> > > > MSI_IOVA_LENGTH,
> > > > > -					 prot,
> IOMMU_RESV_SW_MSI);
> > > > > -	if (!region)
> > > > > -		return;
> > > > > +	smmu = arm_smmu_get_by_fwnode(fwspec-
> >iommu_fwnode);
> > > > >
> > > > > -	list_add_tail(&region->list, head);
> > > > > +	if (smmu && (smmu->options &
> ARM_SMMU_OPT_RESV_HW_MSI)
> > > > &&
> > > > > +		      dev_is_pci(dev)) {
> > > > > +		int ret;
> > > > > +
> > > > > +		ret = iort_iommu_its_get_resv_regions(dev, head);
> > > >
> > > > This should be made fwnode dependent, it makes precious little sense
> to
> > > call
> > > > IORT to reserve regions on a DT based platforms (I know the
> > > > ARM_SMMU_OPT_RESV_HW_MSI option is only selected in ACPI (?)
> but
> > > > comment applies regardless - have you prototyped a DT version too ?).
> > >
> > > Ok. I will add a check here.
> >
> > This is what I have in mind. Please take a look and let me know. I will send
> out
> > a v2 of this series soon.
> > ....
> > if (smmu && (smmu->options & ARM_SMMU_OPT_RESV_HW_MSI) &&
> > 			dev_is_pci(dev)) {
> > 	int ret = -EINVAL;
> >
> > 	if (!is_of_node(smmu->dev->fwnode))
> > 		ret = iort_iommu_its_get_resv_regions(dev, head);
> >
> > 	if (ret) {
> > 		dev_warn(dev, "HW MSI region resv failed: %d\n", ret);
> > 		return;
> > 	}
> > } else {
> 
> The fwnode handling is fine, I do not like much the:
> 
> dev_is_pci()
> 
> check because it relies on implicit knowledge of the platform and
> the quirks you need (ie you know that it is _just_ a PCI RC quirk
> implicitly), the logic behind reserving the regions is a bit
> convoluted and not easy to understand at all.
> 
> Let me try to rephrase it: you know, through an SMMU model number,
> that your PCI RC handles MSI in a specific way, but by reading the
> code above this is not clear at all, at least to me. This is a PCI
> RC quirk but it does not depend on any PCI RC specific firmware binding
> whatsoever, that's what is a bit hard to understand.

Right.  I had a thought of limiting the iort_iommu_its_get_resv_regions
to pci devices only. Just like the way iommu_dma_get_resv_regions()
does now. But not sure there will ever be a platform which requires this
for non-pci devices as well.

Thanks,
Shameer