[PATCH V9 00/11] IOMMU probe deferral support

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
against mainline and linux-next. There is a conflict with patch#9
against iommu-next, but that should go away eventually as iommu-next
is rebased against 4.11-rc1.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [6]. 

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 308 insertions(+), 336 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
against mainline and linux-next. There is a conflict with patch#9
against iommu-next, but that should go away eventually as iommu-next
is rebased against 4.11-rc1.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [6]. 

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 308 insertions(+), 336 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
against mainline and linux-next. There is a conflict with patch#9
against iommu-next, but that should go away eventually as iommu-next
is rebased against 4.11-rc1.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [6]. 

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 308 insertions(+), 336 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 02/11] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 02/11] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 03/11] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 03/11] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 04/11] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 04/11] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 05/11] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 05/11] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 06/11] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5344db5..2aa4ebb 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dfc9a27..5a8dd43 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 0977317..6d08016 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -728,6 +728,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 06/11] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5344db5..2aa4ebb 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dfc9a27..5a8dd43 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 0977317..6d08016 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -728,6 +728,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..ba51ca6 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..ba51ca6 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..ba51ca6 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 3dd9ec3..a1000fb 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..823b005 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2a9a5de 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 673acda..60f0aad 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -765,8 +765,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 3dd9ec3..a1000fb 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..823b005 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2a9a5de 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 673acda..60f0aad 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -765,8 +765,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 09/11] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's iommu is probed, the notifier trick required to handle the
early setup of dma_ops before the iommu group gets created is not
required. So removing the notifier's here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 81cdb2e..b465759 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 09/11] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's iommu is probed, the notifier trick required to handle the
early setup of dma_ops before the iommu group gets created is not
required. So removing the notifier's here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 81cdb2e..b465759 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 10/11] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 5806a6a..9497800 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index abf6496..be53b02 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2075,6 +2075,23 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
+static void arm_smmu_bus_init(void)
+{
+	/* Oh, for a proper bus abstraction */
+	if (!iommu_present(&platform_bus_type))
+		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+	if (!iommu_present(&amba_bustype))
+		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+	if (!iommu_present(&pci_bus_type)) {
+		pci_request_acs();
+		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+	}
+#endif
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2180,21 +2197,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-	}
-#endif
+	/*
+	 * For ACPI and generic DT bindings, an SMMU will be probed before
+	 * any device which might need it, so we want the bus ops in place
+	 * ready to handle default domain setup as soon as any SMMU exists.
+	 */
+	if (!using_legacy_binding)
+		arm_smmu_bus_init();
+
+	return 0;
+}
+
+/*
+ * With the legacy DT binding in play, though, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		arm_smmu_bus_init();
 	return 0;
 }
+device_initcall_sync(arm_smmu_legacy_bus_init);
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
@@ -2219,56 +2245,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 10/11] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 5806a6a..9497800 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index abf6496..be53b02 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2075,6 +2075,23 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
+static void arm_smmu_bus_init(void)
+{
+	/* Oh, for a proper bus abstraction */
+	if (!iommu_present(&platform_bus_type))
+		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+	if (!iommu_present(&amba_bustype))
+		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+	if (!iommu_present(&pci_bus_type)) {
+		pci_request_acs();
+		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+	}
+#endif
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2180,21 +2197,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-	}
-#endif
+	/*
+	 * For ACPI and generic DT bindings, an SMMU will be probed before
+	 * any device which might need it, so we want the bus ops in place
+	 * ready to handle default domain setup as soon as any SMMU exists.
+	 */
+	if (!using_legacy_binding)
+		arm_smmu_bus_init();
+
+	return 0;
+}
+
+/*
+ * With the legacy DT binding in play, though, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		arm_smmu_bus_init();
 	return 0;
 }
+device_initcall_sync(arm_smmu_legacy_bus_init);
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
@@ -2219,56 +2245,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 11/11] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index a1000fb..36a9abf 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -992,6 +992,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 0968d13..9faa26c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V9 11/11] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index a1000fb..36a9abf 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -992,6 +992,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 0968d13..9faa26c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Zhou Wang]

On 2017/3/10 3:00, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
> against mainline and linux-next. There is a conflict with patch#9
> against iommu-next, but that should go away eventually as iommu-next
> is rebased against 4.11-rc1.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Hi Sricharan,

I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
D05 board(with Intel 82599 networking card). It failed.

After I used:

echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
echo 0002:81:10.0 > /sys/bus/pci/drivers_probe

to bind vfio-pci driver to Intel 82599 networking card VF.

I got log in host:
[...]
[  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
[  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
[  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
[  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
[  414.448713] iommu: Adding device 0002:81:10.0 to group 4
[  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
[  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
[  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
[  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
[  414.451417] ixgbevf 0002:81:10.0: MAC: 1
[  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
[  414.464271] VFIO - User Level meta-driver version: 0.3
[  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
[  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
[  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here
[  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
[  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Then I tried to boot up VM using:

qemu-system-aarch64 \
-machine virt,gic-version=3 \
-enable-kvm \
-cpu host \
-m 1024 \
-kernel ./Image_vm \
-initrd ./minifs.cpio.gz \
-nographic \
-net none -device vfio-pci,host=0002:81:10.0,id=net0

I got this error:

root@ubuntu:~/scripts# ./qemu_run.sh
[  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
[  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
[  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
[  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
[  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
[  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
[  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
[  402.646338] sp : ffff8017e5a8fb50
[  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
[  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
[  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
[  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
[  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
[  402.676134] x19: 0000000000000001 x18: 0000000000783488
[  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
[  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
[  402.692031] x13: 0000000000783060 x12: 0000000000000012
[  402.697330] x11: 0000000000000001 x10: 0000000000000900
[  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
[  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
[  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
[  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
[  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
[  402.729123]
[  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
[  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
[  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
[  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
[  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
[  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
[  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
[  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
[  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
[  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
[  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
[  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
[  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
[  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
[  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
[  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
[  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
[  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
[  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
[  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
[  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
[  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
[  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
[  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
[  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
[  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
[  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
[  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
[  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
[  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
[  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
[  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
[  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
[  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
[  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
[  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
[  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
[  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
[  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
[  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  403.040775] Call trace:
[  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
[  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
[  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
[  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
[  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
[  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
[  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
[  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
[  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
[  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
[  403.119970] faa0: ffff000008207410 0000ffff967caa50
[  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
[  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
[  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
[  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
[  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
[  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
[  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
[  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
[  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
[  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
[  403.192351] ---[ end trace 0bda8b3549cfd903 ]---

Dropping this series, 82599 VF can work well in QEMU.

P.S.

To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:

@@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
  	arm_smmu_sync_ste_for_sid(smmu, sid);

  	/* It's likely that we'll want to use the new STE soon */
 -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
 +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
  }

  static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)


 @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
  	msg->address_hi		= upper_32_bits(addr);
  	msg->data		= its_get_event_id(d);

 -	iommu_dma_map_msi_msg(d->irq, msg);
 +	//iommu_dma_map_msi_msg(d->irq, msg);
  }

  static struct irq_chip its_irq_chip = {

@@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
  	return ret;
  }

 +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
 +{
 +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
 +	int ret = acs_flags & ~flags ? 0 : 1;
 +
 +	return ret;
 +}
 +
  /*
   * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
 @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
  	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
  	/* Cavium ThunderX */
  	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
 +	/* Hisilicon Hip05/Hip06 root ports */
 +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
  	{ 0 }
  };

If you have time, could you please take a look at this issue?

Thanks,
Zhou

> 
> Previous post of this series [6]. 
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (3):
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |   7 +-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  34 ++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |   3 +
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 308 insertions(+), 336 deletions(-)
> 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Zhou Wang]

On 2017/3/10 3:00, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
> against mainline and linux-next. There is a conflict with patch#9
> against iommu-next, but that should go away eventually as iommu-next
> is rebased against 4.11-rc1.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Hi Sricharan,

I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
D05 board(with Intel 82599 networking card). It failed.

After I used:

echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
echo 0002:81:10.0 > /sys/bus/pci/drivers_probe

to bind vfio-pci driver to Intel 82599 networking card VF.

I got log in host:
[...]
[  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
[  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
[  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
[  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
[  414.448713] iommu: Adding device 0002:81:10.0 to group 4
[  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
[  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
[  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
[  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
[  414.451417] ixgbevf 0002:81:10.0: MAC: 1
[  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
[  414.464271] VFIO - User Level meta-driver version: 0.3
[  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
[  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
[  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here
[  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
[  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Then I tried to boot up VM using:

qemu-system-aarch64 \
-machine virt,gic-version=3 \
-enable-kvm \
-cpu host \
-m 1024 \
-kernel ./Image_vm \
-initrd ./minifs.cpio.gz \
-nographic \
-net none -device vfio-pci,host=0002:81:10.0,id=net0

I got this error:

root@ubuntu:~/scripts# ./qemu_run.sh
[  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
[  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
[  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
[  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
[  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
[  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
[  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
[  402.646338] sp : ffff8017e5a8fb50
[  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
[  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
[  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
[  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
[  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
[  402.676134] x19: 0000000000000001 x18: 0000000000783488
[  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
[  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
[  402.692031] x13: 0000000000783060 x12: 0000000000000012
[  402.697330] x11: 0000000000000001 x10: 0000000000000900
[  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
[  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
[  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
[  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
[  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
[  402.729123]
[  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
[  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
[  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
[  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
[  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
[  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
[  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
[  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
[  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
[  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
[  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
[  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
[  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
[  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
[  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
[  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
[  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
[  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
[  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
[  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
[  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
[  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
[  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
[  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
[  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
[  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
[  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
[  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
[  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
[  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
[  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
[  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
[  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
[  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
[  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
[  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
[  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
[  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
[  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
[  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  403.040775] Call trace:
[  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
[  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
[  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
[  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
[  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
[  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
[  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
[  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
[  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
[  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
[  403.119970] faa0: ffff000008207410 0000ffff967caa50
[  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
[  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
[  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
[  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
[  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
[  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
[  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
[  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
[  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
[  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
[  403.192351] ---[ end trace 0bda8b3549cfd903 ]---

Dropping this series, 82599 VF can work well in QEMU.

P.S.

To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:

@@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
  	arm_smmu_sync_ste_for_sid(smmu, sid);

  	/* It's likely that we'll want to use the new STE soon */
 -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
 +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
  }

  static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)


 @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
  	msg->address_hi		= upper_32_bits(addr);
  	msg->data		= its_get_event_id(d);

 -	iommu_dma_map_msi_msg(d->irq, msg);
 +	//iommu_dma_map_msi_msg(d->irq, msg);
  }

  static struct irq_chip its_irq_chip = {

@@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
  	return ret;
  }

 +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
 +{
 +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
 +	int ret = acs_flags & ~flags ? 0 : 1;
 +
 +	return ret;
 +}
 +
  /*
   * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
 @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
  	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
  	/* Cavium ThunderX */
  	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
 +	/* Hisilicon Hip05/Hip06 root ports */
 +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
  	{ 0 }
  };

If you have time, could you please take a look at this issue?

Thanks,
Zhou

> 
> Previous post of this series [6]. 
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (3):
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |   7 +-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  34 ++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |   3 +
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 308 insertions(+), 336 deletions(-)
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Zhou Wang]

On 2017/3/10 3:00, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
> against mainline and linux-next. There is a conflict with patch#9
> against iommu-next, but that should go away eventually as iommu-next
> is rebased against 4.11-rc1.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Hi Sricharan,

I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
D05 board(with Intel 82599 networking card). It failed.

After I used:

echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
echo 0002:81:10.0 > /sys/bus/pci/drivers_probe

to bind vfio-pci driver to Intel 82599 networking card VF.

I got log in host:
[...]
[  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
[  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
[  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
[  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
[  414.448713] iommu: Adding device 0002:81:10.0 to group 4
[  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
[  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
[  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
[  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
[  414.451417] ixgbevf 0002:81:10.0: MAC: 1
[  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
[  414.464271] VFIO - User Level meta-driver version: 0.3
[  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
[  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
[  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here
[  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
[  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Then I tried to boot up VM using:

qemu-system-aarch64 \
-machine virt,gic-version=3 \
-enable-kvm \
-cpu host \
-m 1024 \
-kernel ./Image_vm \
-initrd ./minifs.cpio.gz \
-nographic \
-net none -device vfio-pci,host=0002:81:10.0,id=net0

I got this error:

root at ubuntu:~/scripts# ./qemu_run.sh
[  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
[  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
[  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
[  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
[  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
[  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  402.633399] LR is@arm_smmu_install_ste_for_dev+0x9c/0xc0
[  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
[  402.646338] sp : ffff8017e5a8fb50
[  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
[  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
[  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
[  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
[  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
[  402.676134] x19: 0000000000000001 x18: 0000000000783488
[  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
[  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
[  402.692031] x13: 0000000000783060 x12: 0000000000000012
[  402.697330] x11: 0000000000000001 x10: 0000000000000900
[  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
[  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
[  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
[  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
[  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
[  402.729123]
[  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
[  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
[  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
[  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
[  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
[  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
[  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
[  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
[  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
[  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
[  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
[  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
[  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
[  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
[  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
[  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
[  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
[  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
[  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
[  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
[  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
[  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
[  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
[  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
[  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
[  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
[  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
[  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
[  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
[  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
[  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
[  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
[  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
[  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
[  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
[  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
[  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
[  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
[  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
[  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  403.040775] Call trace:
[  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
[  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
[  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
[  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
[  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
[  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
[  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
[  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
[  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
[  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
[  403.119970] faa0: ffff000008207410 0000ffff967caa50
[  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
[  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
[  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
[  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
[  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
[  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
[  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
[  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
[  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
[  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
[  403.192351] ---[ end trace 0bda8b3549cfd903 ]---

Dropping this series, 82599 VF can work well in QEMU.

P.S.

To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:

@@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
  	arm_smmu_sync_ste_for_sid(smmu, sid);

  	/* It's likely that we'll want to use the new STE soon */
 -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
 +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
 +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
  }

  static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)


 @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
  	msg->address_hi		= upper_32_bits(addr);
  	msg->data		= its_get_event_id(d);

 -	iommu_dma_map_msi_msg(d->irq, msg);
 +	//iommu_dma_map_msi_msg(d->irq, msg);
  }

  static struct irq_chip its_irq_chip = {

@@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
  	return ret;
  }

 +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
 +{
 +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
 +	int ret = acs_flags & ~flags ? 0 : 1;
 +
 +	return ret;
 +}
 +
  /*
   * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
 @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
  	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
  	/* Cavium ThunderX */
  	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
 +	/* Hisilicon Hip05/Hip06 root ports */
 +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
  	{ 0 }
  };

If you have time, could you please take a look at this issue?

Thanks,
Zhou

> 
> Previous post of this series [6]. 
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (3):
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |   7 +-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  34 ++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |   3 +
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 308 insertions(+), 336 deletions(-)
> 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Zhou,

On 3/24/2017 9:23 AM, Zhou Wang wrote:
> On 2017/3/10 3:00, Sricharan R wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
>> against mainline and linux-next. There is a conflict with patch#9
>> against iommu-next, but that should go away eventually as iommu-next
>> is rebased against 4.11-rc1.
>>
>> * Tested with platform and pci devices for probe deferral
>>   and reprobe on arm64 based platform.
>
> Hi Sricharan,
>
> I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
> D05 board(with Intel 82599 networking card). It failed.
>
> After I used:
>
> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>
> to bind vfio-pci driver to Intel 82599 networking card VF.
>
> I got log in host:
> [...]
> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
> [  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
> [  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
> [  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
> [  414.448713] iommu: Adding device 0002:81:10.0 to group 4
> [  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
> [  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
> [  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
> [  414.451417] ixgbevf 0002:81:10.0: MAC: 1
> [  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
> [  414.464271] VFIO - User Level meta-driver version: 0.3
> [  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
> [  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here

Looks like this triggers the start of the bug.
So the below check in iommu_dma_init_domain fails,

         if (domain->geometry.force_aperture) {
                 if (base > domain->geometry.aperture_end ||
                     base + size <= domain->geometry.aperture_start) {

and the rest goes out of sync after that. Can you print out the
base, aperture_start and end values to see why the check fails ?

The change that this series does is trying to add the
dma/iommu ops to the device after the iommu is actually probed.
So in your working case, does the device initially gets hooked
to iommu_ops and the above same check passes in working case ?

Regards,
  Sricharan

> [  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
> [  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX
>
> Then I tried to boot up VM using:
>
> qemu-system-aarch64 \
> -machine virt,gic-version=3 \
> -enable-kvm \
> -cpu host \
> -m 1024 \
> -kernel ./Image_vm \
> -initrd ./minifs.cpio.gz \
> -nographic \
> -net none -device vfio-pci,host=0002:81:10.0,id=net0
>
> I got this error:
>
> root@ubuntu:~/scripts# ./qemu_run.sh
> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
> [  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
> [  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
> [  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
> [  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
> [  402.646338] sp : ffff8017e5a8fb50
> [  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
> [  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
> [  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
> [  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
> [  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
> [  402.676134] x19: 0000000000000001 x18: 0000000000783488
> [  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
> [  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
> [  402.692031] x13: 0000000000783060 x12: 0000000000000012
> [  402.697330] x11: 0000000000000001 x10: 0000000000000900
> [  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
> [  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
> [  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
> [  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
> [  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
> [  402.729123]
> [  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
> [  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
> [  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
> [  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
> [  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
> [  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
> [  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
> [  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
> [  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
> [  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
> [  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
> [  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
> [  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
> [  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
> [  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
> [  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
> [  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
> [  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
> [  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
> [  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
> [  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
> [  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
> [  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
> [  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
> [  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
> [  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
> [  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
> [  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
> [  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
> [  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
> [  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
> [  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
> [  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
> [  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
> [  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
> [  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
> [  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
> [  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
> [  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
> [  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [  403.040775] Call trace:
> [  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
> [  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
> [  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
> [  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
> [  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
> [  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
> [  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
> [  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
> [  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
> [  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
> [  403.119970] faa0: ffff000008207410 0000ffff967caa50
> [  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
> [  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
> [  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
> [  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
> [  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
> [  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
> [  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
> [  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
> [  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
> [  403.192351] ---[ end trace 0bda8b3549cfd903 ]---
>
> Dropping this series, 82599 VF can work well in QEMU.
>
> P.S.
>
> To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:
>
> @@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
>   	arm_smmu_sync_ste_for_sid(smmu, sid);
>
>   	/* It's likely that we'll want to use the new STE soon */
>  -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>  +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>   }
>
>   static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)
>
>
>  @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
>   	msg->address_hi		= upper_32_bits(addr);
>   	msg->data		= its_get_event_id(d);
>
>  -	iommu_dma_map_msi_msg(d->irq, msg);
>  +	//iommu_dma_map_msi_msg(d->irq, msg);
>   }
>
>   static struct irq_chip its_irq_chip = {
>
> @@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
>   	return ret;
>   }
>
>  +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
>  +{
>  +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
>  +	int ret = acs_flags & ~flags ? 0 : 1;
>  +
>  +	return ret;
>  +}
>  +
>   /*
>    * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
>    * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
>  @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
>   	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
>   	/* Cavium ThunderX */
>   	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
>  +	/* Hisilicon Hip05/Hip06 root ports */
>  +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
>   	{ 0 }
>   };
>
> If you have time, could you please take a look at this issue?
>
> Thanks,
> Zhou
>
>>
>> Previous post of this series [6].
>>
>>  [V9]
>>      * Rebased on top of 4.11-rc1.
>>
>>      * Merged Robin's fixes for legacy binding issue,
>>        pci devices with no iommu-map property.
>>
>>  [V8]
>>      * Picked up all the acks and tested tags from Marek and
>>        Hanjun for DT and ACPI patches respectively, since
>>        no functional changes was done.
>>
>>      * Addressed Minor comments Sinan and Bjorn.
>>
>>      * Added Robin's fix for fixing the deferencing NULL for
>>        of_iommu_table after init in patch #2.
>>
>>      * Rebased it on top of linux-next
>>
>>  [V7]
>>      * Updated the subject and commit log for patch #6 as per
>>        comments from Lorenzo. No functional changes.
>>
>>  [V6]
>>      * Fixed a bug in dma_configure function pointed out by
>>        Robin.
>>      * Reordered the patches as per comments from Robin and
>>        Lorenzo.
>>      * Added Tags.
>>
>>  [V5]
>>      * Reworked the pci configuration code hanging outside and
>>        pushed it to dma_configure as in PATCH#5,6,7.
>>        Also added a couple of patches that Lorenzo provided for
>>        correcting the Probe deferring mechanism in case of
>>        ACPI devices from here [5].
>>
>>  [V4]
>>      * Took the reworked patches [2] from Robin's branch and
>>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
>>
>>      * Added the patches for moving the dma ops configuration of
>>        acpi based devices to probe time as well.
>>  [V3]
>>      * Removed the patch to split dma_masks/dma_ops configuration
>>        separately based on review comments that both masks and ops are
>>        required only during the device probe time.
>>
>>      * Reworked the series based on Generic DT bindings series.
>>
>>      * Added call to iommu's remove_device in the cleanup path for arm and
>>        arm64.
>>
>>      * Removed the notifier trick in arm64 to handle early device
>>        registration.
>>
>>      * Added reset of dma_ops in cleanup path for arm based on comments.
>>
>>      * Fixed the pci_iommu_configure path and tested with PCI device as
>>        well.
>>
>>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>>        last post.
>>
>>      * Fixed few other cosmetic comments.
>>
>>  [V2]
>>      * Updated the Initial post to call dma_configure/deconfigure from
>>        generic code
>>
>>      * Added iommu add_device callback from of_iommu_configure path
>>
>>  [V1]
>>      * Initial post from Laurent Pinchart [1]
>>
>> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
>> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
>> [3] https://lkml.org/lkml/2016/11/21/141
>> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
>> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
>> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
>> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
>>
>> Laurent Pinchart (3):
>>   of: dma: Move range size workaround to of_dma_get_range()
>>   of: dma: Make of_dma_deconfigure() public
>>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
>>
>> Lorenzo Pieralisi (2):
>>   ACPI/IORT: Add function to check SMMUs drivers presence
>>   ACPI/IORT: Remove linker section for IORT entries probing
>>
>> Robin Murphy (3):
>>   iommu/of: Refactor of_iommu_configure() for error handling
>>   iommu/of: Prepare for deferred IOMMU configuration
>>   iommu/arm-smmu: Clean up early-probing workarounds
>>
>> Sricharan R (3):
>>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>>     bus devices
>>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>>     error
>>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>>     of dma_ops
>>
>>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>>  drivers/acpi/glue.c               |   5 --
>>  drivers/acpi/scan.c               |   7 +-
>>  drivers/base/dd.c                 |   9 +++
>>  drivers/base/dma-mapping.c        |  41 +++++++++++
>>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>>  drivers/of/address.c              |  20 +++++-
>>  drivers/of/device.c               |  34 ++++-----
>>  drivers/of/platform.c             |  10 +--
>>  drivers/pci/probe.c               |  28 --------
>>  include/acpi/acpi_bus.h           |   2 +-
>>  include/asm-generic/vmlinux.lds.h |   1 -
>>  include/linux/acpi.h              |   7 +-
>>  include/linux/acpi_iort.h         |   3 -
>>  include/linux/dma-mapping.h       |   3 +
>>  include/linux/of_device.h         |  10 ++-
>>  19 files changed, 308 insertions(+), 336 deletions(-)
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Zhou,

On 3/24/2017 9:23 AM, Zhou Wang wrote:
> On 2017/3/10 3:00, Sricharan R wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
>> against mainline and linux-next. There is a conflict with patch#9
>> against iommu-next, but that should go away eventually as iommu-next
>> is rebased against 4.11-rc1.
>>
>> * Tested with platform and pci devices for probe deferral
>>   and reprobe on arm64 based platform.
>
> Hi Sricharan,
>
> I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
> D05 board(with Intel 82599 networking card). It failed.
>
> After I used:
>
> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>
> to bind vfio-pci driver to Intel 82599 networking card VF.
>
> I got log in host:
> [...]
> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
> [  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
> [  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
> [  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
> [  414.448713] iommu: Adding device 0002:81:10.0 to group 4
> [  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
> [  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
> [  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
> [  414.451417] ixgbevf 0002:81:10.0: MAC: 1
> [  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
> [  414.464271] VFIO - User Level meta-driver version: 0.3
> [  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
> [  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here

Looks like this triggers the start of the bug.
So the below check in iommu_dma_init_domain fails,

         if (domain->geometry.force_aperture) {
                 if (base > domain->geometry.aperture_end ||
                     base + size <= domain->geometry.aperture_start) {

and the rest goes out of sync after that. Can you print out the
base, aperture_start and end values to see why the check fails ?

The change that this series does is trying to add the
dma/iommu ops to the device after the iommu is actually probed.
So in your working case, does the device initially gets hooked
to iommu_ops and the above same check passes in working case ?

Regards,
  Sricharan

> [  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
> [  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX
>
> Then I tried to boot up VM using:
>
> qemu-system-aarch64 \
> -machine virt,gic-version=3 \
> -enable-kvm \
> -cpu host \
> -m 1024 \
> -kernel ./Image_vm \
> -initrd ./minifs.cpio.gz \
> -nographic \
> -net none -device vfio-pci,host=0002:81:10.0,id=net0
>
> I got this error:
>
> root@ubuntu:~/scripts# ./qemu_run.sh
> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
> [  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
> [  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
> [  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
> [  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
> [  402.646338] sp : ffff8017e5a8fb50
> [  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
> [  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
> [  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
> [  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
> [  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
> [  402.676134] x19: 0000000000000001 x18: 0000000000783488
> [  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
> [  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
> [  402.692031] x13: 0000000000783060 x12: 0000000000000012
> [  402.697330] x11: 0000000000000001 x10: 0000000000000900
> [  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
> [  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
> [  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
> [  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
> [  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
> [  402.729123]
> [  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
> [  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
> [  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
> [  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
> [  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
> [  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
> [  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
> [  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
> [  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
> [  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
> [  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
> [  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
> [  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
> [  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
> [  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
> [  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
> [  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
> [  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
> [  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
> [  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
> [  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
> [  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
> [  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
> [  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
> [  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
> [  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
> [  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
> [  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
> [  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
> [  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
> [  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
> [  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
> [  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
> [  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
> [  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
> [  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
> [  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
> [  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
> [  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
> [  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [  403.040775] Call trace:
> [  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
> [  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
> [  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
> [  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
> [  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
> [  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
> [  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
> [  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
> [  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
> [  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
> [  403.119970] faa0: ffff000008207410 0000ffff967caa50
> [  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
> [  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
> [  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
> [  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
> [  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
> [  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
> [  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
> [  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
> [  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
> [  403.192351] ---[ end trace 0bda8b3549cfd903 ]---
>
> Dropping this series, 82599 VF can work well in QEMU.
>
> P.S.
>
> To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:
>
> @@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
>   	arm_smmu_sync_ste_for_sid(smmu, sid);
>
>   	/* It's likely that we'll want to use the new STE soon */
>  -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>  +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>   }
>
>   static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)
>
>
>  @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
>   	msg->address_hi		= upper_32_bits(addr);
>   	msg->data		= its_get_event_id(d);
>
>  -	iommu_dma_map_msi_msg(d->irq, msg);
>  +	//iommu_dma_map_msi_msg(d->irq, msg);
>   }
>
>   static struct irq_chip its_irq_chip = {
>
> @@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
>   	return ret;
>   }
>
>  +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
>  +{
>  +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
>  +	int ret = acs_flags & ~flags ? 0 : 1;
>  +
>  +	return ret;
>  +}
>  +
>   /*
>    * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
>    * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
>  @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
>   	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
>   	/* Cavium ThunderX */
>   	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
>  +	/* Hisilicon Hip05/Hip06 root ports */
>  +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
>   	{ 0 }
>   };
>
> If you have time, could you please take a look at this issue?
>
> Thanks,
> Zhou
>
>>
>> Previous post of this series [6].
>>
>>  [V9]
>>      * Rebased on top of 4.11-rc1.
>>
>>      * Merged Robin's fixes for legacy binding issue,
>>        pci devices with no iommu-map property.
>>
>>  [V8]
>>      * Picked up all the acks and tested tags from Marek and
>>        Hanjun for DT and ACPI patches respectively, since
>>        no functional changes was done.
>>
>>      * Addressed Minor comments Sinan and Bjorn.
>>
>>      * Added Robin's fix for fixing the deferencing NULL for
>>        of_iommu_table after init in patch #2.
>>
>>      * Rebased it on top of linux-next
>>
>>  [V7]
>>      * Updated the subject and commit log for patch #6 as per
>>        comments from Lorenzo. No functional changes.
>>
>>  [V6]
>>      * Fixed a bug in dma_configure function pointed out by
>>        Robin.
>>      * Reordered the patches as per comments from Robin and
>>        Lorenzo.
>>      * Added Tags.
>>
>>  [V5]
>>      * Reworked the pci configuration code hanging outside and
>>        pushed it to dma_configure as in PATCH#5,6,7.
>>        Also added a couple of patches that Lorenzo provided for
>>        correcting the Probe deferring mechanism in case of
>>        ACPI devices from here [5].
>>
>>  [V4]
>>      * Took the reworked patches [2] from Robin's branch and
>>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
>>
>>      * Added the patches for moving the dma ops configuration of
>>        acpi based devices to probe time as well.
>>  [V3]
>>      * Removed the patch to split dma_masks/dma_ops configuration
>>        separately based on review comments that both masks and ops are
>>        required only during the device probe time.
>>
>>      * Reworked the series based on Generic DT bindings series.
>>
>>      * Added call to iommu's remove_device in the cleanup path for arm and
>>        arm64.
>>
>>      * Removed the notifier trick in arm64 to handle early device
>>        registration.
>>
>>      * Added reset of dma_ops in cleanup path for arm based on comments.
>>
>>      * Fixed the pci_iommu_configure path and tested with PCI device as
>>        well.
>>
>>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>>        last post.
>>
>>      * Fixed few other cosmetic comments.
>>
>>  [V2]
>>      * Updated the Initial post to call dma_configure/deconfigure from
>>        generic code
>>
>>      * Added iommu add_device callback from of_iommu_configure path
>>
>>  [V1]
>>      * Initial post from Laurent Pinchart [1]
>>
>> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
>> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
>> [3] https://lkml.org/lkml/2016/11/21/141
>> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
>> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
>> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
>> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
>>
>> Laurent Pinchart (3):
>>   of: dma: Move range size workaround to of_dma_get_range()
>>   of: dma: Make of_dma_deconfigure() public
>>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
>>
>> Lorenzo Pieralisi (2):
>>   ACPI/IORT: Add function to check SMMUs drivers presence
>>   ACPI/IORT: Remove linker section for IORT entries probing
>>
>> Robin Murphy (3):
>>   iommu/of: Refactor of_iommu_configure() for error handling
>>   iommu/of: Prepare for deferred IOMMU configuration
>>   iommu/arm-smmu: Clean up early-probing workarounds
>>
>> Sricharan R (3):
>>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>>     bus devices
>>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>>     error
>>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>>     of dma_ops
>>
>>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>>  drivers/acpi/glue.c               |   5 --
>>  drivers/acpi/scan.c               |   7 +-
>>  drivers/base/dd.c                 |   9 +++
>>  drivers/base/dma-mapping.c        |  41 +++++++++++
>>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>>  drivers/of/address.c              |  20 +++++-
>>  drivers/of/device.c               |  34 ++++-----
>>  drivers/of/platform.c             |  10 +--
>>  drivers/pci/probe.c               |  28 --------
>>  include/acpi/acpi_bus.h           |   2 +-
>>  include/asm-generic/vmlinux.lds.h |   1 -
>>  include/linux/acpi.h              |   7 +-
>>  include/linux/acpi_iort.h         |   3 -
>>  include/linux/dma-mapping.h       |   3 +
>>  include/linux/of_device.h         |  10 ++-
>>  19 files changed, 308 insertions(+), 336 deletions(-)
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Zhou,

On 3/24/2017 9:23 AM, Zhou Wang wrote:
> On 2017/3/10 3:00, Sricharan R wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> Rebased the series against mainline 4.11-rc1. Applies and builds cleanly
>> against mainline and linux-next. There is a conflict with patch#9
>> against iommu-next, but that should go away eventually as iommu-next
>> is rebased against 4.11-rc1.
>>
>> * Tested with platform and pci devices for probe deferral
>>   and reprobe on arm64 based platform.
>
> Hi Sricharan,
>
> I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon
> D05 board(with Intel 82599 networking card). It failed.
>
> After I used:
>
> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>
> to bind vfio-pci driver to Intel 82599 networking card VF.
>
> I got log in host:
> [...]
> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
> [  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
> [  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
> [  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
> [  414.448713] iommu: Adding device 0002:81:10.0 to group 4
> [  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
> [  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
> [  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
> [  414.451417] ixgbevf 0002:81:10.0: MAC: 1
> [  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
> [  414.464271] VFIO - User Level meta-driver version: 0.3
> [  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
> [  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here

Looks like this triggers the start of the bug.
So the below check in iommu_dma_init_domain fails,

         if (domain->geometry.force_aperture) {
                 if (base > domain->geometry.aperture_end ||
                     base + size <= domain->geometry.aperture_start) {

and the rest goes out of sync after that. Can you print out the
base, aperture_start and end values to see why the check fails ?

The change that this series does is trying to add the
dma/iommu ops to the device after the iommu is actually probed.
So in your working case, does the device initially gets hooked
to iommu_ops and the above same check passes in working case ?

Regards,
  Sricharan

> [  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
> [  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX
>
> Then I tried to boot up VM using:
>
> qemu-system-aarch64 \
> -machine virt,gic-version=3 \
> -enable-kvm \
> -cpu host \
> -m 1024 \
> -kernel ./Image_vm \
> -initrd ./minifs.cpio.gz \
> -nographic \
> -net none -device vfio-pci,host=0002:81:10.0,id=net0
>
> I got this error:
>
> root at ubuntu:~/scripts# ./qemu_run.sh
> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
> [  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
> [  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
> [  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
> [  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
> [  402.646338] sp : ffff8017e5a8fb50
> [  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
> [  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
> [  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
> [  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
> [  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
> [  402.676134] x19: 0000000000000001 x18: 0000000000783488
> [  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
> [  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
> [  402.692031] x13: 0000000000783060 x12: 0000000000000012
> [  402.697330] x11: 0000000000000001 x10: 0000000000000900
> [  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
> [  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
> [  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
> [  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
> [  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
> [  402.729123]
> [  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
> [  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
> [  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
> [  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
> [  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
> [  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
> [  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
> [  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
> [  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
> [  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
> [  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
> [  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
> [  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
> [  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
> [  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
> [  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
> [  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
> [  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
> [  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
> [  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
> [  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
> [  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
> [  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
> [  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
> [  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
> [  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
> [  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
> [  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
> [  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
> [  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
> [  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
> [  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
> [  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
> [  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
> [  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
> [  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
> [  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
> [  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
> [  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
> [  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [  403.040775] Call trace:
> [  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
> [  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
> [  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
> [  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
> [  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
> [  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
> [  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
> [  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
> [  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
> [  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
> [  403.119970] faa0: ffff000008207410 0000ffff967caa50
> [  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
> [  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
> [  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
> [  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
> [  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
> [  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
> [  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
> [  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
> [  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
> [  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
> [  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
> [  403.192351] ---[ end trace 0bda8b3549cfd903 ]---
>
> Dropping this series, 82599 VF can work well in QEMU.
>
> P.S.
>
> To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:
>
> @@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
>   	arm_smmu_sync_ste_for_sid(smmu, sid);
>
>   	/* It's likely that we'll want to use the new STE soon */
>  -	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  -		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>  +//	if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
>  +//		arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
>   }
>
>   static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)
>
>
>  @@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
>   	msg->address_hi		= upper_32_bits(addr);
>   	msg->data		= its_get_event_id(d);
>
>  -	iommu_dma_map_msi_msg(d->irq, msg);
>  +	//iommu_dma_map_msi_msg(d->irq, msg);
>   }
>
>   static struct irq_chip its_irq_chip = {
>
> @@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
>   	return ret;
>   }
>
>  +static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags)
>  +{
>  +	u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
>  +	int ret = acs_flags & ~flags ? 0 : 1;
>  +
>  +	return ret;
>  +}
>  +
>   /*
>    * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
>    * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
>  @@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
>   	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
>   	/* Cavium ThunderX */
>   	{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
>  +	/* Hisilicon Hip05/Hip06 root ports */
>  +	{ PCI_VENDOR_ID_HUAWEI,	0x1610, pci_quirk_hisi_rp_acs },
>   	{ 0 }
>   };
>
> If you have time, could you please take a look at this issue?
>
> Thanks,
> Zhou
>
>>
>> Previous post of this series [6].
>>
>>  [V9]
>>      * Rebased on top of 4.11-rc1.
>>
>>      * Merged Robin's fixes for legacy binding issue,
>>        pci devices with no iommu-map property.
>>
>>  [V8]
>>      * Picked up all the acks and tested tags from Marek and
>>        Hanjun for DT and ACPI patches respectively, since
>>        no functional changes was done.
>>
>>      * Addressed Minor comments Sinan and Bjorn.
>>
>>      * Added Robin's fix for fixing the deferencing NULL for
>>        of_iommu_table after init in patch #2.
>>
>>      * Rebased it on top of linux-next
>>
>>  [V7]
>>      * Updated the subject and commit log for patch #6 as per
>>        comments from Lorenzo. No functional changes.
>>
>>  [V6]
>>      * Fixed a bug in dma_configure function pointed out by
>>        Robin.
>>      * Reordered the patches as per comments from Robin and
>>        Lorenzo.
>>      * Added Tags.
>>
>>  [V5]
>>      * Reworked the pci configuration code hanging outside and
>>        pushed it to dma_configure as in PATCH#5,6,7.
>>        Also added a couple of patches that Lorenzo provided for
>>        correcting the Probe deferring mechanism in case of
>>        ACPI devices from here [5].
>>
>>  [V4]
>>      * Took the reworked patches [2] from Robin's branch and
>>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
>>
>>      * Added the patches for moving the dma ops configuration of
>>        acpi based devices to probe time as well.
>>  [V3]
>>      * Removed the patch to split dma_masks/dma_ops configuration
>>        separately based on review comments that both masks and ops are
>>        required only during the device probe time.
>>
>>      * Reworked the series based on Generic DT bindings series.
>>
>>      * Added call to iommu's remove_device in the cleanup path for arm and
>>        arm64.
>>
>>      * Removed the notifier trick in arm64 to handle early device
>>        registration.
>>
>>      * Added reset of dma_ops in cleanup path for arm based on comments.
>>
>>      * Fixed the pci_iommu_configure path and tested with PCI device as
>>        well.
>>
>>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>>        last post.
>>
>>      * Fixed few other cosmetic comments.
>>
>>  [V2]
>>      * Updated the Initial post to call dma_configure/deconfigure from
>>        generic code
>>
>>      * Added iommu add_device callback from of_iommu_configure path
>>
>>  [V1]
>>      * Initial post from Laurent Pinchart [1]
>>
>> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
>> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
>> [3] https://lkml.org/lkml/2016/11/21/141
>> [4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
>> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
>> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
>> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
>>
>> Laurent Pinchart (3):
>>   of: dma: Move range size workaround to of_dma_get_range()
>>   of: dma: Make of_dma_deconfigure() public
>>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
>>
>> Lorenzo Pieralisi (2):
>>   ACPI/IORT: Add function to check SMMUs drivers presence
>>   ACPI/IORT: Remove linker section for IORT entries probing
>>
>> Robin Murphy (3):
>>   iommu/of: Refactor of_iommu_configure() for error handling
>>   iommu/of: Prepare for deferred IOMMU configuration
>>   iommu/arm-smmu: Clean up early-probing workarounds
>>
>> Sricharan R (3):
>>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>>     bus devices
>>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>>     error
>>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>>     of dma_ops
>>
>>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>>  drivers/acpi/arm64/iort.c         |  40 ++++++++++-
>>  drivers/acpi/glue.c               |   5 --
>>  drivers/acpi/scan.c               |   7 +-
>>  drivers/base/dd.c                 |   9 +++
>>  drivers/base/dma-mapping.c        |  41 +++++++++++
>>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>>  drivers/of/address.c              |  20 +++++-
>>  drivers/of/device.c               |  34 ++++-----
>>  drivers/of/platform.c             |  10 +--
>>  drivers/pci/probe.c               |  28 --------
>>  include/acpi/acpi_bus.h           |   2 +-
>>  include/asm-generic/vmlinux.lds.h |   1 -
>>  include/linux/acpi.h              |   7 +-
>>  include/linux/acpi_iort.h         |   3 -
>>  include/linux/dma-mapping.h       |   3 +
>>  include/linux/of_device.h         |  10 ++-
>>  19 files changed, 308 insertions(+), 336 deletions(-)
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

Hi Sricharan,

> -----Original Message-----
> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
> Sent: Friday, March 24, 2017 7:10 AM
> To: Wangzhou (B); robin.murphy-5wv7dgnIgG8@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org;
> joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> foundation.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-
> msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
> okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> Cc: Shameerali Kolothum Thodi
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi Zhou,
> 
> On 3/24/2017 9:23 AM, Zhou Wang wrote:
> > On 2017/3/10 3:00, Sricharan R wrote:
> >> This series calls the dma ops configuration for the devices at a
> >> generic place so that it works for all busses.
> >> The dma_configure_ops for a device is now called during the
> >> device_attach callback just before the probe of the bus/driver is
> >> called. Similarly dma_deconfigure is called during
> >> device/driver_detach path.
> >>
> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >>        |                         |
> >> pci_bus_add_device     (device_add/driver_register)
> >>        |                         |
> >> device_attach           device_initial_probe
> >>        |                         |
> >> __device_attach_driver    __device_attach_driver
> >>        |
> >> driver_probe_device
> >>        |
> >> really_probe
> >>        |
> >> dma_configure
> >>
> >> Similarly on the device/driver_unregister path
> >> __device_release_driver is called which inturn calls dma_deconfigure.
> >>
> >> Rebased the series against mainline 4.11-rc1. Applies and builds
> >> cleanly against mainline and linux-next. There is a conflict with
> >> patch#9 against iommu-next, but that should go away eventually as
> >> iommu-next is rebased against 4.11-rc1.
> >>
> >> * Tested with platform and pci devices for probe deferral
> >>   and reprobe on arm64 based platform.
> >
> > Hi Sricharan,
> >
> > I applied this series on v4.11-rc1 to test PCIe pass through in
> > HiSilicon
> > D05 board(with Intel 82599 networking card). It failed.
> >
> > After I used:
> >
> > echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> > echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> > echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
> >
> > to bind vfio-pci driver to Intel 82599 networking card VF.
> >
> > I got log in host:
> > [...]
> > [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
> > Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
> (c) 2009 - 2015 Intel Corporation.
> > [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
> > 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> > [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
> > ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
> > count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
> > 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
> > 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
> > 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
> up?
> > [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
> > 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
> > ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
> > Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
> > meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
> > PHC device on eth12
> > [  414.700493] specified DMA range outside IOMMU capability
> <-- error here
> > [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
> platform DMA ops        <-- error here
> 
> Looks like this triggers the start of the bug.
> So the below check in iommu_dma_init_domain fails,
> 
>          if (domain->geometry.force_aperture) {
>                  if (base > domain->geometry.aperture_end ||
>                      base + size <= domain->geometry.aperture_start) {
> 
> and the rest goes out of sync after that. Can you print out the base,
> aperture_start and end values to see why the check fails ?

dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);

[  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
.....
[  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff

Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

@@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
  	if (ret < 0) {
  		dma_addr = offset = 0;
 -		size = dev->coherent_dma_mask + 1;
 +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

@@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
  	 * Assume dma valid range starts at 0 and covers the whole
  	 * coherent_dma_mask.
  	 */
 -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 +	arch_setup_dma_ops(dev, 0, size, iommu,
  			   attr == DEV_DMA_COHERENT);

With the above fixes, DT boot works fine. But we still get the below crash on ACPI

> > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> vfio_virqfd vfio ixgbevf ixgb

> The change that this series does is trying to add the dma/iommu ops to the
> device after the iommu is actually probed.
> So in your working case, does the device initially gets hooked to iommu_ops
> and the above same check passes in working case ?

I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
in the working case.
 
Thanks,
Shameer

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

Hi Sricharan,

> -----Original Message-----
> From: Sricharan R [mailto:sricharan@codeaurora.org]
> Sent: Friday, March 24, 2017 7:10 AM
> To: Wangzhou (B); robin.murphy@arm.com; will.deacon@arm.com;
> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
> msm@vger.kernel.org; m.szyprowski@samsung.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
> okaya@codeaurora.org
> Cc: Shameerali Kolothum Thodi
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi Zhou,
> 
> On 3/24/2017 9:23 AM, Zhou Wang wrote:
> > On 2017/3/10 3:00, Sricharan R wrote:
> >> This series calls the dma ops configuration for the devices at a
> >> generic place so that it works for all busses.
> >> The dma_configure_ops for a device is now called during the
> >> device_attach callback just before the probe of the bus/driver is
> >> called. Similarly dma_deconfigure is called during
> >> device/driver_detach path.
> >>
> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >>        |                         |
> >> pci_bus_add_device     (device_add/driver_register)
> >>        |                         |
> >> device_attach           device_initial_probe
> >>        |                         |
> >> __device_attach_driver    __device_attach_driver
> >>        |
> >> driver_probe_device
> >>        |
> >> really_probe
> >>        |
> >> dma_configure
> >>
> >> Similarly on the device/driver_unregister path
> >> __device_release_driver is called which inturn calls dma_deconfigure.
> >>
> >> Rebased the series against mainline 4.11-rc1. Applies and builds
> >> cleanly against mainline and linux-next. There is a conflict with
> >> patch#9 against iommu-next, but that should go away eventually as
> >> iommu-next is rebased against 4.11-rc1.
> >>
> >> * Tested with platform and pci devices for probe deferral
> >>   and reprobe on arm64 based platform.
> >
> > Hi Sricharan,
> >
> > I applied this series on v4.11-rc1 to test PCIe pass through in
> > HiSilicon
> > D05 board(with Intel 82599 networking card). It failed.
> >
> > After I used:
> >
> > echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> > echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> > echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
> >
> > to bind vfio-pci driver to Intel 82599 networking card VF.
> >
> > I got log in host:
> > [...]
> > [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
> > Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
> (c) 2009 - 2015 Intel Corporation.
> > [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
> > 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> > [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
> > ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
> > count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
> > 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
> > 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
> > 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
> up?
> > [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
> > 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
> > ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
> > Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
> > meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
> > PHC device on eth12
> > [  414.700493] specified DMA range outside IOMMU capability
> <-- error here
> > [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
> platform DMA ops        <-- error here
> 
> Looks like this triggers the start of the bug.
> So the below check in iommu_dma_init_domain fails,
> 
>          if (domain->geometry.force_aperture) {
>                  if (base > domain->geometry.aperture_end ||
>                      base + size <= domain->geometry.aperture_start) {
> 
> and the rest goes out of sync after that. Can you print out the base,
> aperture_start and end values to see why the check fails ?

dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);

[  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
.....
[  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff

Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

@@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
  	if (ret < 0) {
  		dma_addr = offset = 0;
 -		size = dev->coherent_dma_mask + 1;
 +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

@@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
  	 * Assume dma valid range starts at 0 and covers the whole
  	 * coherent_dma_mask.
  	 */
 -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 +	arch_setup_dma_ops(dev, 0, size, iommu,
  			   attr == DEV_DMA_COHERENT);

With the above fixes, DT boot works fine. But we still get the below crash on ACPI

> > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> vfio_virqfd vfio ixgbevf ixgb

> The change that this series does is trying to add the dma/iommu ops to the
> device after the iommu is actually probed.
> So in your working case, does the device initially gets hooked to iommu_ops
> and the above same check passes in working case ?

I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
in the working case.
 
Thanks,
Shameer

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

Hi Sricharan,

> -----Original Message-----
> From: Sricharan R [mailto:sricharan at codeaurora.org]
> Sent: Friday, March 24, 2017 7:10 AM
> To: Wangzhou (B); robin.murphy at arm.com; will.deacon at arm.com;
> joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-
> foundation.org; linux-arm-kernel at lists.infradead.org; linux-arm-
> msm at vger.kernel.org; m.szyprowski at samsung.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
> okaya at codeaurora.org
> Cc: Shameerali Kolothum Thodi
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi Zhou,
> 
> On 3/24/2017 9:23 AM, Zhou Wang wrote:
> > On 2017/3/10 3:00, Sricharan R wrote:
> >> This series calls the dma ops configuration for the devices at a
> >> generic place so that it works for all busses.
> >> The dma_configure_ops for a device is now called during the
> >> device_attach callback just before the probe of the bus/driver is
> >> called. Similarly dma_deconfigure is called during
> >> device/driver_detach path.
> >>
> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >>        |                         |
> >> pci_bus_add_device     (device_add/driver_register)
> >>        |                         |
> >> device_attach           device_initial_probe
> >>        |                         |
> >> __device_attach_driver    __device_attach_driver
> >>        |
> >> driver_probe_device
> >>        |
> >> really_probe
> >>        |
> >> dma_configure
> >>
> >> Similarly on the device/driver_unregister path
> >> __device_release_driver is called which inturn calls dma_deconfigure.
> >>
> >> Rebased the series against mainline 4.11-rc1. Applies and builds
> >> cleanly against mainline and linux-next. There is a conflict with
> >> patch#9 against iommu-next, but that should go away eventually as
> >> iommu-next is rebased against 4.11-rc1.
> >>
> >> * Tested with platform and pci devices for probe deferral
> >>   and reprobe on arm64 based platform.
> >
> > Hi Sricharan,
> >
> > I applied this series on v4.11-rc1 to test PCIe pass through in
> > HiSilicon
> > D05 board(with Intel 82599 networking card). It failed.
> >
> > After I used:
> >
> > echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> > echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> > echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
> >
> > to bind vfio-pci driver to Intel 82599 networking card VF.
> >
> > I got log in host:
> > [...]
> > [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
> > Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
> (c) 2009 - 2015 Intel Corporation.
> > [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
> > 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> > [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
> > ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
> > count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
> > 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
> > 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
> > 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
> up?
> > [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
> > 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
> > ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
> > Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
> > meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
> > PHC device on eth12
> > [  414.700493] specified DMA range outside IOMMU capability
> <-- error here
> > [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
> platform DMA ops        <-- error here
> 
> Looks like this triggers the start of the bug.
> So the below check in iommu_dma_init_domain fails,
> 
>          if (domain->geometry.force_aperture) {
>                  if (base > domain->geometry.aperture_end ||
>                      base + size <= domain->geometry.aperture_start) {
> 
> and the rest goes out of sync after that. Can you print out the base,
> aperture_start and end values to see why the check fails ?

dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);

[  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
.....
[  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff

Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

@@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
  	if (ret < 0) {
  		dma_addr = offset = 0;
 -		size = dev->coherent_dma_mask + 1;
 +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

@@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
  	 * Assume dma valid range starts at 0 and covers the whole
  	 * coherent_dma_mask.
  	 */
 -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 +	arch_setup_dma_ops(dev, 0, size, iommu,
  			   attr == DEV_DMA_COHERENT);

With the above fixes, DT boot works fine. But we still get the below crash on ACPI

> > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> vfio_virqfd vfio ixgbevf ixgb

> The change that this series does is trying to add the dma/iommu ops to the
> device after the iommu is actually probed.
> So in your working case, does the device initially gets hooked to iommu_ops
> and the above same check passes in working case ?

I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
in the working case.
 
Thanks,
Shameer

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Shameer,


>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
>
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
>
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
>
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.
>
> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = ';
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>

But without this series, size is still set as
dev->coherent_dma_mask + 1 , somehow not sure how it works
fine in that case ?

I remember i had this change in the V7 patchset, but later
dropped it for the reason that this change is not relevant
for this series, but should be there/sent to address the 64bit
overflow separately.

> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
>
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
>
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!

Looks like this happens when the ste_live becomes true during
the initial attach, but later without an detach,
attach once again happens from vfio. Just thinking why the
detach_dev is not happening only in ACPI case. Actually not
having any of arm-smmuv3 or ACPI based setup in my place.
Can i get some help by having some logs from the arm-smmv3
driver ?

>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb
>
>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
>
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.

ok, as i said above not sure why the overflow does not affect without
this series.

Regards,
  Sricharan


>
> Thanks,
> Shameer
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Shameer,


>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
>
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
>
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
>
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.
>
> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = ';
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>

But without this series, size is still set as
dev->coherent_dma_mask + 1 , somehow not sure how it works
fine in that case ?

I remember i had this change in the V7 patchset, but later
dropped it for the reason that this change is not relevant
for this series, but should be there/sent to address the 64bit
overflow separately.

> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
>
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
>
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!

Looks like this happens when the ste_live becomes true during
the initial attach, but later without an detach,
attach once again happens from vfio. Just thinking why the
detach_dev is not happening only in ACPI case. Actually not
having any of arm-smmuv3 or ACPI based setup in my place.
Can i get some help by having some logs from the arm-smmv3
driver ?

>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb
>
>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
>
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.

ok, as i said above not sure why the overflow does not affect without
this series.

Regards,
  Sricharan


>
> Thanks,
> Shameer
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi Shameer,


>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
>
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
>
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
>
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.
>
> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = ';
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>

But without this series, size is still set as
dev->coherent_dma_mask + 1 , somehow not sure how it works
fine in that case ?

I remember i had this change in the V7 patchset, but later
dropped it for the reason that this change is not relevant
for this series, but should be there/sent to address the 64bit
overflow separately.

> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
>
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
>
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!

Looks like this happens when the ste_live becomes true during
the initial attach, but later without an detach,
attach once again happens from vfio. Just thinking why the
detach_dev is not happening only in ACPI case. Actually not
having any of arm-smmuv3 or ACPI based setup in my place.
Can i get some help by having some logs from the arm-smmv3
driver ?

>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb
>
>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
>
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.

ok, as i said above not sure why the overflow does not affect without
this series.

Regards,
  Sricharan


>
> Thanks,
> Shameer
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:

[...]

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

size is not defined in acpi_dma_configure(), so there is
something missing in the diff above, just to make sure we are
testing the same thing.

Mind posting kernel logs with and without patch above (fixed) for
DT boot and ACPI boot please.

Thanks,
Lorenzo

>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
> > > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> > vfio_virqfd vfio ixgbevf ixgb
> 
> > The change that this series does is trying to add the dma/iommu ops to the
> > device after the iommu is actually probed.
> > So in your working case, does the device initially gets hooked to iommu_ops
> > and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:

[...]

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

size is not defined in acpi_dma_configure(), so there is
something missing in the diff above, just to make sure we are
testing the same thing.

Mind posting kernel logs with and without patch above (fixed) for
DT boot and ACPI boot please.

Thanks,
Lorenzo

>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
> > > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> > vfio_virqfd vfio ixgbevf ixgb
> 
> > The change that this series does is trying to add the dma/iommu ops to the
> > device after the iommu is actually probed.
> > So in your working case, does the device initially gets hooked to iommu_ops
> > and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:

[...]

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

size is not defined in acpi_dma_configure(), so there is
something missing in the diff above, just to make sure we are
testing the same thing.

Mind posting kernel logs with and without patch above (fixed) for
DT boot and ACPI boot please.

Thanks,
Lorenzo

>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
> > > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> > vfio_virqfd vfio ixgbevf ixgb
> 
> > The change that this series does is trying to add the dma/iommu ops to the
> > device after the iommu is actually probed.
> > So in your working case, does the device initially gets hooked to iommu_ops
> > and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org]
> Sent: Friday, March 24, 2017 2:44 PM
> To: Shameerali Kolothum Thodi
> Cc: Sricharan R; Wangzhou (B); robin.murphy-5wv7dgnIgG8@public.gmane.org;
> will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org;
> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
> m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org; bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-
> pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org;
> hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:
> 
> [...]
> 
> > @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct
> device_node *np)
> >   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
> >   	if (ret < 0) {
> >   		dma_addr = offset = 0;
> >  -		size = dev->coherent_dma_mask + 1;
> >  +		size = max(dev->coherent_dma_mask, dev-
> >coherent_dma_mask + 1);
> >
> > @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum
> dev_dma_attr attr)
> >   	 * Assume dma valid range starts at 0 and covers the whole
> >   	 * coherent_dma_mask.
> >   	 */
> >  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1,
> iommu,
> >  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask +
> 1);
> 
> size is not defined in acpi_dma_configure(), so there is something missing in
> the diff above, just to make sure we are testing the same thing.
> 
> Mind posting kernel logs with and without patch above (fixed) for DT boot
> and ACPI boot please.

Sure. We will rerun the tests to create the logs again and share.

Also need to double check, v9 patches are applied correctly on top of our branch.
If I remember correctly v7 was fine with DT(didn't test with ACPI). But then as 
Shicharan said it had that size fix.

Thanks,
Shameer
[...]

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> Sent: Friday, March 24, 2017 2:44 PM
> To: Shameerali Kolothum Thodi
> Cc: Sricharan R; Wangzhou (B); robin.murphy@arm.com;
> will.deacon@arm.com; joro@8bytes.org; iommu@lists.linux-foundation.org;
> linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org;
> m.szyprowski@samsung.com; bhelgaas@google.com; linux-
> pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com;
> hanjun.guo@linaro.org; okaya@codeaurora.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:
> 
> [...]
> 
> > @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct
> device_node *np)
> >   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
> >   	if (ret < 0) {
> >   		dma_addr = offset = 0;
> >  -		size = dev->coherent_dma_mask + 1;
> >  +		size = max(dev->coherent_dma_mask, dev-
> >coherent_dma_mask + 1);
> >
> > @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum
> dev_dma_attr attr)
> >   	 * Assume dma valid range starts at 0 and covers the whole
> >   	 * coherent_dma_mask.
> >   	 */
> >  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1,
> iommu,
> >  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask +
> 1);
> 
> size is not defined in acpi_dma_configure(), so there is something missing in
> the diff above, just to make sure we are testing the same thing.
> 
> Mind posting kernel logs with and without patch above (fixed) for DT boot
> and ACPI boot please.

Sure. We will rerun the tests to create the logs again and share.

Also need to double check, v9 patches are applied correctly on top of our branch.
If I remember correctly v7 was fine with DT(didn't test with ACPI). But then as 
Shicharan said it had that size fix.

Thanks,
Shameer
[...]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi at arm.com]
> Sent: Friday, March 24, 2017 2:44 PM
> To: Shameerali Kolothum Thodi
> Cc: Sricharan R; Wangzhou (B); robin.murphy at arm.com;
> will.deacon at arm.com; joro at 8bytes.org; iommu at lists.linux-foundation.org;
> linux-arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org;
> m.szyprowski at samsung.com; bhelgaas at google.com; linux-
> pci at vger.kernel.org; linux-acpi at vger.kernel.org; tn at semihalf.com;
> hanjun.guo at linaro.org; okaya at codeaurora.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:
> 
> [...]
> 
> > @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct
> device_node *np)
> >   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
> >   	if (ret < 0) {
> >   		dma_addr = offset = 0;
> >  -		size = dev->coherent_dma_mask + 1;
> >  +		size = max(dev->coherent_dma_mask, dev-
> >coherent_dma_mask + 1);
> >
> > @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum
> dev_dma_attr attr)
> >   	 * Assume dma valid range starts at 0 and covers the whole
> >   	 * coherent_dma_mask.
> >   	 */
> >  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1,
> iommu,
> >  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask +
> 1);
> 
> size is not defined in acpi_dma_configure(), so there is something missing in
> the diff above, just to make sure we are testing the same thing.
> 
> Mind posting kernel logs with and without patch above (fixed) for DT boot
> and ACPI boot please.

Sure. We will rerun the tests to create the logs again and share.

Also need to double check, v9 patches are applied correctly on top of our branch.
If I remember correctly v7 was fine with DT(didn't test with ACPI). But then as 
Shicharan said it had that size fix.

Thanks,
Shameer
[...]

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> Hi Sricharan,
> 
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>> Sent: Friday, March 24, 2017 7:10 AM
>> To: Wangzhou (B); robin.murphy@arm.com; will.deacon@arm.com;
>> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
>> msm@vger.kernel.org; m.szyprowski@samsung.com;
>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>> okaya@codeaurora.org
>> Cc: Shameerali Kolothum Thodi
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi Zhou,
>>
>> On 3/24/2017 9:23 AM, Zhou Wang wrote:
>>> On 2017/3/10 3:00, Sricharan R wrote:
>>>> This series calls the dma ops configuration for the devices at a
>>>> generic place so that it works for all busses.
>>>> The dma_configure_ops for a device is now called during the
>>>> device_attach callback just before the probe of the bus/driver is
>>>> called. Similarly dma_deconfigure is called during
>>>> device/driver_detach path.
>>>>
>>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>>        |                         |
>>>> pci_bus_add_device     (device_add/driver_register)
>>>>        |                         |
>>>> device_attach           device_initial_probe
>>>>        |                         |
>>>> __device_attach_driver    __device_attach_driver
>>>>        |
>>>> driver_probe_device
>>>>        |
>>>> really_probe
>>>>        |
>>>> dma_configure
>>>>
>>>> Similarly on the device/driver_unregister path
>>>> __device_release_driver is called which inturn calls dma_deconfigure.
>>>>
>>>> Rebased the series against mainline 4.11-rc1. Applies and builds
>>>> cleanly against mainline and linux-next. There is a conflict with
>>>> patch#9 against iommu-next, but that should go away eventually as
>>>> iommu-next is rebased against 4.11-rc1.
>>>>
>>>> * Tested with platform and pci devices for probe deferral
>>>>   and reprobe on arm64 based platform.
>>>
>>> Hi Sricharan,
>>>
>>> I applied this series on v4.11-rc1 to test PCIe pass through in
>>> HiSilicon
>>> D05 board(with Intel 82599 networking card). It failed.
>>>
>>> After I used:
>>>
>>> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>>>
>>> to bind vfio-pci driver to Intel 82599 networking card VF.
>>>
>>> I got log in host:
>>> [...]
>>> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
>>> Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
>> (c) 2009 - 2015 Intel Corporation.
>>> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
>>> 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
>>> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
>>> ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
>>> count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
>>> 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
>>> 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
>>> 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
>> up?
>>> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
>>> 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
>>> ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
>>> Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
>>> meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
>>> PHC device on eth12
>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
> 
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
> 
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> 
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

AFAICS, this is either down to patch 3 (which should apply on its own
easily enough for testing), or patch 6, implying that somehow the
vfio-pci device gets its DMA mask widened to 64 bits somewhere between
very soon after after creation (where we originally called
of_dma_configure()) and immediately before probe (where we now call it).

Either way I guess this is yet more motivation to write that "change the
arch_setup_dma_ops() interface to take a mask instead of a size" patch...

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb

I can't see how ACPI vs. DT would make any difference to the domain
attach/detach mechanics getting into an invalid state (if the DT case
works then unbinding the ixgbevf driver clearly does release the Stream
ID correctly in general), so I'm more inclined to believe that something
goes wrong with the fwspec in the ACPI path such that we end up touching
the wrong STE altogether.

Robin.

>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer
> 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> Hi Sricharan,
> 
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>> Sent: Friday, March 24, 2017 7:10 AM
>> To: Wangzhou (B); robin.murphy@arm.com; will.deacon@arm.com;
>> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
>> msm@vger.kernel.org; m.szyprowski@samsung.com;
>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>> okaya@codeaurora.org
>> Cc: Shameerali Kolothum Thodi
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi Zhou,
>>
>> On 3/24/2017 9:23 AM, Zhou Wang wrote:
>>> On 2017/3/10 3:00, Sricharan R wrote:
>>>> This series calls the dma ops configuration for the devices at a
>>>> generic place so that it works for all busses.
>>>> The dma_configure_ops for a device is now called during the
>>>> device_attach callback just before the probe of the bus/driver is
>>>> called. Similarly dma_deconfigure is called during
>>>> device/driver_detach path.
>>>>
>>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>>        |                         |
>>>> pci_bus_add_device     (device_add/driver_register)
>>>>        |                         |
>>>> device_attach           device_initial_probe
>>>>        |                         |
>>>> __device_attach_driver    __device_attach_driver
>>>>        |
>>>> driver_probe_device
>>>>        |
>>>> really_probe
>>>>        |
>>>> dma_configure
>>>>
>>>> Similarly on the device/driver_unregister path
>>>> __device_release_driver is called which inturn calls dma_deconfigure.
>>>>
>>>> Rebased the series against mainline 4.11-rc1. Applies and builds
>>>> cleanly against mainline and linux-next. There is a conflict with
>>>> patch#9 against iommu-next, but that should go away eventually as
>>>> iommu-next is rebased against 4.11-rc1.
>>>>
>>>> * Tested with platform and pci devices for probe deferral
>>>>   and reprobe on arm64 based platform.
>>>
>>> Hi Sricharan,
>>>
>>> I applied this series on v4.11-rc1 to test PCIe pass through in
>>> HiSilicon
>>> D05 board(with Intel 82599 networking card). It failed.
>>>
>>> After I used:
>>>
>>> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>>>
>>> to bind vfio-pci driver to Intel 82599 networking card VF.
>>>
>>> I got log in host:
>>> [...]
>>> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
>>> Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
>> (c) 2009 - 2015 Intel Corporation.
>>> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
>>> 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
>>> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
>>> ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
>>> count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
>>> 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
>>> 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
>>> 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
>> up?
>>> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
>>> 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
>>> ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
>>> Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
>>> meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
>>> PHC device on eth12
>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
> 
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
> 
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> 
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

AFAICS, this is either down to patch 3 (which should apply on its own
easily enough for testing), or patch 6, implying that somehow the
vfio-pci device gets its DMA mask widened to 64 bits somewhere between
very soon after after creation (where we originally called
of_dma_configure()) and immediately before probe (where we now call it).

Either way I guess this is yet more motivation to write that "change the
arch_setup_dma_ops() interface to take a mask instead of a size" patch...

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb

I can't see how ACPI vs. DT would make any difference to the domain
attach/detach mechanics getting into an invalid state (if the DT case
works then unbinding the ixgbevf driver clearly does release the Stream
ID correctly in general), so I'm more inclined to believe that something
goes wrong with the fwspec in the ACPI path such that we end up touching
the wrong STE altogether.

Robin.

>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer
> 

[PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> Hi Sricharan,
> 
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan at codeaurora.org]
>> Sent: Friday, March 24, 2017 7:10 AM
>> To: Wangzhou (B); robin.murphy at arm.com; will.deacon at arm.com;
>> joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-
>> foundation.org; linux-arm-kernel at lists.infradead.org; linux-arm-
>> msm at vger.kernel.org; m.szyprowski at samsung.com;
>> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
>> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
>> okaya at codeaurora.org
>> Cc: Shameerali Kolothum Thodi
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi Zhou,
>>
>> On 3/24/2017 9:23 AM, Zhou Wang wrote:
>>> On 2017/3/10 3:00, Sricharan R wrote:
>>>> This series calls the dma ops configuration for the devices at a
>>>> generic place so that it works for all busses.
>>>> The dma_configure_ops for a device is now called during the
>>>> device_attach callback just before the probe of the bus/driver is
>>>> called. Similarly dma_deconfigure is called during
>>>> device/driver_detach path.
>>>>
>>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>>        |                         |
>>>> pci_bus_add_device     (device_add/driver_register)
>>>>        |                         |
>>>> device_attach           device_initial_probe
>>>>        |                         |
>>>> __device_attach_driver    __device_attach_driver
>>>>        |
>>>> driver_probe_device
>>>>        |
>>>> really_probe
>>>>        |
>>>> dma_configure
>>>>
>>>> Similarly on the device/driver_unregister path
>>>> __device_release_driver is called which inturn calls dma_deconfigure.
>>>>
>>>> Rebased the series against mainline 4.11-rc1. Applies and builds
>>>> cleanly against mainline and linux-next. There is a conflict with
>>>> patch#9 against iommu-next, but that should go away eventually as
>>>> iommu-next is rebased against 4.11-rc1.
>>>>
>>>> * Tested with platform and pci devices for probe deferral
>>>>   and reprobe on arm64 based platform.
>>>
>>> Hi Sricharan,
>>>
>>> I applied this series on v4.11-rc1 to test PCIe pass through in
>>> HiSilicon
>>> D05 board(with Intel 82599 networking card). It failed.
>>>
>>> After I used:
>>>
>>> echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
>>> echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
>>>
>>> to bind vfio-pci driver to Intel 82599 networking card VF.
>>>
>>> I got log in host:
>>> [...]
>>> [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
>>> Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
>> (c) 2009 - 2015 Intel Corporation.
>>> [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
>>> 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
>>> [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
>>> ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
>>> count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
>>> 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
>>> 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
>>> 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
>> up?
>>> [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
>>> 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
>>> ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
>>> Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
>>> meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
>>> PHC device on eth12
>>> [  414.700493] specified DMA range outside IOMMU capability
>> <-- error here
>>> [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
>> platform DMA ops        <-- error here
>>
>> Looks like this triggers the start of the bug.
>> So the below check in iommu_dma_init_domain fails,
>>
>>          if (domain->geometry.force_aperture) {
>>                  if (base > domain->geometry.aperture_end ||
>>                      base + size <= domain->geometry.aperture_start) {
>>
>> and the rest goes out of sync after that. Can you print out the base,
>> aperture_start and end values to see why the check fails ?
> 
> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);
> 
> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> .....
> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> 
> Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
> somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

AFAICS, this is either down to patch 3 (which should apply on its own
easily enough for testing), or patch 6, implying that somehow the
vfio-pci device gets its DMA mask widened to 64 bits somewhere between
very soon after after creation (where we originally called
of_dma_configure()) and immediately before probe (where we now call it).

Either way I guess this is yet more motivation to write that "change the
arch_setup_dma_ops() interface to take a mask instead of a size" patch...

> @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
>   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>   	if (ret < 0) {
>   		dma_addr = offset = 0;
>  -		size = dev->coherent_dma_mask + 1;
>  +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
> 
> @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>   	 * Assume dma valid range starts at 0 and covers the whole
>   	 * coherent_dma_mask.
>   	 */
>  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
>  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  +	arch_setup_dma_ops(dev, 0, size, iommu,
>   			   attr == DEV_DMA_COHERENT);
> 
> With the above fixes, DT boot works fine. But we still get the below crash on ACPI
> 
>>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
>>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
>>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
>> vfio_virqfd vfio ixgbevf ixgb

I can't see how ACPI vs. DT would make any difference to the domain
attach/detach mechanics getting into an invalid state (if the DT case
works then unbinding the ixgbevf driver clearly does release the Stream
ID correctly in general), so I'm more inclined to believe that something
goes wrong with the fwspec in the ACPI path such that we end up touching
the wrong STE altogether.

Robin.

>> The change that this series does is trying to add the dma/iommu ops to the
>> device after the iommu is actually probed.
>> So in your working case, does the device initially gets hooked to iommu_ops
>> and the above same check passes in working case ?
> 
> I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
> in the working case.
>  
> Thanks,
> Shameer
> 

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

[-- Attachment #1: Type: text/plain, Size: 7404 bytes --]


> -----Original Message-----
> From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
> Sent: Friday, March 24, 2017 6:39 PM
> To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
> will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org;
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org;
> linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
> okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> > Hi Sricharan,
> >
> >> -----Original Message-----
> >> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
[...]
> >> Looks like this triggers the start of the bug.
> >> So the below check in iommu_dma_init_domain fails,
> >>
> >>          if (domain->geometry.force_aperture) {
> >>                  if (base > domain->geometry.aperture_end ||
> >>                      base + size <= domain->geometry.aperture_start) {
> >>
> >> and the rest goes out of sync after that. Can you print out the base,
> >> aperture_start and end values to see why the check fails ?
> >
> > dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
> domain->geometry.aperture_start, domain->geometry.aperture_end,
> *dev->dma_mask, dev->coherent_dma_mask);
> >
> > [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff
> > .....
> > [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
> 0xffffffffffffffff 0xffffffffffffffff
> >
> > Yes, size seems to be the problem here. When the VF  device gets attached
> to vfio-pci,
> > somehow the dev->coherent_dma_mask is set to 64 bits and size become
> zero.
> 
> AFAICS, this is either down to patch 3 (which should apply on its own
> easily enough for testing), or patch 6, implying that somehow the
> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
> very soon after after creation (where we originally called
> of_dma_configure()) and immediately before probe (where we now call it).
> 
> Either way I guess this is yet more motivation to write that "change the
> arch_setup_dma_ops() interface to take a mask instead of a size" patch...

Just applying the patch 3 and binding the device into vfio-pci is fine. Please find the 
log below (with  dev_info debug added to iommu_dma_init_domain ).
...
[  142.851906] iommu: Adding device 0000:81:10.0 to group 6
[  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff   ---->dev_info()
[  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
[  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
[  142.865692] ixgbevf 0000:81:10.0: MAC: 1
[  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  152.902894] nfs: server 172.18.45.166 not responding, still trying
[  188.980933] nfs: server 172.18.45.166 not responding, still trying
[  188.981298] nfs: server 172.18.45.166 OK
[  188.981593] nfs: server 172.18.45.166 OK
[  221.755626] VFIO - User Level meta-driver version: 0.3
...

Applied up to patch 6, and the issue appeared,

[  145.212351] iommu: Adding device 0000:81:10.0 to group 5
[  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
[  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
[  145.225944] ixgbevf 0000:81:10.0: MAC: 1
[  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  154.947742] nfs: server 172.18.45.166 not responding, still trying
[  191.025780] nfs: server 172.18.45.166 not responding, still trying
[  191.026122] nfs: server 172.18.45.166 OK
[  191.026317] nfs: server 172.18.45.166 OK
[  263.706402] VFIO - User Level meta-driver version: 0.3
[  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  269.757617] specified DMA range outside IOMMU capability
[  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

 From the logs its clear that  when ixgbevf driver originally probes and adds the device
 to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.

Also It looks like the iommu_dma_init_domain fn was not invoked in the first case when
the vfio-pci binds the device.

I am attaching the full logs for both cases.

> > @@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct
> device_node *np)
> >   	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
> >   	if (ret < 0) {
> >   		dma_addr = offset = 0;
> >  -		size = dev->coherent_dma_mask + 1;
> >  +		size = max(dev->coherent_dma_mask, dev-
> >coherent_dma_mask + 1);
> >
> > @@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum
> dev_dma_attr attr)
> >   	 * Assume dma valid range starts at 0 and covers the whole
> >   	 * coherent_dma_mask.
> >   	 */
> >  -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1,
> iommu,
> >  +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask +
> 1);
> >  +	arch_setup_dma_ops(dev, 0, size, iommu,
> >   			   attr == DEV_DMA_COHERENT);
> >
> > With the above fixes, DT boot works fine. But we still get the below crash
> on ACPI
> >
> >>> [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> >>> [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> >>> [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> >> vfio_virqfd vfio ixgbevf ixgb
> 
> I can't see how ACPI vs. DT would make any difference to the domain
> attach/detach mechanics getting into an invalid state (if the DT case
> works then unbinding the ixgbevf driver clearly does release the Stream
> ID correctly in general), so I'm more inclined to believe that something
> goes wrong with the fwspec in the ACPI path such that we end up touching
> the wrong STE altogether.

Ok. We will debug this separately and update.

Thanks,
Shameer
 

[-- Attachment #2: dmesg_4.11_hisi_patch3.log --]
[-- Type: application/octet-stream, Size: 52140 bytes --]

[    0.000000] Booting Linux on physical CPU 0x10000
[    0.000000] Linux version 4.11.0-rc1-g74f9bd0-dirty (shameer@shameer-ubuntu) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #156 SMP PREEMPT Mon Mar 27 13:13:50 BST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd082]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by EDK II
[    0.000000] efi:  SMBIOS=0x3f050000  SMBIOS 3.0=0x39b00000  ACPI=0x39bd0000  ACPI 2.0=0x39bd0014  MEMATTR=0x3c1b7018 
[    0.000000] cma: Reserved 16 MiB at 0x000000003e000000
[    0.000000] NUMA: Adding memblock [0x0 - 0x3fffffff] on node 0
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: Warning: invalid memblk node 4 [mem 0x1040000000-0x17fbffffff]
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] NUMA: Adding memblock [0x0 - 0x257ff] on node 0
[    0.000000] NUMA: Adding memblock [0x25800 - 0x3187ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31880000 - 0x3188ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31890000 - 0x318fafff] on node 0
[    0.000000] NUMA: Adding memblock [0x318fb000 - 0x318fefff] on node 0
[    0.000000] NUMA: Adding memblock [0x318ff000 - 0x3999ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x399a0000 - 0x3a05ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3a060000 - 0x3f04ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f050000 - 0x3f07ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f080000 - 0x3fbfffff] on node 0
[    0.000000] NUMA: Adding memblock [0x1040000000 - 0x17fbffffff] on node 0
[    0.000000] NUMA: Initmem setup node 0 [mem 0x00000000-0x17fbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x17fbfc4d00-0x17fbfc67ff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000017fbffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000000024fff]
[    0.000000]   node   0: [mem 0x0000000000026000-0x000000003187ffff]
[    0.000000]   node   0: [mem 0x0000000031880000-0x000000003188ffff]
[    0.000000]   node   0: [mem 0x0000000031890000-0x00000000318fafff]
[    0.000000]   node   0: [mem 0x00000000318fb000-0x00000000318fefff]
[    0.000000]   node   0: [mem 0x00000000318ff000-0x000000003999ffff]
[    0.000000]   node   0: [mem 0x00000000399a0000-0x000000003a05ffff]
[    0.000000]   node   0: [mem 0x000000003a060000-0x000000003f04ffff]
[    0.000000]   node   0: [mem 0x000000003f050000-0x000000003f07ffff]
[    0.000000]   node   0: [mem 0x000000003f080000-0x000000003fbfffff]
[    0.000000]   node   0: [mem 0x0000001040000000-0x00000017fbffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] On node 0 totalpages: 8371199
[    0.000000]   DMA zone: 4080 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261119 pages, LIFO batch:31
[    0.000000]   Normal zone: 126720 pages used for memmap
[    0.000000]   Normal zone: 8110080 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 23 pages/cpu @ffff8017db640000 s56960 r8192 d29056 u94208
[    0.000000] pcpu-alloc: s56960 r8192 d29056 u94208 alloc=23*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23 
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31 
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39 
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42 [0] 43 [0] 44 [0] 45 [0] 46 [0] 47 
[    0.000000] pcpu-alloc: [0] 48 [0] 49 [0] 50 [0] 51 [0] 52 [0] 53 [0] 54 [0] 55 
[    0.000000] pcpu-alloc: [0] 56 [0] 57 [0] 58 [0] 59 [0] 60 [0] 61 [0] 62 [0] 63 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 8240399
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/shameer/Image_smmu_4.11 rdinit=/init console=ttyAMA0,115200 earlycon=sbsa-uart,mmio,0x602b0000 pcie_aspm=off acpi=off noinitrd root=/dev/nfs rw nfsroot=172.18.45.166:/home/shameer/sherlok-files/ubuntu-sher/ubuntu,nfsvers=3 ip=dhcp::eth4:on:172.18.45.1:8.8.8.8 user_debug=1 nfsrootdebug pcie_acs_override=downstream
[    0.000000] PCIe ASPM is disabled
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 258048 bytes
[    0.000000] log_buf_len min size: 131072 bytes
[    0.000000] log_buf_len: 524288 bytes
[    0.000000] early log buf free: 125900(96%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] software IO TLB [mem 0x359a0000-0x399a0000] (64MB) mapped at [ffff8000359a0000-ffff80003999ffff]
[    0.000000] Memory: 32848756K/33484796K available (8828K kernel code, 964K rwdata, 3976K rodata, 1088K init, 398K bss, 619656K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008920000   (  8832 KB)
[    0.000000]     .rodata : 0xffff000008920000 - 0xffff000008d10000   (  4032 KB)
[    0.000000]       .init : 0xffff000008d10000 - 0xffff000008e20000   (  1088 KB)
[    0.000000]       .data : 0xffff000008e20000 - 0xffff000008f11200   (   965 KB)
[    0.000000]        .bss : 0xffff000008f11200 - 0xffff000008f74db8   (   399 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e005ff00000   (  1535 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff8017fc000000   ( 98240 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] ITS [mem 0x4c000000-0x4c03ffff]
[    0.000000] ITS@0x000000004c000000: allocated 524288 Devices @17da800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 2048 Virtual CPUs @17db024000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 512 Interrupt Collections @17db021000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x6c000000-0x6c03ffff]
[    0.000000] ITS@0x000000006c000000: allocated 524288 Devices @17da400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 2048 Virtual CPUs @17db028000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 512 Interrupt Collections @17db023000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0xc6000000-0xc603ffff]
[    0.000000] ITS@0x00000000c6000000: allocated 524288 Devices @17da000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 2048 Virtual CPUs @17db02c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 512 Interrupt Collections @17db060000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x8c6000000-0x8c603ffff]
[    0.000000] ITS@0x00000008c6000000: allocated 524288 Devices @17d9c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 2048 Virtual CPUs @17db064000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 512 Interrupt Collections @17db061000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4004c000000-0x4004c03ffff]
[    0.000000] ITS@0x000004004c000000: allocated 524288 Devices @17d9800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 2048 Virtual CPUs @17db068000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 512 Interrupt Collections @17db062000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4006c000000-0x4006c03ffff]
[    0.000000] ITS@0x000004006c000000: allocated 524288 Devices @17d9400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 2048 Virtual CPUs @17db06c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 512 Interrupt Collections @17db0a0000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x400c6000000-0x400c603ffff]
[    0.000000] ITS@0x00000400c6000000: allocated 524288 Devices @17d9000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 2048 Virtual CPUs @17db0a4000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 512 Interrupt Collections @17db0a1000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x408c6000000-0x408c603ffff]
[    0.000000] ITS@0x00000408c6000000: allocated 524288 Devices @17d8c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 2048 Virtual CPUs @17db0a8000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 512 Interrupt Collections @17db0a2000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000017db0d0000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 10000 region 0:0x000000004d100000
[    0.000000] CPU0: using LPI pending table @0x00000017db0e0000
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000001] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.000105] Console: colour dummy device 80x25
[    0.000137] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.000141] pid_max: default: 65536 minimum: 512
[    0.000184] Security Framework initialized
[    0.003838] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.013302] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.017183] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.017242] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.043989] ASID allocator initialised with 65536 entries
[    0.064017] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064021] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064024] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064027] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064031] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064034] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064037] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064040] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064066] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064069] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064072] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064075] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064078] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064081] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064084] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064087] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064273] Remapping and enabling EFI services.
[    0.064309]   EFI remap 0x00000000399a0000 => 0000000020000000
[    0.064330]   EFI remap 0x00000000399f0000 => 0000000020050000
[    0.064339]   EFI remap 0x0000000039a40000 => 00000000200a0000
[    0.064370]   EFI remap 0x0000000039af0000 => 00000000200b0000
[    0.064392]   EFI remap 0x0000000039b70000 => 0000000020130000
[    0.064413]   EFI remap 0x0000000039be0000 => 0000000020180000
[    0.064439]   EFI remap 0x0000000039c30000 => 00000000201d0000
[    0.064459]   EFI remap 0x0000000039c80000 => 0000000020220000
[    0.064481]   EFI remap 0x0000000039cd0000 => 0000000020270000
[    0.064502]   EFI remap 0x0000000039d20000 => 00000000202c0000
[    0.064524]   EFI remap 0x0000000039d70000 => 0000000020310000
[    0.064546]   EFI remap 0x0000000039dc0000 => 0000000020360000
[    0.064566]   EFI remap 0x0000000039e10000 => 00000000203b0000
[    0.064597]   EFI remap 0x0000000039e60000 => 0000000020400000
[    0.064618]   EFI remap 0x0000000039ed0000 => 0000000020470000
[    0.064639]   EFI remap 0x0000000039f20000 => 00000000204c0000
[    0.064661]   EFI remap 0x0000000039f70000 => 0000000020510000
[    0.064683]   EFI remap 0x0000000039fc0000 => 0000000020560000
[    0.064704]   EFI remap 0x000000003a010000 => 00000000205b0000
[    0.064722]   EFI remap 0x000000003f050000 => 0000000020600000
[    0.064731]   EFI remap 0x0000000078000000 => 0000000020630000
[    0.064737]   EFI remap 0x00000000a4000000 => 0000000020800000
[    0.064749]   EFI remap 0x00000000a6000000 => 0000000021800000
[    0.064758]   EFI remap 0x00000000d00e0000 => 0000000021810000
[    0.080009] smp: Bringing up secondary CPUs ...
[    0.162612] Detected PIPT I-cache on CPU1
[    0.162623] GICv3: CPU1: found redistributor 10001 region 0:0x000000004d140000
[    0.162646] CPU1: using LPI pending table @0x00000017d5850000
[    0.162793] CPU1: Booted secondary processor [410fd082]
[    0.245232] Detected PIPT I-cache on CPU2
[    0.245239] GICv3: CPU2: found redistributor 10002 region 0:0x000000004d180000
[    0.245261] CPU2: using LPI pending table @0x00000017d5880000
[    0.245401] CPU2: Booted secondary processor [410fd082]
[    0.327854] Detected PIPT I-cache on CPU3
[    0.327859] GICv3: CPU3: found redistributor 10003 region 0:0x000000004d1c0000
[    0.327882] CPU3: using LPI pending table @0x00000017d58c0000
[    0.328023] CPU3: Booted secondary processor [410fd082]
[    0.410479] Detected PIPT I-cache on CPU4
[    0.410490] GICv3: CPU4: found redistributor 10100 region 0:0x000000004d200000
[    0.410513] CPU4: using LPI pending table @0x00000017d58f0000
[    0.410655] CPU4: Booted secondary processor [410fd082]
[    0.493101] Detected PIPT I-cache on CPU5
[    0.493108] GICv3: CPU5: found redistributor 10101 region 0:0x000000004d240000
[    0.493129] CPU5: using LPI pending table @0x00000017d5920000
[    0.493269] CPU5: Booted secondary processor [410fd082]
[    0.575725] Detected PIPT I-cache on CPU6
[    0.575732] GICv3: CPU6: found redistributor 10102 region 0:0x000000004d280000
[    0.575755] CPU6: using LPI pending table @0x00000017d5950000
[    0.575895] CPU6: Booted secondary processor [410fd082]
[    0.658350] Detected PIPT I-cache on CPU7
[    0.658356] GICv3: CPU7: found redistributor 10103 region 0:0x000000004d2c0000
[    0.658379] CPU7: using LPI pending table @0x00000017d5980000
[    0.658520] CPU7: Booted secondary processor [410fd082]
[    0.740977] Detected PIPT I-cache on CPU8
[    0.740988] GICv3: CPU8: found redistributor 10200 region 0:0x000000004d300000
[    0.741011] CPU8: using LPI pending table @0x00000017d59b0000
[    0.741152] CPU8: Booted secondary processor [410fd082]
[    0.823600] Detected PIPT I-cache on CPU9
[    0.823606] GICv3: CPU9: found redistributor 10201 region 0:0x000000004d340000
[    0.823628] CPU9: using LPI pending table @0x00000017d59f0000
[    0.823767] CPU9: Booted secondary processor [410fd082]
[    0.906224] Detected PIPT I-cache on CPU10
[    0.906231] GICv3: CPU10: found redistributor 10202 region 0:0x000000004d380000
[    0.906253] CPU10: using LPI pending table @0x00000017d5a20000
[    0.906393] CPU10: Booted secondary processor [410fd082]
[    0.988849] Detected PIPT I-cache on CPU11
[    0.988855] GICv3: CPU11: found redistributor 10203 region 0:0x000000004d3c0000
[    0.988878] CPU11: using LPI pending table @0x00000017d5a70000
[    0.989108] CPU11: Booted secondary processor [410fd082]
[    1.071477] Detected PIPT I-cache on CPU12
[    1.071488] GICv3: CPU12: found redistributor 10300 region 0:0x000000004d400000
[    1.071511] CPU12: using LPI pending table @0x00000017d5aa0000
[    1.071654] CPU12: Booted secondary processor [410fd082]
[    1.154099] Detected PIPT I-cache on CPU13
[    1.154106] GICv3: CPU13: found redistributor 10301 region 0:0x000000004d440000
[    1.154129] CPU13: using LPI pending table @0x00000017d5ad0000
[    1.154267] CPU13: Booted secondary processor [410fd082]
[    1.236724] Detected PIPT I-cache on CPU14
[    1.236731] GICv3: CPU14: found redistributor 10302 region 0:0x000000004d480000
[    1.236753] CPU14: using LPI pending table @0x00000017d5b00000
[    1.236894] CPU14: Booted secondary processor [410fd082]
[    1.319348] Detected PIPT I-cache on CPU15
[    1.319355] GICv3: CPU15: found redistributor 10303 region 0:0x000000004d4c0000
[    1.319378] CPU15: using LPI pending table @0x00000017d5b40000
[    1.319519] CPU15: Booted secondary processor [410fd082]
[    1.401626] Detected PIPT I-cache on CPU16
[    1.401635] GICv3: CPU16: found redistributor 30000 region 1:0x000000006d100000
[    1.401652] CPU16: using LPI pending table @0x00000017d5b70000
[    1.401758] CPU16: Booted secondary processor [410fd082]
[    1.484254] Detected PIPT I-cache on CPU17
[    1.484261] GICv3: CPU17: found redistributor 30001 region 1:0x000000006d140000
[    1.484278] CPU17: using LPI pending table @0x00000017d5b90000
[    1.484384] CPU17: Booted secondary processor [410fd082]
[    1.566884] Detected PIPT I-cache on CPU18
[    1.566891] GICv3: CPU18: found redistributor 30002 region 1:0x000000006d180000
[    1.566910] CPU18: using LPI pending table @0x00000017d5bd0000
[    1.567017] CPU18: Booted secondary processor [410fd082]
[    1.649513] Detected PIPT I-cache on CPU19
[    1.649520] GICv3: CPU19: found redistributor 30003 region 1:0x000000006d1c0000
[    1.649537] CPU19: using LPI pending table @0x00000017d5400000
[    1.649643] CPU19: Booted secondary processor [410fd082]
[    1.732144] Detected PIPT I-cache on CPU20
[    1.732153] GICv3: CPU20: found redistributor 30100 region 1:0x000000006d200000
[    1.732170] CPU20: using LPI pending table @0x00000017d5430000
[    1.732277] CPU20: Booted secondary processor [410fd082]
[    1.814771] Detected PIPT I-cache on CPU21
[    1.814779] GICv3: CPU21: found redistributor 30101 region 1:0x000000006d240000
[    1.814796] CPU21: using LPI pending table @0x00000017d5460000
[    1.814901] CPU21: Booted secondary processor [410fd082]
[    1.897401] Detected PIPT I-cache on CPU22
[    1.897408] GICv3: CPU22: found redistributor 30102 region 1:0x000000006d280000
[    1.897425] CPU22: using LPI pending table @0x00000017d5490000
[    1.897529] CPU22: Booted secondary processor [410fd082]
[    1.980030] Detected PIPT I-cache on CPU23
[    1.980038] GICv3: CPU23: found redistributor 30103 region 1:0x000000006d2c0000
[    1.980055] CPU23: using LPI pending table @0x00000017d54c0000
[    1.980161] CPU23: Booted secondary processor [410fd082]
[    2.062661] Detected PIPT I-cache on CPU24
[    2.062671] GICv3: CPU24: found redistributor 30200 region 1:0x000000006d300000
[    2.062688] CPU24: using LPI pending table @0x00000017d54f0000
[    2.062794] CPU24: Booted secondary processor [410fd082]
[    2.145288] Detected PIPT I-cache on CPU25
[    2.145296] GICv3: CPU25: found redistributor 30201 region 1:0x000000006d340000
[    2.145313] CPU25: using LPI pending table @0x00000017d5530000
[    2.145419] CPU25: Booted secondary processor [410fd082]
[    2.227917] Detected PIPT I-cache on CPU26
[    2.227925] GICv3: CPU26: found redistributor 30202 region 1:0x000000006d380000
[    2.227943] CPU26: using LPI pending table @0x00000017d5560000
[    2.228047] CPU26: Booted secondary processor [410fd082]
[    2.310547] Detected PIPT I-cache on CPU27
[    2.310555] GICv3: CPU27: found redistributor 30203 region 1:0x000000006d3c0000
[    2.310572] CPU27: using LPI pending table @0x00000017d5590000
[    2.310679] CPU27: Booted secondary processor [410fd082]
[    2.393177] Detected PIPT I-cache on CPU28
[    2.393187] GICv3: CPU28: found redistributor 30300 region 1:0x000000006d400000
[    2.393205] CPU28: using LPI pending table @0x00000017d55c0000
[    2.393309] CPU28: Booted secondary processor [410fd082]
[    2.475805] Detected PIPT I-cache on CPU29
[    2.475813] GICv3: CPU29: found redistributor 30301 region 1:0x000000006d440000
[    2.475830] CPU29: using LPI pending table @0x00000017d55f0000
[    2.475935] CPU29: Booted secondary processor [410fd082]
[    2.558433] Detected PIPT I-cache on CPU30
[    2.558441] GICv3: CPU30: found redistributor 30302 region 1:0x000000006d480000
[    2.558458] CPU30: using LPI pending table @0x00000017d5630000
[    2.558562] CPU30: Booted secondary processor [410fd082]
[    2.641062] Detected PIPT I-cache on CPU31
[    2.641070] GICv3: CPU31: found redistributor 30303 region 1:0x000000006d4c0000
[    2.641087] CPU31: using LPI pending table @0x00000017d5660000
[    2.641192] CPU31: Booted secondary processor [410fd082]
[    2.723375] Detected PIPT I-cache on CPU32
[    2.723411] GICv3: CPU32: found redistributor 50000 region 2:0x000004004d100000
[    2.723462] CPU32: using LPI pending table @0x00000017d5690000
[    2.723612] CPU32: Booted secondary processor [410fd082]
[    2.805997] Detected PIPT I-cache on CPU33
[    2.806014] GICv3: CPU33: found redistributor 50001 region 2:0x000004004d140000
[    2.806055] CPU33: using LPI pending table @0x00000017d56c0000
[    2.806160] CPU33: Booted secondary processor [410fd082]
[    2.888634] Detected PIPT I-cache on CPU34
[    2.888650] GICv3: CPU34: found redistributor 50002 region 2:0x000004004d180000
[    2.888693] CPU34: using LPI pending table @0x00000017d56f0000
[    2.888799] CPU34: Booted secondary processor [410fd082]
[    2.971274] Detected PIPT I-cache on CPU35
[    2.971290] GICv3: CPU35: found redistributor 50003 region 2:0x000004004d1c0000
[    2.971333] CPU35: using LPI pending table @0x00000017d5720000
[    2.971438] CPU35: Booted secondary processor [410fd082]
[    3.053915] Detected PIPT I-cache on CPU36
[    3.053932] GICv3: CPU36: found redistributor 50100 region 2:0x000004004d200000
[    3.053975] CPU36: using LPI pending table @0x00000017d5760000
[    3.054082] CPU36: Booted secondary processor [410fd082]
[    3.136554] Detected PIPT I-cache on CPU37
[    3.136570] GICv3: CPU37: found redistributor 50101 region 2:0x000004004d240000
[    3.136613] CPU37: using LPI pending table @0x00000017d5790000
[    3.136717] CPU37: Booted secondary processor [410fd082]
[    3.219195] Detected PIPT I-cache on CPU38
[    3.219211] GICv3: CPU38: found redistributor 50102 region 2:0x000004004d280000
[    3.219254] CPU38: using LPI pending table @0x00000017d57c0000
[    3.219360] CPU38: Booted secondary processor [410fd082]
[    3.301835] Detected PIPT I-cache on CPU39
[    3.301852] GICv3: CPU39: found redistributor 50103 region 2:0x000004004d2c0000
[    3.301895] CPU39: using LPI pending table @0x00000017d57f0000
[    3.302000] CPU39: Booted secondary processor [410fd082]
[    3.384477] Detected PIPT I-cache on CPU40
[    3.384494] GICv3: CPU40: found redistributor 50200 region 2:0x000004004d300000
[    3.384538] CPU40: using LPI pending table @0x00000017d5020000
[    3.384646] CPU40: Booted secondary processor [410fd082]
[    3.467115] Detected PIPT I-cache on CPU41
[    3.467131] GICv3: CPU41: found redistributor 50201 region 2:0x000004004d340000
[    3.467174] CPU41: using LPI pending table @0x00000017d5050000
[    3.467279] CPU41: Booted secondary processor [410fd082]
[    3.549753] Detected PIPT I-cache on CPU42
[    3.549770] GICv3: CPU42: found redistributor 50202 region 2:0x000004004d380000
[    3.549815] CPU42: using LPI pending table @0x00000017d5080000
[    3.549921] CPU42: Booted secondary processor [410fd082]
[    3.632394] Detected PIPT I-cache on CPU43
[    3.632410] GICv3: CPU43: found redistributor 50203 region 2:0x000004004d3c0000
[    3.632454] CPU43: using LPI pending table @0x00000017d50c0000
[    3.632559] CPU43: Booted secondary processor [410fd082]
[    3.715034] Detected PIPT I-cache on CPU44
[    3.715052] GICv3: CPU44: found redistributor 50300 region 2:0x000004004d400000
[    3.715097] CPU44: using LPI pending table @0x00000017d50e0000
[    3.715204] CPU44: Booted secondary processor [410fd082]
[    3.797673] Detected PIPT I-cache on CPU45
[    3.797690] GICv3: CPU45: found redistributor 50301 region 2:0x000004004d440000
[    3.797732] CPU45: using LPI pending table @0x00000017d5120000
[    3.797836] CPU45: Booted secondary processor [410fd082]
[    3.880315] Detected PIPT I-cache on CPU46
[    3.880331] GICv3: CPU46: found redistributor 50302 region 2:0x000004004d480000
[    3.880374] CPU46: using LPI pending table @0x00000017d5150000
[    3.880481] CPU46: Booted secondary processor [410fd082]
[    3.962953] Detected PIPT I-cache on CPU47
[    3.962970] GICv3: CPU47: found redistributor 50303 region 2:0x000004004d4c0000
[    3.963014] CPU47: using LPI pending table @0x00000017d5180000
[    3.963119] CPU47: Booted secondary processor [410fd082]
[    4.045250] Detected PIPT I-cache on CPU48
[    4.045273] GICv3: CPU48: found redistributor 70000 region 3:0x000004006d100000
[    4.045318] CPU48: using LPI pending table @0x00000017d51b0000
[    4.045465] CPU48: Booted secondary processor [410fd082]
[    4.127889] Detected PIPT I-cache on CPU49
[    4.127908] GICv3: CPU49: found redistributor 70001 region 3:0x000004006d140000
[    4.127953] CPU49: using LPI pending table @0x00000017d51e0000
[    4.128097] CPU49: Booted secondary processor [410fd082]
[    4.210530] Detected PIPT I-cache on CPU50
[    4.210550] GICv3: CPU50: found redistributor 70002 region 3:0x000004006d180000
[    4.210595] CPU50: using LPI pending table @0x00000017d5210000
[    4.210741] CPU50: Booted secondary processor [410fd082]
[    4.293170] Detected PIPT I-cache on CPU51
[    4.293189] GICv3: CPU51: found redistributor 70003 region 3:0x000004006d1c0000
[    4.293235] CPU51: using LPI pending table @0x00000017d5250000
[    4.293378] CPU51: Booted secondary processor [410fd082]
[    4.375814] Detected PIPT I-cache on CPU52
[    4.375836] GICv3: CPU52: found redistributor 70100 region 3:0x000004006d200000
[    4.375885] CPU52: using LPI pending table @0x00000017d5280000
[    4.376033] CPU52: Booted secondary processor [410fd082]
[    4.458453] Detected PIPT I-cache on CPU53
[    4.458472] GICv3: CPU53: found redistributor 70101 region 3:0x000004006d240000
[    4.458519] CPU53: using LPI pending table @0x00000017d52b0000
[    4.458664] CPU53: Booted secondary processor [410fd082]
[    4.541096] Detected PIPT I-cache on CPU54
[    4.541115] GICv3: CPU54: found redistributor 70102 region 3:0x000004006d280000
[    4.541161] CPU54: using LPI pending table @0x00000017d52e0000
[    4.541306] CPU54: Booted secondary processor [410fd082]
[    4.623738] Detected PIPT I-cache on CPU55
[    4.623757] GICv3: CPU55: found redistributor 70103 region 3:0x000004006d2c0000
[    4.623803] CPU55: using LPI pending table @0x00000017d5320000
[    4.623947] CPU55: Booted secondary processor [410fd082]
[    4.706381] Detected PIPT I-cache on CPU56
[    4.706403] GICv3: CPU56: found redistributor 70200 region 3:0x000004006d300000
[    4.706451] CPU56: using LPI pending table @0x00000017d5340000
[    4.706599] CPU56: Booted secondary processor [410fd082]
[    4.789020] Detected PIPT I-cache on CPU57
[    4.789040] GICv3: CPU57: found redistributor 70201 region 3:0x000004006d340000
[    4.789088] CPU57: using LPI pending table @0x00000017d5380000
[    4.789230] CPU57: Booted secondary processor [410fd082]
[    4.871663] Detected PIPT I-cache on CPU58
[    4.871682] GICv3: CPU58: found redistributor 70202 region 3:0x000004006d380000
[    4.871729] CPU58: using LPI pending table @0x00000017d53b0000
[    4.871874] CPU58: Booted secondary processor [410fd082]
[    4.954305] Detected PIPT I-cache on CPU59
[    4.954324] GICv3: CPU59: found redistributor 70203 region 3:0x000004006d3c0000
[    4.954370] CPU59: using LPI pending table @0x00000017d53e0000
[    4.954514] CPU59: Booted secondary processor [410fd082]
[    5.036947] Detected PIPT I-cache on CPU60
[    5.036970] GICv3: CPU60: found redistributor 70300 region 3:0x000004006d400000
[    5.037017] CPU60: using LPI pending table @0x00000017d4c10000
[    5.037164] CPU60: Booted secondary processor [410fd082]
[    5.119587] Detected PIPT I-cache on CPU61
[    5.119606] GICv3: CPU61: found redistributor 70301 region 3:0x000004006d440000
[    5.119652] CPU61: using LPI pending table @0x00000017d4c40000
[    5.119795] CPU61: Booted secondary processor [410fd082]
[    5.202229] Detected PIPT I-cache on CPU62
[    5.202249] GICv3: CPU62: found redistributor 70302 region 3:0x000004006d480000
[    5.202295] CPU62: using LPI pending table @0x00000017d4c70000
[    5.202441] CPU62: Booted secondary processor [410fd082]
[    5.284870] Detected PIPT I-cache on CPU63
[    5.284889] GICv3: CPU63: found redistributor 70303 region 3:0x000004006d4c0000
[    5.284936] CPU63: using LPI pending table @0x00000017d4cd0000
[    5.285078] CPU63: Booted secondary processor [410fd082]
[    5.285155] smp: Brought up 1 node, 64 CPUs
[    5.285297] SMP: Total of 64 processors activated.
[    5.285301] CPU features: detected feature: GIC system register CPU interface
[    5.285304] CPU features: detected feature: 32-bit EL0 Support
[    5.285541] CPU: All CPU(s) started at EL2
[    5.286196] devtmpfs: initialized
[    5.287679] SMBIOS 3.0.0 present.
[    5.287686] DMI: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 03/16/2017
[    5.287921] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.288154] futex hash table entries: 16384 (order: 9, 2097152 bytes)
[    5.288748] pinctrl core: initialized pinctrl subsystem
[    5.289166] NET: Registered protocol family 16
[    5.290546] cpuidle: using governor menu
[    5.290736] irq: no irq domain found for /interrupt-controller@a0080000/intc_smmu_pcie !
[    5.290901] arm-smmu-v3 a0040000.smmu_pcie: option hisilicon,broken-prefetch-cmd
[    5.290908] arm-smmu-v3 a0040000.smmu_pcie: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.294474] irq: no irq domain found for /interrupt-controller@c0080000/intc_smmu_dsa !
[    5.294574] arm-smmu-v3 c0040000.smmu_dsa: option hisilicon,broken-prefetch-cmd
[    5.294589] arm-smmu-v3 c0040000.smmu_dsa: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.298006] vdso: 2 pages (1 code @ ffff000008927000, 1 data @ ffff000008e25000)
[    5.298048] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    5.298984] DMA: preallocated 256 KiB pool for atomic allocations
[    5.299157] Serial: AMBA PL011 UART driver
[    5.299631] irq: no irq domain found for /interrupt-controller@60080000/uart_intc !
[    5.299710] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299850] iommu: Adding device a7030000.ohci to group 0
[    5.299893] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299960] iommu: Adding device a7020000.ehci to group 1
[    5.301608] irq: no irq domain found for /interrupt-controller@c0080000/intc_dsaf0 !
[    5.301731] iommu: Adding device c5000000.dsa to group 2
[    5.301885] irq: no irq domain found for /interrupt-controller@c0080000/intc-roce !
[    5.307388] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    5.308320] ACPI: Interpreter disabled.
[    5.308562] vgaarb: loaded
[    5.308650] SCSI subsystem initialized
[    5.308813] libata version 3.00 loaded.
[    5.308914] usbcore: registered new interface driver usbfs
[    5.308934] usbcore: registered new interface driver hub
[    5.309117] usbcore: registered new device driver usb
[    5.309229] pps_core: LinuxPPS API ver. 1 registered
[    5.309231] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    5.309239] PTP clock support registered
[    5.309323] Registered efivars operations
[    5.309503] Advanced Linux Sound Architecture Driver Initialized.
[    5.310075] clocksource: Switched to clocksource arch_sys_counter
[    5.310221] VFS: Disk quotas dquot_6.6.0
[    5.310250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    5.310353] pnp: PnP ACPI: disabled
[    5.313377] NET: Registered protocol family 2
[    5.314108] TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
[    5.314882] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    5.315277] TCP: Hash tables configured (established 262144 bind 65536)
[    5.315550] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    5.315787] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    5.316176] NET: Registered protocol family 1
[    5.316505] RPC: Registered named UNIX socket transport module.
[    5.316507] RPC: Registered udp transport module.
[    5.316510] RPC: Registered tcp transport module.
[    5.316511] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.316516] PCI: CLS 0 bytes, default 128
[    5.318900] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    5.319160] kvm [1]: 8-bit VMID
[    5.319163] kvm [1]: IDMAP page: 90e000
[    5.319166] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[    5.320161] kvm [1]: Hyp mode initialized successfully
[    5.320274] kvm [1]: vgic-v2@fe020000
[    5.320366] kvm [1]: GIC system register CPU interface enabled
[    5.321105] kvm [1]: vgic interrupt IRQ1
[    5.321113] kvm [1]: virtual timer IRQ4
[    5.324516] audit: initializing netlink subsys (disabled)
[    5.324576] audit: type=2000 audit(3.644:1): state=initialized audit_enabled=0 res=1
[    5.324955] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    5.327563] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    5.327875] NFS: Registering the id_resolver key type
[    5.327888] Key type id_resolver registered
[    5.327891] Key type id_legacy registered
[    5.327897] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    5.327956] 9p: Installing v9fs 9p2000 file system support
[    5.329207] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    5.329212] io scheduler noop registered
[    5.329275] io scheduler cfq registered (default)
[    5.329278] io scheduler mq-deadline registered
[    5.331779] OF: PCI: host bridge /soc/pcie@a00a0000 ranges:
[    5.331789] OF: PCI:   MEM 0xa8800000..0xaffeffff -> 0xa8800000
[    5.331794] OF: PCI:    IO 0xafff0000..0xafffffff -> 0x00000000
[    5.331832] hisi-pcie-almost-ecam a8000000.pcie: ECAM at [mem 0xa8000000-0xa87fffff] for [bus 80-87]
[    5.331887] hisi-pcie-almost-ecam a8000000.pcie: PCI host bridge to bus 0000:80
[    5.331891] pci_bus 0000:80: root bus resource [bus 80-87]
[    5.331894] pci_bus 0000:80: root bus resource [mem 0xa8800000-0xaffeffff]
[    5.331897] pci_bus 0000:80: root bus resource [io  0x0000-0xffff]
[    5.331916] pci 0000:80:00.0: [19e5:1610] type 01 class 0x060400
[    5.331929] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331935] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331939] pci 0000:80:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[    5.331942] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331948] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331953] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331959] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331990] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x52 may corrupt adjacent RW1C bits
[    5.332014] pci 0000:80:00.0: supports D1 D2
[    5.332017] pci 0000:80:00.0: PME# supported from D0 D1 D3hot
[    5.332021] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x44 may corrupt adjacent RW1C bits
[    5.332233] iommu: Adding device 0000:80:00.0 to group 3
[    5.332259] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332265] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332305] pci_bus 0000:81: busn_res: can not insert [bus 81-ff] under [bus 80-87] (conflicts with (null) [bus 80-87])
[    5.332322] pci 0000:81:00.0: [8086:10fb] type 00 class 0x020000
[    5.332339] pci 0000:81:00.0: reg 0x10: [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.332345] pci 0000:81:00.0: reg 0x18: [io  0x0000-0x001f]
[    5.332360] pci 0000:81:00.0: reg 0x20: [mem 0xa9804000-0xa9807fff 64bit pref]
[    5.332367] pci 0000:81:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.332433] pci 0000:81:00.0: PME# supported from D0 D3hot
[    5.332468] pci 0000:81:00.0: reg 0x184: [mem 0xa9704000-0xa9707fff 64bit pref]
[    5.332471] pci 0000:81:00.0: VF(n) BAR0 space: [mem 0xa9704000-0xa9803fff 64bit pref] (contains BAR0 for 64 VFs)
[    5.332488] pci 0000:81:00.0: reg 0x190: [mem 0xa9604000-0xa9607fff 64bit pref]
[    5.332490] pci 0000:81:00.0: VF(n) BAR3 space: [mem 0xa9604000-0xa9703fff 64bit pref] (contains BAR3 for 64 VFs)
[    5.332844] iommu: Adding device 0000:81:00.0 to group 4
[    5.332875] pci 0000:81:00.1: [8086:10fb] type 00 class 0x020000
[    5.332892] pci 0000:81:00.1: reg 0x10: [mem 0xa8c00000-0xa8ffffff 64bit pref]
[    5.332898] pci 0000:81:00.1: reg 0x18: [io  0x0000-0x001f]
[    5.332914] pci 0000:81:00.1: reg 0x20: [mem 0xa9600000-0xa9603fff 64bit pref]
[    5.332920] pci 0000:81:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.332984] pci 0000:81:00.1: PME# supported from D0 D3hot
[    5.333012] pci 0000:81:00.1: reg 0x184: [mem 0xa9500000-0xa9503fff 64bit pref]
[    5.333015] pci 0000:81:00.1: VF(n) BAR0 space: [mem 0xa9500000-0xa95fffff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333031] pci 0000:81:00.1: reg 0x190: [mem 0xa9400000-0xa9403fff 64bit pref]
[    5.333033] pci 0000:81:00.1: VF(n) BAR3 space: [mem 0xa9400000-0xa94fffff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333322] iommu: Adding device 0000:81:00.1 to group 5
[    5.333344] pci_bus 0000:81: busn_res: [bus 81-ff] end is updated to 81
[    5.333365] pci 0000:80:00.0: BAR 14: assigned [mem 0xa8800000-0xa8ffffff]
[    5.333369] pci 0000:80:00.0: BAR 15: assigned [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333372] pci 0000:80:00.0: BAR 0: assigned [mem 0xa9e00000-0xa9e0ffff]
[    5.333376] pci 0000:80:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.333381] pci 0000:81:00.0: BAR 0: assigned [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.333390] pci 0000:81:00.0: BAR 6: assigned [mem 0xa8800000-0xa8bfffff pref]
[    5.333393] pci 0000:81:00.1: BAR 0: assigned [mem 0xa9400000-0xa97fffff 64bit pref]
[    5.333401] pci 0000:81:00.1: BAR 6: assigned [mem 0xa8c00000-0xa8ffffff pref]
[    5.333404] pci 0000:81:00.0: BAR 4: assigned [mem 0xa9800000-0xa9803fff 64bit pref]
[    5.333413] pci 0000:81:00.0: BAR 7: assigned [mem 0xa9804000-0xa9903fff 64bit pref]
[    5.333418] pci 0000:81:00.0: BAR 10: assigned [mem 0xa9904000-0xa9a03fff 64bit pref]
[    5.333423] pci 0000:81:00.1: BAR 4: assigned [mem 0xa9a04000-0xa9a07fff 64bit pref]
[    5.333432] pci 0000:81:00.1: BAR 7: assigned [mem 0xa9a08000-0xa9b07fff 64bit pref]
[    5.333437] pci 0000:81:00.1: BAR 10: assigned [mem 0xa9b08000-0xa9c07fff 64bit pref]
[    5.333442] pci 0000:81:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    5.333447] pci 0000:81:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    5.333454] pci 0000:80:00.0: PCI bridge to [bus 81]
[    5.333458] pci 0000:80:00.0:   bridge window [io  0x1000-0x1fff]
[    5.333463] pci 0000:80:00.0:   bridge window [mem 0xa8800000-0xa8ffffff]
[    5.333466] pci 0000:80:00.0:   bridge window [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333483] pcieport 0000:80:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.334485] pcieport 0000:80:00.0: Signaling PME with IRQ 7
[    5.334527] pcieport 0000:80:00.0: AER enabled with IRQ 7
[    5.334648] ipmi message handler version 39.2
[    5.334654] ipmi device interface
[    5.334694] IPMI System Interface driver.
[    5.334721] ipmi_si: probing via SMBIOS
[    5.334724] ipmi_si: SMBIOS: io 0xe4 regsize 1 spacing 1 irq 0
[    5.334728] ipmi_si: Adding SMBIOS-specified bt state machine
[    5.334732] ipmi_si: Trying SMBIOS-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
[    5.334747] ipmi_si ipmi_si.0: Interface detection failed
[    5.335888] xenfs: not registering filesystem on non-xen platform
[    5.337169] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    5.337673] SuperH (H)SCI(F) driver initialized
[    5.337773] msm_serial: driver initialized
[    5.354990] loop: module loaded
[    5.355308] hisi_sas: driver version v1.6
[    5.356635] libphy: Fixed MDIO Bus: probed
[    5.357109] tun: Universal TUN/TAP device driver, 1.6
[    5.357111] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    5.357535] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    5.357537] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    5.357592] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    5.357594] igb: Copyright (c) 2007-2014 Intel Corporation.
[    5.357645] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    5.357647] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    5.357689] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.0.0-k
[    5.357691] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    5.357751] ixgbe 0000:81:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.358611] ixgbe 0000:81:00.0: enabling device (0000 -> 0002)
[    5.603275] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    5.603403] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available
[    5.603407] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    5.603492] ixgbe 0000:81:00.0: MAC: 2, PHY: 15, SFP+: 11, PBA No: FFFFFF-0FF
[    5.603495] ixgbe 0000:81:00.0: 68:a8:28:17:97:07
[    5.608297] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    5.608331] ixgbe 0000:81:00.1: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.609152] ixgbe 0000:81:00.1: enabling device (0000 -> 0002)
[    6.787178] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    6.787307] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available
[    6.787310] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    6.787393] ixgbe 0000:81:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    6.787395] ixgbe 0000:81:00.1: 68:a8:28:17:97:08
[    6.792053] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
[    6.792099] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[    6.792102] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    6.792166] sky2: driver version 1.30
[    6.792615] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.792618] ehci-pci: EHCI PCI platform driver
[    6.792633] ehci-platform: EHCI generic platform driver
[    6.792663] ehci-platform a7020000.ehci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.793574] ehci-platform a7020000.ehci: EHCI Host Controller
[    6.793604] ehci-platform a7020000.ehci: new USB bus registered, assigned bus number 1
[    6.793700] ehci-platform a7020000.ehci: irq 136, io mem 0xa7020000
[    6.806061] ehci-platform a7020000.ehci: USB 2.0 started, EHCI 1.00
[    6.806295] hub 1-0:1.0: USB hub found
[    6.806306] hub 1-0:1.0: 2 ports detected
[    6.806453] ehci-exynos: EHCI EXYNOS driver
[    6.806494] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    6.806519] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.806529] ohci-pci: OHCI PCI platform driver
[    6.806553] ohci-platform: OHCI generic platform driver
[    6.806573] ohci-platform a7030000.ohci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.807473] ohci-platform a7030000.ohci: Generic Platform OHCI controller
[    6.807483] ohci-platform a7030000.ohci: new USB bus registered, assigned bus number 2
[    6.807546] ohci-platform a7030000.ohci: irq 137, io mem 0xa7030000
[    6.870326] hub 2-0:1.0: USB hub found
[    6.870338] hub 2-0:1.0: 2 ports detected
[    6.870465] ohci-exynos: OHCI EXYNOS driver
[    6.870633] usbcore: registered new interface driver usb-storage
[    6.997025] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[    6.997205] i2c /dev entries driver
[    6.997868] sdhci: Secure Digital Host Controller Interface driver
[    6.997871] sdhci: Copyright(c) Pierre Ossman
[    6.997951] Synopsys Designware Multimedia Card Interface Driver
[    6.998132] sdhci-pltfm: SDHCI platform and OF driver helper
[    6.999163] ledtrig-cpu: registered to indicate activity on CPUs
[    6.999499] usbcore: registered new interface driver usbhid
[    6.999502] usbhid: USB HID core driver
[    7.000144] NET: Registered protocol family 17
[    7.000181] 9pnet: Installing 9P2000 support
[    7.000210] Key type dns_resolver registered
[    7.000460] registered taskstats version 1
[    7.000689] 602b0000.uart: ttyAMA0 at MMIO 0x602b0000 (irq = 138, base_baud = 0) is a SBSA
[    7.146062] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    7.294762] hub 1-1:1.0: USB hub found
[    7.294856] hub 1-1:1.0: 4 ports detected
[    7.414056] usb 1-2: new high-speed USB device number 3 using ehci-platform
[    7.562759] hub 1-2:1.0: USB hub found
[    7.562855] hub 1-2:1.0: 4 ports detected
[    7.850053] usb 1-2.1: new full-speed USB device number 4 using ehci-platform
[    7.962516] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7020000.ehci/usb1/1-2/1-2.1/1-2.1:1.0/0003:12D1:0003.0001/input/input0
[    8.022252] hid-generic 0003:12D1:0003.0001: input: USB HID v1.10 Keyboard [Keyboard/Mouse KVM 1.1.0] on usb-a7020000.ehci-2.1/input0
[    8.023043] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7020000.ehci/usb1/1-2/1-2.1/1-2.1:1.1/0003:12D1:0003.0002/input/input1
[    8.023080] hid-generic 0003:12D1:0003.0002: input: USB HID v1.10 Mouse [Keyboard/Mouse KVM 1.1.0] on usb-a7020000.ehci-2.1/input1
[   10.972582] console [ttyAMA0] enabled
[   11.039405] rtc-efi rtc-efi: setting system clock to 2017-03-27 12:17:52 UTC (1490617072)
[   11.338770] ixgbe 0000:81:00.0: registered PHC device on eth0
[   11.515447] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[   11.651534] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[   11.744809] ixgbe 0000:81:00.1: registered PHC device on eth1
[   11.878052] Sending DHCP requests ....
[   38.422250] random: fast init done
[   39.978047] .
[   52.514266] random: crng init done
[   63.554048] ., OK
[   64.572007] IP-Config: Got DHCP answer from 172.18.45.1, my address is 172.18.45.129
[   64.579746] IP-Config: Complete:
[   64.582974]      device=eth0, hwaddr=68:a8:28:17:97:07, ipaddr=172.18.45.129, mask=255.255.255.0, gw=172.18.45.1
[   64.593140]      host=172.18.45.129, domain=, nis-domain=(none)
[   64.599054]      bootserver=0.0.0.0, rootserver=172.18.45.166, rootpath=     nameserver0=8.8.8.8, nameserver1=4.4.4.4
[   64.610619] ixgbe 0000:81:00.1: removed PHC on eth1
[   65.018144] ALSA device list:
[   65.021102]   No soundcards found.
[   65.029925] VFS: Mounted root (nfs filesystem) on device 0:14.
[   65.036024] devtmpfs: mounted
[   65.039696] Freeing unused kernel memory: 1088K
[   65.322553] init: plymouth-upstart-bridge main process (1942) terminated with status 1
[   65.330532] init: plymouth-upstart-bridge main process ended, respawning
[   65.345855] init: plymouth-upstart-bridge main process (1992) terminated with status 1
[   65.354059] init: plymouth-upstart-bridge main process ended, respawning
[   65.544750] init: ureadahead main process (1945) terminated with status 5
[   73.154611] systemd-udevd[2193]: starting version 208
[   73.431168] ixgbe 0000:81:00.1 eth26: renamed from eth1
[   73.482158] systemd-udevd[2590]: renamed network interface eth1 to eth26
[   73.714196] Bluetooth: Core ver 2.22
[   73.714265] NET: Registered protocol family 31
[   73.714266] Bluetooth: HCI device and connection manager initialized
[   73.714279] Bluetooth: HCI socket layer initialized
[   73.714284] Bluetooth: L2CAP socket layer initialized
[   73.714299] Bluetooth: SCO socket layer initialized
[   74.373037] init: failsafe main process (2723) killed by TERM signal
[   74.485356] NET: Registered protocol family 10
[   74.488119] Segment Routing with IPv6
[   76.520945] ip_tables: (C) 2000-2006 Netfilter Core Team
[   76.624565] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   76.935734] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   77.915416] init: tty1 main process (3310) killed by TERM signal
[   78.048409] init: plymouth-upstart-bridge main process ended, respawning
[  142.404919] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  142.405042] ixgbe 0000:81:00.0: removed PHC on eth0
[  142.614933] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  142.635890] ixgbe 0000:81:00.0: registered PHC device on eth0
[  142.808352] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  142.851412] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  142.851906] iommu: Adding device 0000:81:10.0 to group 6
[  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
[  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
[  142.865692] ixgbevf 0000:81:10.0: MAC: 1
[  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  152.902894] nfs: server 172.18.45.166 not responding, still trying
[  188.980933] nfs: server 172.18.45.166 not responding, still trying
[  188.981298] nfs: server 172.18.45.166 OK
[  188.981593] nfs: server 172.18.45.166 OK
[  221.755626] VFIO - User Level meta-driver version: 0.3

[-- Attachment #3: dmesg_4.11_hisi_patch6.log --]
[-- Type: application/octet-stream, Size: 52509 bytes --]

[    0.000000] Booting Linux on physical CPU 0x10000
[    0.000000] Linux version 4.11.0-rc1-g2802757-dirty (shameer@shameer-ubuntu) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #154 SMP PREEMPT Mon Mar 27 12:24:59 BST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd082]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by EDK II
[    0.000000] efi:  SMBIOS=0x3f050000  SMBIOS 3.0=0x39b00000  ACPI=0x39bd0000  ACPI 2.0=0x39bd0014  MEMATTR=0x3c1b6018 
[    0.000000] cma: Reserved 16 MiB at 0x000000003e000000
[    0.000000] NUMA: Adding memblock [0x0 - 0x3fffffff] on node 0
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: Warning: invalid memblk node 4 [mem 0x1040000000-0x17fbffffff]
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] NUMA: Adding memblock [0x0 - 0x257ff] on node 0
[    0.000000] NUMA: Adding memblock [0x25800 - 0x3187ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31880000 - 0x3188ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31890000 - 0x318fafff] on node 0
[    0.000000] NUMA: Adding memblock [0x318fb000 - 0x318fefff] on node 0
[    0.000000] NUMA: Adding memblock [0x318ff000 - 0x3999ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x399a0000 - 0x3a05ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3a060000 - 0x3f04ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f050000 - 0x3f07ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f080000 - 0x3fbfffff] on node 0
[    0.000000] NUMA: Adding memblock [0x1040000000 - 0x17fbffffff] on node 0
[    0.000000] NUMA: Initmem setup node 0 [mem 0x00000000-0x17fbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x17fbfc4d00-0x17fbfc67ff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000017fbffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000000024fff]
[    0.000000]   node   0: [mem 0x0000000000026000-0x000000003187ffff]
[    0.000000]   node   0: [mem 0x0000000031880000-0x000000003188ffff]
[    0.000000]   node   0: [mem 0x0000000031890000-0x00000000318fafff]
[    0.000000]   node   0: [mem 0x00000000318fb000-0x00000000318fefff]
[    0.000000]   node   0: [mem 0x00000000318ff000-0x000000003999ffff]
[    0.000000]   node   0: [mem 0x00000000399a0000-0x000000003a05ffff]
[    0.000000]   node   0: [mem 0x000000003a060000-0x000000003f04ffff]
[    0.000000]   node   0: [mem 0x000000003f050000-0x000000003f07ffff]
[    0.000000]   node   0: [mem 0x000000003f080000-0x000000003fbfffff]
[    0.000000]   node   0: [mem 0x0000001040000000-0x00000017fbffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] On node 0 totalpages: 8371199
[    0.000000]   DMA zone: 4080 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261119 pages, LIFO batch:31
[    0.000000]   Normal zone: 126720 pages used for memmap
[    0.000000]   Normal zone: 8110080 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 23 pages/cpu @ffff8017db640000 s56960 r8192 d29056 u94208
[    0.000000] pcpu-alloc: s56960 r8192 d29056 u94208 alloc=23*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23 
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31 
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39 
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42 [0] 43 [0] 44 [0] 45 [0] 46 [0] 47 
[    0.000000] pcpu-alloc: [0] 48 [0] 49 [0] 50 [0] 51 [0] 52 [0] 53 [0] 54 [0] 55 
[    0.000000] pcpu-alloc: [0] 56 [0] 57 [0] 58 [0] 59 [0] 60 [0] 61 [0] 62 [0] 63 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 8240399
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/shameer/Image_smmu_4.11 rdinit=/init console=ttyAMA0,115200 earlycon=sbsa-uart,mmio,0x602b0000 pcie_aspm=off acpi=off noinitrd root=/dev/nfs rw nfsroot=172.18.45.166:/home/shameer/sherlok-files/ubuntu-sher/ubuntu,nfsvers=3 ip=dhcp::eth4:on:172.18.45.1:8.8.8.8 user_debug=1 nfsrootdebug pcie_acs_override=downstream
[    0.000000] PCIe ASPM is disabled
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 258048 bytes
[    0.000000] log_buf_len min size: 131072 bytes
[    0.000000] log_buf_len: 524288 bytes
[    0.000000] early log buf free: 125900(96%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] software IO TLB [mem 0x359a0000-0x399a0000] (64MB) mapped at [ffff8000359a0000-ffff80003999ffff]
[    0.000000] Memory: 32848756K/33484796K available (8828K kernel code, 964K rwdata, 3976K rodata, 1088K init, 398K bss, 619656K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008920000   (  8832 KB)
[    0.000000]     .rodata : 0xffff000008920000 - 0xffff000008d10000   (  4032 KB)
[    0.000000]       .init : 0xffff000008d10000 - 0xffff000008e20000   (  1088 KB)
[    0.000000]       .data : 0xffff000008e20000 - 0xffff000008f11200   (   965 KB)
[    0.000000]        .bss : 0xffff000008f11200 - 0xffff000008f74db8   (   399 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e005ff00000   (  1535 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff8017fc000000   ( 98240 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] ITS [mem 0x4c000000-0x4c03ffff]
[    0.000000] ITS@0x000000004c000000: allocated 524288 Devices @17da800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 2048 Virtual CPUs @17db024000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 512 Interrupt Collections @17db021000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x6c000000-0x6c03ffff]
[    0.000000] ITS@0x000000006c000000: allocated 524288 Devices @17da400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 2048 Virtual CPUs @17db028000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 512 Interrupt Collections @17db023000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0xc6000000-0xc603ffff]
[    0.000000] ITS@0x00000000c6000000: allocated 524288 Devices @17da000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 2048 Virtual CPUs @17db02c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 512 Interrupt Collections @17db060000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x8c6000000-0x8c603ffff]
[    0.000000] ITS@0x00000008c6000000: allocated 524288 Devices @17d9c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 2048 Virtual CPUs @17db064000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 512 Interrupt Collections @17db061000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4004c000000-0x4004c03ffff]
[    0.000000] ITS@0x000004004c000000: allocated 524288 Devices @17d9800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 2048 Virtual CPUs @17db068000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 512 Interrupt Collections @17db062000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4006c000000-0x4006c03ffff]
[    0.000000] ITS@0x000004006c000000: allocated 524288 Devices @17d9400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 2048 Virtual CPUs @17db06c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 512 Interrupt Collections @17db0a0000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x400c6000000-0x400c603ffff]
[    0.000000] ITS@0x00000400c6000000: allocated 524288 Devices @17d9000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 2048 Virtual CPUs @17db0a4000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 512 Interrupt Collections @17db0a1000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x408c6000000-0x408c603ffff]
[    0.000000] ITS@0x00000408c6000000: allocated 524288 Devices @17d8c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 2048 Virtual CPUs @17db0a8000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 512 Interrupt Collections @17db0a2000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000017db0d0000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 10000 region 0:0x000000004d100000
[    0.000000] CPU0: using LPI pending table @0x00000017db0e0000
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000001] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.000107] Console: colour dummy device 80x25
[    0.000139] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.000143] pid_max: default: 65536 minimum: 512
[    0.000182] Security Framework initialized
[    0.003835] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.013291] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.017172] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.017230] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.043989] ASID allocator initialised with 65536 entries
[    0.064017] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064021] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064024] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064027] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064030] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064033] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064036] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064039] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064065] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064068] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064071] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064074] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064077] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064080] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064083] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064086] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064272] Remapping and enabling EFI services.
[    0.064309]   EFI remap 0x00000000399a0000 => 0000000020000000
[    0.064331]   EFI remap 0x00000000399f0000 => 0000000020050000
[    0.064340]   EFI remap 0x0000000039a40000 => 00000000200a0000
[    0.064370]   EFI remap 0x0000000039af0000 => 00000000200b0000
[    0.064392]   EFI remap 0x0000000039b70000 => 0000000020130000
[    0.064413]   EFI remap 0x0000000039be0000 => 0000000020180000
[    0.064439]   EFI remap 0x0000000039c30000 => 00000000201d0000
[    0.064460]   EFI remap 0x0000000039c80000 => 0000000020220000
[    0.064482]   EFI remap 0x0000000039cd0000 => 0000000020270000
[    0.064504]   EFI remap 0x0000000039d20000 => 00000000202c0000
[    0.064525]   EFI remap 0x0000000039d70000 => 0000000020310000
[    0.064546]   EFI remap 0x0000000039dc0000 => 0000000020360000
[    0.064567]   EFI remap 0x0000000039e10000 => 00000000203b0000
[    0.064597]   EFI remap 0x0000000039e60000 => 0000000020400000
[    0.064619]   EFI remap 0x0000000039ed0000 => 0000000020470000
[    0.064641]   EFI remap 0x0000000039f20000 => 00000000204c0000
[    0.064662]   EFI remap 0x0000000039f70000 => 0000000020510000
[    0.064684]   EFI remap 0x0000000039fc0000 => 0000000020560000
[    0.064705]   EFI remap 0x000000003a010000 => 00000000205b0000
[    0.064723]   EFI remap 0x000000003f050000 => 0000000020600000
[    0.064732]   EFI remap 0x0000000078000000 => 0000000020630000
[    0.064738]   EFI remap 0x00000000a4000000 => 0000000020800000
[    0.064750]   EFI remap 0x00000000a6000000 => 0000000021800000
[    0.064759]   EFI remap 0x00000000d00e0000 => 0000000021810000
[    0.080009] smp: Bringing up secondary CPUs ...
[    0.162611] Detected PIPT I-cache on CPU1
[    0.162623] GICv3: CPU1: found redistributor 10001 region 0:0x000000004d140000
[    0.162644] CPU1: using LPI pending table @0x00000017d5850000
[    0.162790] CPU1: Booted secondary processor [410fd082]
[    0.245231] Detected PIPT I-cache on CPU2
[    0.245237] GICv3: CPU2: found redistributor 10002 region 0:0x000000004d180000
[    0.245260] CPU2: using LPI pending table @0x00000017d5880000
[    0.245397] CPU2: Booted secondary processor [410fd082]
[    0.327852] Detected PIPT I-cache on CPU3
[    0.327857] GICv3: CPU3: found redistributor 10003 region 0:0x000000004d1c0000
[    0.327880] CPU3: using LPI pending table @0x00000017d58c0000
[    0.328018] CPU3: Booted secondary processor [410fd082]
[    0.410477] Detected PIPT I-cache on CPU4
[    0.410487] GICv3: CPU4: found redistributor 10100 region 0:0x000000004d200000
[    0.410510] CPU4: using LPI pending table @0x00000017d58f0000
[    0.410651] CPU4: Booted secondary processor [410fd082]
[    0.493098] Detected PIPT I-cache on CPU5
[    0.493104] GICv3: CPU5: found redistributor 10101 region 0:0x000000004d240000
[    0.493127] CPU5: using LPI pending table @0x00000017d5920000
[    0.493265] CPU5: Booted secondary processor [410fd082]
[    0.575722] Detected PIPT I-cache on CPU6
[    0.575728] GICv3: CPU6: found redistributor 10102 region 0:0x000000004d280000
[    0.575752] CPU6: using LPI pending table @0x00000017d5950000
[    0.575889] CPU6: Booted secondary processor [410fd082]
[    0.658346] Detected PIPT I-cache on CPU7
[    0.658353] GICv3: CPU7: found redistributor 10103 region 0:0x000000004d2c0000
[    0.658374] CPU7: using LPI pending table @0x00000017d5980000
[    0.658512] CPU7: Booted secondary processor [410fd082]
[    0.740973] Detected PIPT I-cache on CPU8
[    0.740983] GICv3: CPU8: found redistributor 10200 region 0:0x000000004d300000
[    0.741006] CPU8: using LPI pending table @0x00000017d59b0000
[    0.741147] CPU8: Booted secondary processor [410fd082]
[    0.823595] Detected PIPT I-cache on CPU9
[    0.823602] GICv3: CPU9: found redistributor 10201 region 0:0x000000004d340000
[    0.823624] CPU9: using LPI pending table @0x00000017d59f0000
[    0.823762] CPU9: Booted secondary processor [410fd082]
[    0.906220] Detected PIPT I-cache on CPU10
[    0.906227] GICv3: CPU10: found redistributor 10202 region 0:0x000000004d380000
[    0.906249] CPU10: using LPI pending table @0x00000017d5a20000
[    0.906387] CPU10: Booted secondary processor [410fd082]
[    0.988845] Detected PIPT I-cache on CPU11
[    0.988851] GICv3: CPU11: found redistributor 10203 region 0:0x000000004d3c0000
[    0.988874] CPU11: using LPI pending table @0x00000017d5a70000
[    0.989013] CPU11: Booted secondary processor [410fd082]
[    1.071473] Detected PIPT I-cache on CPU12
[    1.071484] GICv3: CPU12: found redistributor 10300 region 0:0x000000004d400000
[    1.071507] CPU12: using LPI pending table @0x00000017d5aa0000
[    1.071649] CPU12: Booted secondary processor [410fd082]
[    1.154094] Detected PIPT I-cache on CPU13
[    1.154101] GICv3: CPU13: found redistributor 10301 region 0:0x000000004d440000
[    1.154124] CPU13: using LPI pending table @0x00000017d5ad0000
[    1.154262] CPU13: Booted secondary processor [410fd082]
[    1.236720] Detected PIPT I-cache on CPU14
[    1.236727] GICv3: CPU14: found redistributor 10302 region 0:0x000000004d480000
[    1.236749] CPU14: using LPI pending table @0x00000017d5b00000
[    1.236889] CPU14: Booted secondary processor [410fd082]
[    1.319344] Detected PIPT I-cache on CPU15
[    1.319351] GICv3: CPU15: found redistributor 10303 region 0:0x000000004d4c0000
[    1.319375] CPU15: using LPI pending table @0x00000017d5b40000
[    1.319513] CPU15: Booted secondary processor [410fd082]
[    1.401620] Detected PIPT I-cache on CPU16
[    1.401629] GICv3: CPU16: found redistributor 30000 region 1:0x000000006d100000
[    1.401646] CPU16: using LPI pending table @0x00000017d5b70000
[    1.401750] CPU16: Booted secondary processor [410fd082]
[    1.484249] Detected PIPT I-cache on CPU17
[    1.484256] GICv3: CPU17: found redistributor 30001 region 1:0x000000006d140000
[    1.484273] CPU17: using LPI pending table @0x00000017d5b90000
[    1.484379] CPU17: Booted secondary processor [410fd082]
[    1.566878] Detected PIPT I-cache on CPU18
[    1.566886] GICv3: CPU18: found redistributor 30002 region 1:0x000000006d180000
[    1.566903] CPU18: using LPI pending table @0x00000017d5bd0000
[    1.567010] CPU18: Booted secondary processor [410fd082]
[    1.649507] Detected PIPT I-cache on CPU19
[    1.649514] GICv3: CPU19: found redistributor 30003 region 1:0x000000006d1c0000
[    1.649532] CPU19: using LPI pending table @0x00000017d5400000
[    1.649637] CPU19: Booted secondary processor [410fd082]
[    1.732137] Detected PIPT I-cache on CPU20
[    1.732145] GICv3: CPU20: found redistributor 30100 region 1:0x000000006d200000
[    1.732163] CPU20: using LPI pending table @0x00000017d5430000
[    1.732269] CPU20: Booted secondary processor [410fd082]
[    1.814765] Detected PIPT I-cache on CPU21
[    1.814773] GICv3: CPU21: found redistributor 30101 region 1:0x000000006d240000
[    1.814789] CPU21: using LPI pending table @0x00000017d5460000
[    1.814894] CPU21: Booted secondary processor [410fd082]
[    1.897395] Detected PIPT I-cache on CPU22
[    1.897403] GICv3: CPU22: found redistributor 30102 region 1:0x000000006d280000
[    1.897419] CPU22: using LPI pending table @0x00000017d5490000
[    1.897523] CPU22: Booted secondary processor [410fd082]
[    1.980024] Detected PIPT I-cache on CPU23
[    1.980031] GICv3: CPU23: found redistributor 30103 region 1:0x000000006d2c0000
[    1.980048] CPU23: using LPI pending table @0x00000017d54c0000
[    1.980153] CPU23: Booted secondary processor [410fd082]
[    2.062655] Detected PIPT I-cache on CPU24
[    2.062664] GICv3: CPU24: found redistributor 30200 region 1:0x000000006d300000
[    2.062682] CPU24: using LPI pending table @0x00000017d54f0000
[    2.062789] CPU24: Booted secondary processor [410fd082]
[    2.145282] Detected PIPT I-cache on CPU25
[    2.145290] GICv3: CPU25: found redistributor 30201 region 1:0x000000006d340000
[    2.145306] CPU25: using LPI pending table @0x00000017d5530000
[    2.145412] CPU25: Booted secondary processor [410fd082]
[    2.227912] Detected PIPT I-cache on CPU26
[    2.227920] GICv3: CPU26: found redistributor 30202 region 1:0x000000006d380000
[    2.227938] CPU26: using LPI pending table @0x00000017d5560000
[    2.228043] CPU26: Booted secondary processor [410fd082]
[    2.310540] Detected PIPT I-cache on CPU27
[    2.310548] GICv3: CPU27: found redistributor 30203 region 1:0x000000006d3c0000
[    2.310565] CPU27: using LPI pending table @0x00000017d5590000
[    2.310671] CPU27: Booted secondary processor [410fd082]
[    2.393171] Detected PIPT I-cache on CPU28
[    2.393180] GICv3: CPU28: found redistributor 30300 region 1:0x000000006d400000
[    2.393199] CPU28: using LPI pending table @0x00000017d55c0000
[    2.393304] CPU28: Booted secondary processor [410fd082]
[    2.475798] Detected PIPT I-cache on CPU29
[    2.475806] GICv3: CPU29: found redistributor 30301 region 1:0x000000006d440000
[    2.475823] CPU29: using LPI pending table @0x00000017d55f0000
[    2.475928] CPU29: Booted secondary processor [410fd082]
[    2.558427] Detected PIPT I-cache on CPU30
[    2.558435] GICv3: CPU30: found redistributor 30302 region 1:0x000000006d480000
[    2.558453] CPU30: using LPI pending table @0x00000017d5630000
[    2.558557] CPU30: Booted secondary processor [410fd082]
[    2.641057] Detected PIPT I-cache on CPU31
[    2.641065] GICv3: CPU31: found redistributor 30303 region 1:0x000000006d4c0000
[    2.641082] CPU31: using LPI pending table @0x00000017d5660000
[    2.641187] CPU31: Booted secondary processor [410fd082]
[    2.723368] Detected PIPT I-cache on CPU32
[    2.723404] GICv3: CPU32: found redistributor 50000 region 2:0x000004004d100000
[    2.723454] CPU32: using LPI pending table @0x00000017d5690000
[    2.723603] CPU32: Booted secondary processor [410fd082]
[    2.805990] Detected PIPT I-cache on CPU33
[    2.806007] GICv3: CPU33: found redistributor 50001 region 2:0x000004004d140000
[    2.806050] CPU33: using LPI pending table @0x00000017d56c0000
[    2.806154] CPU33: Booted secondary processor [410fd082]
[    2.888629] Detected PIPT I-cache on CPU34
[    2.888646] GICv3: CPU34: found redistributor 50002 region 2:0x000004004d180000
[    2.888688] CPU34: using LPI pending table @0x00000017d56f0000
[    2.888794] CPU34: Booted secondary processor [410fd082]
[    2.971267] Detected PIPT I-cache on CPU35
[    2.971284] GICv3: CPU35: found redistributor 50003 region 2:0x000004004d1c0000
[    2.971325] CPU35: using LPI pending table @0x00000017d5720000
[    2.971430] CPU35: Booted secondary processor [410fd082]
[    3.053909] Detected PIPT I-cache on CPU36
[    3.053926] GICv3: CPU36: found redistributor 50100 region 2:0x000004004d200000
[    3.053968] CPU36: using LPI pending table @0x00000017d5760000
[    3.054075] CPU36: Booted secondary processor [410fd082]
[    3.136549] Detected PIPT I-cache on CPU37
[    3.136565] GICv3: CPU37: found redistributor 50101 region 2:0x000004004d240000
[    3.136608] CPU37: using LPI pending table @0x00000017d5790000
[    3.136712] CPU37: Booted secondary processor [410fd082]
[    3.219189] Detected PIPT I-cache on CPU38
[    3.219205] GICv3: CPU38: found redistributor 50102 region 2:0x000004004d280000
[    3.219248] CPU38: using LPI pending table @0x00000017d57c0000
[    3.219353] CPU38: Booted secondary processor [410fd082]
[    3.301832] Detected PIPT I-cache on CPU39
[    3.301848] GICv3: CPU39: found redistributor 50103 region 2:0x000004004d2c0000
[    3.301889] CPU39: using LPI pending table @0x00000017d57f0000
[    3.301994] CPU39: Booted secondary processor [410fd082]
[    3.384474] Detected PIPT I-cache on CPU40
[    3.384492] GICv3: CPU40: found redistributor 50200 region 2:0x000004004d300000
[    3.384533] CPU40: using LPI pending table @0x00000017d5020000
[    3.384641] CPU40: Booted secondary processor [410fd082]
[    3.467112] Detected PIPT I-cache on CPU41
[    3.467129] GICv3: CPU41: found redistributor 50201 region 2:0x000004004d340000
[    3.467171] CPU41: using LPI pending table @0x00000017d5050000
[    3.467276] CPU41: Booted secondary processor [410fd082]
[    3.549753] Detected PIPT I-cache on CPU42
[    3.549769] GICv3: CPU42: found redistributor 50202 region 2:0x000004004d380000
[    3.549813] CPU42: using LPI pending table @0x00000017d5080000
[    3.549920] CPU42: Booted secondary processor [410fd082]
[    3.632393] Detected PIPT I-cache on CPU43
[    3.632409] GICv3: CPU43: found redistributor 50203 region 2:0x000004004d3c0000
[    3.632454] CPU43: using LPI pending table @0x00000017d50c0000
[    3.632560] CPU43: Booted secondary processor [410fd082]
[    3.715033] Detected PIPT I-cache on CPU44
[    3.715051] GICv3: CPU44: found redistributor 50300 region 2:0x000004004d400000
[    3.715095] CPU44: using LPI pending table @0x00000017d50e0000
[    3.715203] CPU44: Booted secondary processor [410fd082]
[    3.797676] Detected PIPT I-cache on CPU45
[    3.797693] GICv3: CPU45: found redistributor 50301 region 2:0x000004004d440000
[    3.797735] CPU45: using LPI pending table @0x00000017d5120000
[    3.797840] CPU45: Booted secondary processor [410fd082]
[    3.880317] Detected PIPT I-cache on CPU46
[    3.880334] GICv3: CPU46: found redistributor 50302 region 2:0x000004004d480000
[    3.880377] CPU46: using LPI pending table @0x00000017d5150000
[    3.880484] CPU46: Booted secondary processor [410fd082]
[    3.962956] Detected PIPT I-cache on CPU47
[    3.962973] GICv3: CPU47: found redistributor 50303 region 2:0x000004004d4c0000
[    3.963016] CPU47: using LPI pending table @0x00000017d5180000
[    3.963122] CPU47: Booted secondary processor [410fd082]
[    4.045257] Detected PIPT I-cache on CPU48
[    4.045280] GICv3: CPU48: found redistributor 70000 region 3:0x000004006d100000
[    4.045326] CPU48: using LPI pending table @0x00000017d51b0000
[    4.045474] CPU48: Booted secondary processor [410fd082]
[    4.127897] Detected PIPT I-cache on CPU49
[    4.127916] GICv3: CPU49: found redistributor 70001 region 3:0x000004006d140000
[    4.127962] CPU49: using LPI pending table @0x00000017d51e0000
[    4.128107] CPU49: Booted secondary processor [410fd082]
[    4.210539] Detected PIPT I-cache on CPU50
[    4.210557] GICv3: CPU50: found redistributor 70002 region 3:0x000004006d180000
[    4.210603] CPU50: using LPI pending table @0x00000017d5210000
[    4.210749] CPU50: Booted secondary processor [410fd082]
[    4.293181] Detected PIPT I-cache on CPU51
[    4.293200] GICv3: CPU51: found redistributor 70003 region 3:0x000004006d1c0000
[    4.293246] CPU51: using LPI pending table @0x00000017d5250000
[    4.293389] CPU51: Booted secondary processor [410fd082]
[    4.375826] Detected PIPT I-cache on CPU52
[    4.375848] GICv3: CPU52: found redistributor 70100 region 3:0x000004006d200000
[    4.375895] CPU52: using LPI pending table @0x00000017d5280000
[    4.376043] CPU52: Booted secondary processor [410fd082]
[    4.458466] Detected PIPT I-cache on CPU53
[    4.458485] GICv3: CPU53: found redistributor 70101 region 3:0x000004006d240000
[    4.458533] CPU53: using LPI pending table @0x00000017d52b0000
[    4.458677] CPU53: Booted secondary processor [410fd082]
[    4.541110] Detected PIPT I-cache on CPU54
[    4.541129] GICv3: CPU54: found redistributor 70102 region 3:0x000004006d280000
[    4.541176] CPU54: using LPI pending table @0x00000017d52e0000
[    4.541322] CPU54: Booted secondary processor [410fd082]
[    4.623752] Detected PIPT I-cache on CPU55
[    4.623771] GICv3: CPU55: found redistributor 70103 region 3:0x000004006d2c0000
[    4.623817] CPU55: using LPI pending table @0x00000017d5320000
[    4.623961] CPU55: Booted secondary processor [410fd082]
[    4.706397] Detected PIPT I-cache on CPU56
[    4.706419] GICv3: CPU56: found redistributor 70200 region 3:0x000004006d300000
[    4.706468] CPU56: using LPI pending table @0x00000017d5340000
[    4.706617] CPU56: Booted secondary processor [410fd082]
[    4.789037] Detected PIPT I-cache on CPU57
[    4.789056] GICv3: CPU57: found redistributor 70201 region 3:0x000004006d340000
[    4.789104] CPU57: using LPI pending table @0x00000017d5380000
[    4.789249] CPU57: Booted secondary processor [410fd082]
[    4.871681] Detected PIPT I-cache on CPU58
[    4.871700] GICv3: CPU58: found redistributor 70202 region 3:0x000004006d380000
[    4.871746] CPU58: using LPI pending table @0x00000017d53b0000
[    4.871892] CPU58: Booted secondary processor [410fd082]
[    4.954321] Detected PIPT I-cache on CPU59
[    4.954340] GICv3: CPU59: found redistributor 70203 region 3:0x000004006d3c0000
[    4.954390] CPU59: using LPI pending table @0x00000017d53e0000
[    4.954532] CPU59: Booted secondary processor [410fd082]
[    5.036966] Detected PIPT I-cache on CPU60
[    5.036989] GICv3: CPU60: found redistributor 70300 region 3:0x000004006d400000
[    5.037035] CPU60: using LPI pending table @0x00000017d4c10000
[    5.037183] CPU60: Booted secondary processor [410fd082]
[    5.119607] Detected PIPT I-cache on CPU61
[    5.119626] GICv3: CPU61: found redistributor 70301 region 3:0x000004006d440000
[    5.119674] CPU61: using LPI pending table @0x00000017d4c40000
[    5.119818] CPU61: Booted secondary processor [410fd082]
[    5.202251] Detected PIPT I-cache on CPU62
[    5.202270] GICv3: CPU62: found redistributor 70302 region 3:0x000004006d480000
[    5.202319] CPU62: using LPI pending table @0x00000017d4c70000
[    5.202465] CPU62: Booted secondary processor [410fd082]
[    5.284892] Detected PIPT I-cache on CPU63
[    5.284911] GICv3: CPU63: found redistributor 70303 region 3:0x000004006d4c0000
[    5.284958] CPU63: using LPI pending table @0x00000017d4cd0000
[    5.285100] CPU63: Booted secondary processor [410fd082]
[    5.285177] smp: Brought up 1 node, 64 CPUs
[    5.285322] SMP: Total of 64 processors activated.
[    5.285326] CPU features: detected feature: GIC system register CPU interface
[    5.285329] CPU features: detected feature: 32-bit EL0 Support
[    5.285561] CPU: All CPU(s) started at EL2
[    5.286204] devtmpfs: initialized
[    5.287662] SMBIOS 3.0.0 present.
[    5.287668] DMI: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 03/16/2017
[    5.287902] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.288140] futex hash table entries: 16384 (order: 9, 2097152 bytes)
[    5.288719] pinctrl core: initialized pinctrl subsystem
[    5.289150] NET: Registered protocol family 16
[    5.290547] cpuidle: using governor menu
[    5.290742] irq: no irq domain found for /interrupt-controller@a0080000/intc_smmu_pcie !
[    5.290904] arm-smmu-v3 a0040000.smmu_pcie: option hisilicon,broken-prefetch-cmd
[    5.290911] arm-smmu-v3 a0040000.smmu_pcie: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.294485] irq: no irq domain found for /interrupt-controller@c0080000/intc_smmu_dsa !
[    5.294581] arm-smmu-v3 c0040000.smmu_dsa: option hisilicon,broken-prefetch-cmd
[    5.294585] arm-smmu-v3 c0040000.smmu_dsa: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.298023] vdso: 2 pages (1 code @ ffff000008927000, 1 data @ ffff000008e25000)
[    5.298064] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    5.299021] DMA: preallocated 256 KiB pool for atomic allocations
[    5.299230] Serial: AMBA PL011 UART driver
[    5.299700] irq: no irq domain found for /interrupt-controller@60080000/uart_intc !
[    5.299774] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299822] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.301494] irq: no irq domain found for /interrupt-controller@c0080000/intc_dsaf0 !
[    5.301660] irq: no irq domain found for /interrupt-controller@c0080000/intc-roce !
[    5.307230] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    5.308232] ACPI: Interpreter disabled.
[    5.308486] vgaarb: loaded
[    5.308575] SCSI subsystem initialized
[    5.308770] libata version 3.00 loaded.
[    5.308867] usbcore: registered new interface driver usbfs
[    5.308888] usbcore: registered new interface driver hub
[    5.309071] usbcore: registered new device driver usb
[    5.309193] pps_core: LinuxPPS API ver. 1 registered
[    5.309195] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    5.309205] PTP clock support registered
[    5.309296] Registered efivars operations
[    5.309483] Advanced Linux Sound Architecture Driver Initialized.
[    5.310080] clocksource: Switched to clocksource arch_sys_counter
[    5.310222] VFS: Disk quotas dquot_6.6.0
[    5.310250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    5.310353] pnp: PnP ACPI: disabled
[    5.313513] NET: Registered protocol family 2
[    5.314224] TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
[    5.315000] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    5.315419] TCP: Hash tables configured (established 262144 bind 65536)
[    5.315699] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    5.315925] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    5.316362] NET: Registered protocol family 1
[    5.316735] RPC: Registered named UNIX socket transport module.
[    5.316738] RPC: Registered udp transport module.
[    5.316740] RPC: Registered tcp transport module.
[    5.316742] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.316746] PCI: CLS 0 bytes, default 128
[    5.319057] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    5.319313] kvm [1]: 8-bit VMID
[    5.319316] kvm [1]: IDMAP page: 90e000
[    5.319319] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[    5.320299] kvm [1]: Hyp mode initialized successfully
[    5.320413] kvm [1]: vgic-v2@fe020000
[    5.320504] kvm [1]: GIC system register CPU interface enabled
[    5.321247] kvm [1]: vgic interrupt IRQ1
[    5.321256] kvm [1]: virtual timer IRQ4
[    5.324764] audit: initializing netlink subsys (disabled)
[    5.324833] audit: type=2000 audit(3.644:1): state=initialized audit_enabled=0 res=1
[    5.325243] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    5.328073] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    5.328424] NFS: Registering the id_resolver key type
[    5.328438] Key type id_resolver registered
[    5.328440] Key type id_legacy registered
[    5.328447] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    5.328511] 9p: Installing v9fs 9p2000 file system support
[    5.329843] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    5.329848] io scheduler noop registered
[    5.329917] io scheduler cfq registered (default)
[    5.329920] io scheduler mq-deadline registered
[    5.332537] OF: PCI: host bridge /soc/pcie@a00a0000 ranges:
[    5.332546] OF: PCI:   MEM 0xa8800000..0xaffeffff -> 0xa8800000
[    5.332551] OF: PCI:    IO 0xafff0000..0xafffffff -> 0x00000000
[    5.332590] hisi-pcie-almost-ecam a8000000.pcie: ECAM at [mem 0xa8000000-0xa87fffff] for [bus 80-87]
[    5.332642] hisi-pcie-almost-ecam a8000000.pcie: PCI host bridge to bus 0000:80
[    5.332646] pci_bus 0000:80: root bus resource [bus 80-87]
[    5.332649] pci_bus 0000:80: root bus resource [mem 0xa8800000-0xaffeffff]
[    5.332652] pci_bus 0000:80: root bus resource [io  0x0000-0xffff]
[    5.332668] pci 0000:80:00.0: [19e5:1610] type 01 class 0x060400
[    5.332681] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332688] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332693] pci 0000:80:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[    5.332696] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332702] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332706] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332712] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332730] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x52 may corrupt adjacent RW1C bits
[    5.332754] pci 0000:80:00.0: supports D1 D2
[    5.332757] pci 0000:80:00.0: PME# supported from D0 D1 D3hot
[    5.332760] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x44 may corrupt adjacent RW1C bits
[    5.332844] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332849] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332886] pci_bus 0000:81: busn_res: can not insert [bus 81-ff] under [bus 80-87] (conflicts with (null) [bus 80-87])
[    5.332903] pci 0000:81:00.0: [8086:10fb] type 00 class 0x020000
[    5.332919] pci 0000:81:00.0: reg 0x10: [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.332925] pci 0000:81:00.0: reg 0x18: [io  0x0000-0x001f]
[    5.332940] pci 0000:81:00.0: reg 0x20: [mem 0xa9804000-0xa9807fff 64bit pref]
[    5.332946] pci 0000:81:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.333005] pci 0000:81:00.0: PME# supported from D0 D3hot
[    5.333040] pci 0000:81:00.0: reg 0x184: [mem 0xa9704000-0xa9707fff 64bit pref]
[    5.333043] pci 0000:81:00.0: VF(n) BAR0 space: [mem 0xa9704000-0xa9803fff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333058] pci 0000:81:00.0: reg 0x190: [mem 0xa9604000-0xa9607fff 64bit pref]
[    5.333061] pci 0000:81:00.0: VF(n) BAR3 space: [mem 0xa9604000-0xa9703fff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333341] pci 0000:81:00.1: [8086:10fb] type 00 class 0x020000
[    5.333357] pci 0000:81:00.1: reg 0x10: [mem 0xa8c00000-0xa8ffffff 64bit pref]
[    5.333364] pci 0000:81:00.1: reg 0x18: [io  0x0000-0x001f]
[    5.333378] pci 0000:81:00.1: reg 0x20: [mem 0xa9600000-0xa9603fff 64bit pref]
[    5.333385] pci 0000:81:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.333443] pci 0000:81:00.1: PME# supported from D0 D3hot
[    5.333471] pci 0000:81:00.1: reg 0x184: [mem 0xa9500000-0xa9503fff 64bit pref]
[    5.333474] pci 0000:81:00.1: VF(n) BAR0 space: [mem 0xa9500000-0xa95fffff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333489] pci 0000:81:00.1: reg 0x190: [mem 0xa9400000-0xa9403fff 64bit pref]
[    5.333492] pci 0000:81:00.1: VF(n) BAR3 space: [mem 0xa9400000-0xa94fffff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333750] pci_bus 0000:81: busn_res: [bus 81-ff] end is updated to 81
[    5.333771] pci 0000:80:00.0: BAR 14: assigned [mem 0xa8800000-0xa8ffffff]
[    5.333775] pci 0000:80:00.0: BAR 15: assigned [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333777] pci 0000:80:00.0: BAR 0: assigned [mem 0xa9e00000-0xa9e0ffff]
[    5.333782] pci 0000:80:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.333787] pci 0000:81:00.0: BAR 0: assigned [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.333796] pci 0000:81:00.0: BAR 6: assigned [mem 0xa8800000-0xa8bfffff pref]
[    5.333799] pci 0000:81:00.1: BAR 0: assigned [mem 0xa9400000-0xa97fffff 64bit pref]
[    5.333808] pci 0000:81:00.1: BAR 6: assigned [mem 0xa8c00000-0xa8ffffff pref]
[    5.333811] pci 0000:81:00.0: BAR 4: assigned [mem 0xa9800000-0xa9803fff 64bit pref]
[    5.333820] pci 0000:81:00.0: BAR 7: assigned [mem 0xa9804000-0xa9903fff 64bit pref]
[    5.333825] pci 0000:81:00.0: BAR 10: assigned [mem 0xa9904000-0xa9a03fff 64bit pref]
[    5.333830] pci 0000:81:00.1: BAR 4: assigned [mem 0xa9a04000-0xa9a07fff 64bit pref]
[    5.333838] pci 0000:81:00.1: BAR 7: assigned [mem 0xa9a08000-0xa9b07fff 64bit pref]
[    5.333844] pci 0000:81:00.1: BAR 10: assigned [mem 0xa9b08000-0xa9c07fff 64bit pref]
[    5.333848] pci 0000:81:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    5.333853] pci 0000:81:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    5.333860] pci 0000:80:00.0: PCI bridge to [bus 81]
[    5.333864] pci 0000:80:00.0:   bridge window [io  0x1000-0x1fff]
[    5.333869] pci 0000:80:00.0:   bridge window [mem 0xa8800000-0xa8ffffff]
[    5.333872] pci 0000:80:00.0:   bridge window [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.334042] iommu: Adding device 0000:80:00.0 to group 0
[    5.334053] pcieport 0000:80:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.335031] pcieport 0000:80:00.0: Signaling PME with IRQ 7
[    5.335077] pcieport 0000:80:00.0: AER enabled with IRQ 7
[    5.335200] ipmi message handler version 39.2
[    5.335206] ipmi device interface
[    5.335248] IPMI System Interface driver.
[    5.335276] ipmi_si: probing via SMBIOS
[    5.335278] ipmi_si: SMBIOS: io 0xe4 regsize 1 spacing 1 irq 0
[    5.335282] ipmi_si: Adding SMBIOS-specified bt state machine
[    5.335287] ipmi_si: Trying SMBIOS-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
[    5.335303] ipmi_si ipmi_si.0: Interface detection failed
[    5.336401] xenfs: not registering filesystem on non-xen platform
[    5.337625] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    5.338157] SuperH (H)SCI(F) driver initialized
[    5.338254] msm_serial: driver initialized
[    5.355682] loop: module loaded
[    5.356033] hisi_sas: driver version v1.6
[    5.356792] libphy: Fixed MDIO Bus: probed
[    5.357045] tun: Universal TUN/TAP device driver, 1.6
[    5.357047] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    5.357460] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    5.357463] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    5.357524] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    5.357526] igb: Copyright (c) 2007-2014 Intel Corporation.
[    5.357586] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    5.357588] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    5.357638] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.0.0-k
[    5.357640] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    5.357875] iommu: Adding device 0000:81:00.0 to group 1
[    5.357885] ixgbe 0000:81:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.358742] ixgbe 0000:81:00.0: enabling device (0000 -> 0002)
[    5.519236] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    5.519365] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available
[    5.519368] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    5.519452] ixgbe 0000:81:00.0: MAC: 2, PHY: 15, SFP+: 11, PBA No: FFFFFF-0FF
[    5.519455] ixgbe 0000:81:00.0: 68:a8:28:17:97:07
[    5.524366] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    5.524491] iommu: Adding device 0000:81:00.1 to group 2
[    5.524499] ixgbe 0000:81:00.1: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.525327] ixgbe 0000:81:00.1: enabling device (0000 -> 0002)
[    6.703188] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    6.703316] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available
[    6.703319] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    6.703402] ixgbe 0000:81:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    6.703405] ixgbe 0000:81:00.1: 68:a8:28:17:97:08
[    6.708048] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
[    6.708094] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[    6.708097] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    6.708175] sky2: driver version 1.30
[    6.708616] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.708619] ehci-pci: EHCI PCI platform driver
[    6.708635] ehci-platform: EHCI generic platform driver
[    6.708802] iommu: Adding device a7020000.ehci to group 3
[    6.708810] ehci-platform a7020000.ehci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.709700] ehci-platform a7020000.ehci: EHCI Host Controller
[    6.709729] ehci-platform a7020000.ehci: new USB bus registered, assigned bus number 1
[    6.709820] ehci-platform a7020000.ehci: irq 136, io mem 0xa7020000
[    6.722071] ehci-platform a7020000.ehci: USB 2.0 started, EHCI 1.00
[    6.722310] hub 1-0:1.0: USB hub found
[    6.722321] hub 1-0:1.0: 2 ports detected
[    6.722462] ehci-exynos: EHCI EXYNOS driver
[    6.722498] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    6.722523] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.722533] ohci-pci: OHCI PCI platform driver
[    6.722555] ohci-platform: OHCI generic platform driver
[    6.722653] iommu: Adding device a7030000.ohci to group 4
[    6.722661] ohci-platform a7030000.ohci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.723537] ohci-platform a7030000.ohci: Generic Platform OHCI controller
[    6.723546] ohci-platform a7030000.ohci: new USB bus registered, assigned bus number 2
[    6.723611] ohci-platform a7030000.ohci: irq 137, io mem 0xa7030000
[    6.786404] hub 2-0:1.0: USB hub found
[    6.786418] hub 2-0:1.0: 2 ports detected
[    6.786582] ohci-exynos: OHCI EXYNOS driver
[    6.786804] usbcore: registered new interface driver usb-storage
[    6.913217] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[    6.913377] i2c /dev entries driver
[    6.913990] sdhci: Secure Digital Host Controller Interface driver
[    6.913992] sdhci: Copyright(c) Pierre Ossman
[    6.914073] Synopsys Designware Multimedia Card Interface Driver
[    6.914237] sdhci-pltfm: SDHCI platform and OF driver helper
[    6.915259] ledtrig-cpu: registered to indicate activity on CPUs
[    6.915544] usbcore: registered new interface driver usbhid
[    6.915547] usbhid: USB HID core driver
[    6.916117] NET: Registered protocol family 17
[    6.916146] 9pnet: Installing 9P2000 support
[    6.916170] Key type dns_resolver registered
[    6.916411] registered taskstats version 1
[    6.916613] 602b0000.uart: ttyAMA0 at MMIO 0x602b0000 (irq = 138, base_baud = 0) is a SBSA
[    7.058068] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    7.058209] ehci-platform a7020000.ehci: fatal error
[    7.058212] ehci-platform a7020000.ehci: HC died; cleaning up
[    7.442068] usb 2-1: new full-speed USB device number 2 using ohci-platform
[    7.653099] usb 2-1: not running at top speed; connect to a high speed hub
[    7.659126] hub 2-1:1.0: USB hub found
[    7.661100] hub 2-1:1.0: 4 ports detected
[    7.850064] usb 2-2: new full-speed USB device number 3 using ohci-platform
[    8.061098] usb 2-2: not running at top speed; connect to a high speed hub
[    8.067126] hub 2-2:1.0: USB hub found
[    8.069099] hub 2-2:1.0: 4 ports detected
[    8.374063] usb 2-2.1: new full-speed USB device number 4 using ohci-platform
[    8.509436] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7030000.ohci/usb2/2-2/2-2.1/2-2.1:1.0/0003:12D1:0003.0001/input/input0
[    8.566249] hid-generic 0003:12D1:0003.0001: input: USB HID v1.10 Keyboard [Keyboard/Mouse KVM 1.1.0] on usb-a7030000.ohci-2.1/input0
[    8.569382] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7030000.ohci/usb2/2-2/2-2.1/2-2.1:1.1/0003:12D1:0003.0002/input/input1
[    8.569428] hid-generic 0003:12D1:0003.0002: input: USB HID v1.10 Mouse [Keyboard/Mouse KVM 1.1.0] on usb-a7030000.ohci-2.1/input1
[   10.914206] console [ttyAMA0] enabled
[   10.980942] rtc-efi rtc-efi: setting system clock to 2017-03-27 11:34:34 UTC (1490614474)
[   11.271062] ixgbe 0000:81:00.0: registered PHC device on eth0
[   11.447464] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[   11.583544] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[   11.677525] ixgbe 0000:81:00.1: registered PHC device on eth1
[   11.810063] Sending DHCP requests .....
[   37.402083] random: fast init done
[   55.946395] random: crng init done
[   58.874060] ., OK
[   59.884016] IP-Config: Got DHCP answer from 172.18.45.1, my address is 172.18.45.129
[   59.891754] IP-Config: Complete:
[   59.894976]      device=eth0, hwaddr=68:a8:28:17:97:07, ipaddr=172.18.45.129, mask=255.255.255.0, gw=172.18.45.1
[   59.905147]      host=172.18.45.129, domain=, nis-domain=(none)
[   59.911059]      bootserver=0.0.0.0, rootserver=172.18.45.166, rootpath=     nameserver0=8.8.8.8, nameserver1=4.4.4.4
[   59.922614] ixgbe 0000:81:00.1: removed PHC on eth1
[   60.337680] ALSA device list:
[   60.340652]   No soundcards found.
[   60.349483] VFS: Mounted root (nfs filesystem) on device 0:14.
[   60.355689] devtmpfs: mounted
[   60.359404] Freeing unused kernel memory: 1088K
[   60.550258] init: plymouth-upstart-bridge main process (1942) terminated with status 1
[   60.558209] init: plymouth-upstart-bridge main process ended, respawning
[   60.721227] init: ureadahead main process (1945) terminated with status 5
[   67.644936] systemd-udevd[2171]: starting version 208
[   67.943594] ixgbe 0000:81:00.1 eth26: renamed from eth1
[   68.058186] systemd-udevd[2574]: renamed network interface eth1 to eth26
[   68.121180] Bluetooth: Core ver 2.22
[   68.121281] NET: Registered protocol family 31
[   68.121282] Bluetooth: HCI device and connection manager initialized
[   68.121293] Bluetooth: HCI socket layer initialized
[   68.121298] Bluetooth: L2CAP socket layer initialized
[   68.121312] Bluetooth: SCO socket layer initialized
[   68.319028] init: failsafe main process (2687) killed by TERM signal
[   68.753718] NET: Registered protocol family 10
[   68.756503] Segment Routing with IPv6
[   69.971495] ip_tables: (C) 2000-2006 Netfilter Core Team
[   70.078809] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   70.197725] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   71.194210] init: tty1 main process (3319) killed by TERM signal
[   71.415444] init: plymouth-upstart-bridge main process ended, respawning
[  144.741629] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  144.741799] ixgbe 0000:81:00.0: removed PHC on eth0
[  144.977338] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  144.997161] ixgbe 0000:81:00.0: registered PHC device on eth0
[  145.169155] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  145.211717] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  145.212351] iommu: Adding device 0000:81:10.0 to group 5
[  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
[  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
[  145.225944] ixgbevf 0000:81:10.0: MAC: 1
[  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  154.947742] nfs: server 172.18.45.166 not responding, still trying
[  191.025780] nfs: server 172.18.45.166 not responding, still trying
[  191.026122] nfs: server 172.18.45.166 OK
[  191.026317] nfs: server 172.18.45.166 OK
[  263.706402] VFIO - User Level meta-driver version: 0.3
[  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  269.757617] specified DMA range outside IOMMU capability
[  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: Monday, March 27, 2017 3:53 PM
> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon-5wv7dgnIgG8@public.gmane.org;
> joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> foundation.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-
> msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
> okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
> 
> 
> > -----Original Message-----
> > From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
> > Sent: Friday, March 24, 2017 6:39 PM
> > To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
> > will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org;
> > iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org;
> > linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> > bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> > acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
> > okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> > Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> >
> > On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> > > Hi Sricharan,
> > >
> > >> -----Original Message-----
> > >> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
> [...]
> > >> Looks like this triggers the start of the bug.
> > >> So the below check in iommu_dma_init_domain fails,
> > >>
> > >>          if (domain->geometry.force_aperture) {
> > >>                  if (base > domain->geometry.aperture_end ||
> > >>                      base + size <= domain->geometry.aperture_start) {
> > >>
> > >> and the rest goes out of sync after that. Can you print out the base,
> > >> aperture_start and end values to see why the check fails ?
> > >
> > > dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
> > domain->geometry.aperture_start, domain->geometry.aperture_end,
> > *dev->dma_mask, dev->coherent_dma_mask);
> > >
> > > [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> > 0xffffffff 0xffffffff
> > > .....
> > > [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
> > 0xffffffffffffffff 0xffffffffffffffff
> > >
> > > Yes, size seems to be the problem here. When the VF  device gets
> attached
> > to vfio-pci,
> > > somehow the dev->coherent_dma_mask is set to 64 bits and size
> become
> > zero.
> >
> > AFAICS, this is either down to patch 3 (which should apply on its own
> > easily enough for testing), or patch 6, implying that somehow the
> > vfio-pci device gets its DMA mask widened to 64 bits somewhere between
> > very soon after after creation (where we originally called
> > of_dma_configure()) and immediately before probe (where we now call
> it).
> >
> > Either way I guess this is yet more motivation to write that "change the
> > arch_setup_dma_ops() interface to take a mask instead of a size" patch...
> 
> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
> find the
> log below (with  dev_info debug added to iommu_dma_init_domain ).
> ...
> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff   ---->dev_info()
> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
> [  188.981298] nfs: server 172.18.45.166 OK
> [  188.981593] nfs: server 172.18.45.166 OK
> [  221.755626] VFIO - User Level meta-driver version: 0.3
> ...
> 
> Applied up to patch 6, and the issue appeared,
> 
> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff
> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> [  191.026122] nfs: server 172.18.45.166 OK
> [  191.026317] nfs: server 172.18.45.166 OK
> [  263.706402] VFIO - User Level meta-driver version: 0.3
> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> 0xffffffffffffffff
> [  269.757617] specified DMA range outside IOMMU capability
> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> platform DMA ops
> 
>  From the logs its clear that  when ixgbevf driver originally probes and adds
> the device
>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.

Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

[  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
[  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
[  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  128.358416] iommu: Adding device 0000:81:10.0 to group 5
[  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
[  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
[  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
[  128.372176] ixgbevf 0000:81:10.0: MAC: 1
[  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  138.089869] nfs: server 172.18.45.166 not responding, still trying
[  174.697868] nfs: server 172.18.45.166 not responding, still trying
[  174.698359] nfs: server 172.18.45.166 OK
[  174.698582] nfs: server 172.18.45.166 OK
[  465.942259] VFIO - User Level meta-driver version: 0.3
 [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  472.754075] specified DMA range outside IOMMU capability
[  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996


 

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: Monday, March 27, 2017 3:53 PM
> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon@arm.com;
> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
> msm@vger.kernel.org; m.szyprowski@samsung.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
> okaya@codeaurora.org
> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
> 
> 
> > -----Original Message-----
> > From: Robin Murphy [mailto:robin.murphy@arm.com]
> > Sent: Friday, March 24, 2017 6:39 PM
> > To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
> > will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com;
> > iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
> > linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com;
> > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
> > okaya@codeaurora.org
> > Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> >
> > On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> > > Hi Sricharan,
> > >
> > >> -----Original Message-----
> > >> From: Sricharan R [mailto:sricharan@codeaurora.org]
> [...]
> > >> Looks like this triggers the start of the bug.
> > >> So the below check in iommu_dma_init_domain fails,
> > >>
> > >>          if (domain->geometry.force_aperture) {
> > >>                  if (base > domain->geometry.aperture_end ||
> > >>                      base + size <= domain->geometry.aperture_start) {
> > >>
> > >> and the rest goes out of sync after that. Can you print out the base,
> > >> aperture_start and end values to see why the check fails ?
> > >
> > > dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
> > domain->geometry.aperture_start, domain->geometry.aperture_end,
> > *dev->dma_mask, dev->coherent_dma_mask);
> > >
> > > [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> > 0xffffffff 0xffffffff
> > > .....
> > > [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
> > 0xffffffffffffffff 0xffffffffffffffff
> > >
> > > Yes, size seems to be the problem here. When the VF  device gets
> attached
> > to vfio-pci,
> > > somehow the dev->coherent_dma_mask is set to 64 bits and size
> become
> > zero.
> >
> > AFAICS, this is either down to patch 3 (which should apply on its own
> > easily enough for testing), or patch 6, implying that somehow the
> > vfio-pci device gets its DMA mask widened to 64 bits somewhere between
> > very soon after after creation (where we originally called
> > of_dma_configure()) and immediately before probe (where we now call
> it).
> >
> > Either way I guess this is yet more motivation to write that "change the
> > arch_setup_dma_ops() interface to take a mask instead of a size" patch...
> 
> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
> find the
> log below (with  dev_info debug added to iommu_dma_init_domain ).
> ...
> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff   ---->dev_info()
> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
> [  188.981298] nfs: server 172.18.45.166 OK
> [  188.981593] nfs: server 172.18.45.166 OK
> [  221.755626] VFIO - User Level meta-driver version: 0.3
> ...
> 
> Applied up to patch 6, and the issue appeared,
> 
> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff
> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> [  191.026122] nfs: server 172.18.45.166 OK
> [  191.026317] nfs: server 172.18.45.166 OK
> [  263.706402] VFIO - User Level meta-driver version: 0.3
> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> 0xffffffffffffffff
> [  269.757617] specified DMA range outside IOMMU capability
> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> platform DMA ops
> 
>  From the logs its clear that  when ixgbevf driver originally probes and adds
> the device
>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.

Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

[  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
[  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
[  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  128.358416] iommu: Adding device 0000:81:10.0 to group 5
[  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
[  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
[  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
[  128.372176] ixgbevf 0000:81:10.0: MAC: 1
[  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  138.089869] nfs: server 172.18.45.166 not responding, still trying
[  174.697868] nfs: server 172.18.45.166 not responding, still trying
[  174.698359] nfs: server 172.18.45.166 OK
[  174.698582] nfs: server 172.18.45.166 OK
[  465.942259] VFIO - User Level meta-driver version: 0.3
 [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  472.754075] specified DMA range outside IOMMU capability
[  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996


 

[PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: Monday, March 27, 2017 3:53 PM
> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon at arm.com;
> joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-
> foundation.org; linux-arm-kernel at lists.infradead.org; linux-arm-
> msm at vger.kernel.org; m.szyprowski at samsung.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
> okaya at codeaurora.org
> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
> 
> 
> > -----Original Message-----
> > From: Robin Murphy [mailto:robin.murphy at arm.com]
> > Sent: Friday, March 24, 2017 6:39 PM
> > To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
> > will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com;
> > iommu at lists.linux-foundation.org; linux-arm-kernel at lists.infradead.org;
> > linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com;
> > bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> > acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
> > okaya at codeaurora.org
> > Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> >
> > On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
> > > Hi Sricharan,
> > >
> > >> -----Original Message-----
> > >> From: Sricharan R [mailto:sricharan at codeaurora.org]
> [...]
> > >> Looks like this triggers the start of the bug.
> > >> So the below check in iommu_dma_init_domain fails,
> > >>
> > >>          if (domain->geometry.force_aperture) {
> > >>                  if (base > domain->geometry.aperture_end ||
> > >>                      base + size <= domain->geometry.aperture_start) {
> > >>
> > >> and the rest goes out of sync after that. Can you print out the base,
> > >> aperture_start and end values to see why the check fails ?
> > >
> > > dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
> > domain->geometry.aperture_start, domain->geometry.aperture_end,
> > *dev->dma_mask, dev->coherent_dma_mask);
> > >
> > > [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> > 0xffffffff 0xffffffff
> > > .....
> > > [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
> > 0xffffffffffffffff 0xffffffffffffffff
> > >
> > > Yes, size seems to be the problem here. When the VF  device gets
> attached
> > to vfio-pci,
> > > somehow the dev->coherent_dma_mask is set to 64 bits and size
> become
> > zero.
> >
> > AFAICS, this is either down to patch 3 (which should apply on its own
> > easily enough for testing), or patch 6, implying that somehow the
> > vfio-pci device gets its DMA mask widened to 64 bits somewhere between
> > very soon after after creation (where we originally called
> > of_dma_configure()) and immediately before probe (where we now call
> it).
> >
> > Either way I guess this is yet more motivation to write that "change the
> > arch_setup_dma_ops() interface to take a mask instead of a size" patch...
> 
> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
> find the
> log below (with  dev_info debug added to iommu_dma_init_domain ).
> ...
> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff   ---->dev_info()
> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
> [  188.981298] nfs: server 172.18.45.166 OK
> [  188.981593] nfs: server 172.18.45.166 OK
> [  221.755626] VFIO - User Level meta-driver version: 0.3
> ...
> 
> Applied up to patch 6, and the issue appeared,
> 
> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> 0xffffffff 0xffffffff
> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> may have to assign one manually
> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> administrator.
> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> None
> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> [  191.026122] nfs: server 172.18.45.166 OK
> [  191.026317] nfs: server 172.18.45.166 OK
> [  263.706402] VFIO - User Level meta-driver version: 0.3
> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> 0xffffffffffffffff
> [  269.757617] specified DMA range outside IOMMU capability
> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> platform DMA ops
> 
>  From the logs its clear that  when ixgbevf driver originally probes and adds
> the device
>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.

Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

[  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
[  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
[  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  128.358416] iommu: Adding device 0000:81:10.0 to group 5
[  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
[  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
[  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
[  128.372176] ixgbevf 0000:81:10.0: MAC: 1
[  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  138.089869] nfs: server 172.18.45.166 not responding, still trying
[  174.697868] nfs: server 172.18.45.166 not responding, still trying
[  174.698359] nfs: server 172.18.45.166 OK
[  174.698582] nfs: server 172.18.45.166 OK
[  465.942259] VFIO - User Level meta-driver version: 0.3
 [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  472.754075] specified DMA range outside IOMMU capability
[  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996


 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 27/03/17 16:58, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: Shameerali Kolothum Thodi
>> Sent: Monday, March 27, 2017 3:53 PM
>> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon@arm.com;
>> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
>> msm@vger.kernel.org; m.szyprowski@samsung.com;
>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>> okaya@codeaurora.org
>> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
>>
>>
>>> -----Original Message-----
>>> From: Robin Murphy [mailto:robin.murphy@arm.com]
>>> Sent: Friday, March 24, 2017 6:39 PM
>>> To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
>>> will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com;
>>> iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
>>> linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com;
>>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>>> okaya@codeaurora.org
>>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>>
>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>> Hi Sricharan,
>>>>
>>>>> -----Original Message-----
>>>>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>> [...]
>>>>> Looks like this triggers the start of the bug.
>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>
>>>>>          if (domain->geometry.force_aperture) {
>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>
>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>> aperture_start and end values to see why the check fails ?
>>>>
>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>
>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>>> .....
>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>
>>>> Yes, size seems to be the problem here. When the VF  device gets
>> attached
>>> to vfio-pci,
>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>> become
>>> zero.
>>>
>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>> easily enough for testing), or patch 6, implying that somehow the
>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>> very soon after after creation (where we originally called
>>> of_dma_configure()) and immediately before probe (where we now call
>> it).
>>>
>>> Either way I guess this is yet more motivation to write that "change the
>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>
>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>> find the
>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>> ...
>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff   ---->dev_info()
>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>> [  188.981298] nfs: server 172.18.45.166 OK
>> [  188.981593] nfs: server 172.18.45.166 OK
>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>> ...
>>
>> Applied up to patch 6, and the issue appeared,
>>
>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff
>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>> [  191.026122] nfs: server 172.18.45.166 OK
>> [  191.026317] nfs: server 172.18.45.166 OK
>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>> 0xffffffffffffffff
>> [  269.757617] specified DMA range outside IOMMU capability
>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>> platform DMA ops
>>
>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>> the device
>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> 
> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

Aha, but of course it's still the same struct device getting bound to
VFIO later, so whatever mask the first driver set is still in there when
we go through of_dma_configure() the second time (and the fact that we
go through more than once being the new behaviour). So yes, this is a
legitimate problem and we really do need to be robust against size
overflow. I reckon the below tweak of your fix is probably the way to
go; cleaning up the arch_setup_dma_ops() interface can happen later.

Robin.

-----8<-----
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 9933077df7b7..77d080bde52d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);


----->8-----

> [  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
> [  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
> [  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
> [  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
> [  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
> [  128.358416] iommu: Adding device 0000:81:10.0 to group 5
> [  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> [  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
> [  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
> [  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
> [  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
> [  128.372176] ixgbevf 0000:81:10.0: MAC: 1
> [  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
> [  138.089869] nfs: server 172.18.45.166 not responding, still trying
> [  174.697868] nfs: server 172.18.45.166 not responding, still trying
> [  174.698359] nfs: server 172.18.45.166 OK
> [  174.698582] nfs: server 172.18.45.166 OK
> [  465.942259] VFIO - User Level meta-driver version: 0.3
>  [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> [  472.754075] specified DMA range outside IOMMU capability
> [  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops
> 
> 1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996
> 
> 
>  
> 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 27/03/17 16:58, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: Shameerali Kolothum Thodi
>> Sent: Monday, March 27, 2017 3:53 PM
>> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon@arm.com;
>> joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>> foundation.org; linux-arm-kernel@lists.infradead.org; linux-arm-
>> msm@vger.kernel.org; m.szyprowski@samsung.com;
>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>> okaya@codeaurora.org
>> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
>>
>>
>>> -----Original Message-----
>>> From: Robin Murphy [mailto:robin.murphy@arm.com]
>>> Sent: Friday, March 24, 2017 6:39 PM
>>> To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
>>> will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com;
>>> iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
>>> linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com;
>>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>>> okaya@codeaurora.org
>>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>>
>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>> Hi Sricharan,
>>>>
>>>>> -----Original Message-----
>>>>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>> [...]
>>>>> Looks like this triggers the start of the bug.
>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>
>>>>>          if (domain->geometry.force_aperture) {
>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>
>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>> aperture_start and end values to see why the check fails ?
>>>>
>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>
>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>>> .....
>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>
>>>> Yes, size seems to be the problem here. When the VF  device gets
>> attached
>>> to vfio-pci,
>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>> become
>>> zero.
>>>
>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>> easily enough for testing), or patch 6, implying that somehow the
>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>> very soon after after creation (where we originally called
>>> of_dma_configure()) and immediately before probe (where we now call
>> it).
>>>
>>> Either way I guess this is yet more motivation to write that "change the
>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>
>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>> find the
>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>> ...
>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff   ---->dev_info()
>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>> [  188.981298] nfs: server 172.18.45.166 OK
>> [  188.981593] nfs: server 172.18.45.166 OK
>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>> ...
>>
>> Applied up to patch 6, and the issue appeared,
>>
>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff
>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>> [  191.026122] nfs: server 172.18.45.166 OK
>> [  191.026317] nfs: server 172.18.45.166 OK
>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>> 0xffffffffffffffff
>> [  269.757617] specified DMA range outside IOMMU capability
>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>> platform DMA ops
>>
>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>> the device
>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> 
> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

Aha, but of course it's still the same struct device getting bound to
VFIO later, so whatever mask the first driver set is still in there when
we go through of_dma_configure() the second time (and the fact that we
go through more than once being the new behaviour). So yes, this is a
legitimate problem and we really do need to be robust against size
overflow. I reckon the below tweak of your fix is probably the way to
go; cleaning up the arch_setup_dma_ops() interface can happen later.

Robin.

-----8<-----
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 9933077df7b7..77d080bde52d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);


----->8-----

> [  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
> [  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
> [  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
> [  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
> [  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
> [  128.358416] iommu: Adding device 0000:81:10.0 to group 5
> [  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> [  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
> [  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
> [  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
> [  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
> [  128.372176] ixgbevf 0000:81:10.0: MAC: 1
> [  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
> [  138.089869] nfs: server 172.18.45.166 not responding, still trying
> [  174.697868] nfs: server 172.18.45.166 not responding, still trying
> [  174.698359] nfs: server 172.18.45.166 OK
> [  174.698582] nfs: server 172.18.45.166 OK
> [  465.942259] VFIO - User Level meta-driver version: 0.3
>  [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> [  472.754075] specified DMA range outside IOMMU capability
> [  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops
> 
> 1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996
> 
> 
>  
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Robin Murphy]

On 27/03/17 16:58, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: Shameerali Kolothum Thodi
>> Sent: Monday, March 27, 2017 3:53 PM
>> To: 'Robin Murphy'; Sricharan R; Wangzhou (B); will.deacon at arm.com;
>> joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-
>> foundation.org; linux-arm-kernel at lists.infradead.org; linux-arm-
>> msm at vger.kernel.org; m.szyprowski at samsung.com;
>> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
>> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
>> okaya at codeaurora.org
>> Subject: RE: [PATCH V9 00/11] IOMMU probe deferral support
>>
>>
>>> -----Original Message-----
>>> From: Robin Murphy [mailto:robin.murphy at arm.com]
>>> Sent: Friday, March 24, 2017 6:39 PM
>>> To: Shameerali Kolothum Thodi; Sricharan R; Wangzhou (B);
>>> will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com;
>>> iommu at lists.linux-foundation.org; linux-arm-kernel at lists.infradead.org;
>>> linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com;
>>> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
>>> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
>>> okaya at codeaurora.org
>>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>>
>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>> Hi Sricharan,
>>>>
>>>>> -----Original Message-----
>>>>> From: Sricharan R [mailto:sricharan at codeaurora.org]
>> [...]
>>>>> Looks like this triggers the start of the bug.
>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>
>>>>>          if (domain->geometry.force_aperture) {
>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>
>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>> aperture_start and end values to see why the check fails ?
>>>>
>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>
>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>>> .....
>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>
>>>> Yes, size seems to be the problem here. When the VF  device gets
>> attached
>>> to vfio-pci,
>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>> become
>>> zero.
>>>
>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>> easily enough for testing), or patch 6, implying that somehow the
>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>> very soon after after creation (where we originally called
>>> of_dma_configure()) and immediately before probe (where we now call
>> it).
>>>
>>> Either way I guess this is yet more motivation to write that "change the
>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>
>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>> find the
>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>> ...
>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff   ---->dev_info()
>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>> [  188.981298] nfs: server 172.18.45.166 OK
>> [  188.981593] nfs: server 172.18.45.166 OK
>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>> ...
>>
>> Applied up to patch 6, and the issue appeared,
>>
>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>> 0xffffffff 0xffffffff
>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>> may have to assign one manually
>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>> administrator.
>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>> None
>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>> [  191.026122] nfs: server 172.18.45.166 OK
>> [  191.026317] nfs: server 172.18.45.166 OK
>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>> 0xffffffffffffffff
>> [  269.757617] specified DMA range outside IOMMU capability
>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>> platform DMA ops
>>
>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>> the device
>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> 
> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

Aha, but of course it's still the same struct device getting bound to
VFIO later, so whatever mask the first driver set is still in there when
we go through of_dma_configure() the second time (and the fact that we
go through more than once being the new behaviour). So yes, this is a
legitimate problem and we really do need to be robust against size
overflow. I reckon the below tweak of your fix is probably the way to
go; cleaning up the arch_setup_dma_ops() interface can happen later.

Robin.

-----8<-----
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 9933077df7b7..77d080bde52d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);


----->8-----

> [  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
> [  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
> [  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
> [  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
> [  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
> [  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
> [  128.358416] iommu: Adding device 0000:81:10.0 to group 5
> [  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
> [  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> [  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
> [  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> [  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
> [  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
> [  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
> [  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
> [  128.372176] ixgbevf 0000:81:10.0: MAC: 1
> [  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> [  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
> [  138.089869] nfs: server 172.18.45.166 not responding, still trying
> [  174.697868] nfs: server 172.18.45.166 not responding, still trying
> [  174.698359] nfs: server 172.18.45.166 OK
> [  174.698582] nfs: server 172.18.45.166 OK
> [  465.942259] VFIO - User Level meta-driver version: 0.3
>  [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
> [  472.754075] specified DMA range outside IOMMU capability
> [  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops
> 
> 1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996
> 
> 
>  
> 

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Mon, Mar 27, 2017 at 05:18:15PM +0100, Robin Murphy wrote:

[...]

> >> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> >> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> >> 0xffffffff 0xffffffff
> >> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> >> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> >> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> >> may have to assign one manually
> >> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> >> administrator.
> >> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> >> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> >> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> >> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> >> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> >> None
> >> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.026122] nfs: server 172.18.45.166 OK
> >> [  191.026317] nfs: server 172.18.45.166 OK
> >> [  263.706402] VFIO - User Level meta-driver version: 0.3
> >> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> >> 0xffffffffffffffff
> >> [  269.757617] specified DMA range outside IOMMU capability
> >> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> >> platform DMA ops
> >>
> >>  From the logs its clear that  when ixgbevf driver originally probes and adds
> >> the device
> >>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> > 
> > Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
> 
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.

Yes. On ACPI the additional issue is, given that we now call
*_dma_configure multiple times, we end up adding the _same_ stream
id to an existing fwspec ids array which triggers the SMMUv3 driver bug,
because we try to configure the same id twice (it does not happen
in DT since you have a check similar to the one I added below).

ACPI fix(es) below please let me know if it actually fixes the issues:

-- >8 --
diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 36a9abf..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 823b005..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1376,18 +1376,20 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
 	if (IS_ERR(iommu))
 		return PTR_ERR(iommu);
+
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
 
 	return 0;
 }

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Mon, Mar 27, 2017 at 05:18:15PM +0100, Robin Murphy wrote:

[...]

> >> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> >> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> >> 0xffffffff 0xffffffff
> >> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> >> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> >> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> >> may have to assign one manually
> >> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> >> administrator.
> >> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> >> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> >> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> >> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> >> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> >> None
> >> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.026122] nfs: server 172.18.45.166 OK
> >> [  191.026317] nfs: server 172.18.45.166 OK
> >> [  263.706402] VFIO - User Level meta-driver version: 0.3
> >> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> >> 0xffffffffffffffff
> >> [  269.757617] specified DMA range outside IOMMU capability
> >> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> >> platform DMA ops
> >>
> >>  From the logs its clear that  when ixgbevf driver originally probes and adds
> >> the device
> >>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> > 
> > Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
> 
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.

Yes. On ACPI the additional issue is, given that we now call
*_dma_configure multiple times, we end up adding the _same_ stream
id to an existing fwspec ids array which triggers the SMMUv3 driver bug,
because we try to configure the same id twice (it does not happen
in DT since you have a check similar to the one I added below).

ACPI fix(es) below please let me know if it actually fixes the issues:

-- >8 --
diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 36a9abf..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 823b005..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1376,18 +1376,20 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
 	if (IS_ERR(iommu))
 		return PTR_ERR(iommu);
+
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
 
 	return 0;
 }

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Mon, Mar 27, 2017 at 05:18:15PM +0100, Robin Murphy wrote:

[...]

> >> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
> >> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
> >> 0xffffffff 0xffffffff
> >> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
> >> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
> >> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
> >> may have to assign one manually
> >> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
> >> administrator.
> >> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
> >> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
> >> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
> >> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
> >> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
> >> None
> >> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
> >> [  191.026122] nfs: server 172.18.45.166 OK
> >> [  191.026317] nfs: server 172.18.45.166 OK
> >> [  263.706402] VFIO - User Level meta-driver version: 0.3
> >> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
> >> 0xffffffffffffffff
> >> [  269.757617] specified DMA range outside IOMMU capability
> >> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
> >> platform DMA ops
> >>
> >>  From the logs its clear that  when ixgbevf driver originally probes and adds
> >> the device
> >>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
> > 
> > Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
> 
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.

Yes. On ACPI the additional issue is, given that we now call
*_dma_configure multiple times, we end up adding the _same_ stream
id to an existing fwspec ids array which triggers the SMMUv3 driver bug,
because we try to configure the same id twice (it does not happen
in DT since you have a check similar to the one I added below).

ACPI fix(es) below please let me know if it actually fixes the issues:

-- >8 --
diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 36a9abf..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 823b005..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1376,18 +1376,20 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
 	if (IS_ERR(iommu))
 		return PTR_ERR(iommu);
+
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
 
 	return 0;
 }

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

>>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>>> Hi Sricharan,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
>>> [...]
>>>>>> Looks like this triggers the start of the bug.
>>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>>
>>>>>>          if (domain->geometry.force_aperture) {
>>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>>
>>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>>> aperture_start and end values to see why the check fails ?
>>>>>
>>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>>
>>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>>> 0xffffffff 0xffffffff
>>>>> .....
>>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>>
>>>>> Yes, size seems to be the problem here. When the VF  device gets
>>> attached
>>>> to vfio-pci,
>>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>>> become
>>>> zero.
>>>>
>>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>>> easily enough for testing), or patch 6, implying that somehow the
>>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>>> very soon after after creation (where we originally called
>>>> of_dma_configure()) and immediately before probe (where we now call
>>> it).
>>>>
>>>> Either way I guess this is yet more motivation to write that "change the
>>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>>
>>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>>> find the
>>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>>> ...
>>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff   ---->dev_info()
>>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.981298] nfs: server 172.18.45.166 OK
>>> [  188.981593] nfs: server 172.18.45.166 OK
>>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>>> ...
>>>
>>> Applied up to patch 6, and the issue appeared,
>>>
>>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.026122] nfs: server 172.18.45.166 OK
>>> [  191.026317] nfs: server 172.18.45.166 OK
>>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>>> 0xffffffffffffffff
>>> [  269.757617] specified DMA range outside IOMMU capability
>>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>>> platform DMA ops
>>>
>>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>>> the device
>>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
>>
>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
>
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.
>

ok, i will add this fix separately and also the acpi fix that
lorenzo has suggested in patch #8 in to the series after
testing confirmation.

Regards,
  Sricharan

>
> -----8<-----
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 9933077df7b7..77d080bde52d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
> device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

>>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>>> Hi Sricharan,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>>> [...]
>>>>>> Looks like this triggers the start of the bug.
>>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>>
>>>>>>          if (domain->geometry.force_aperture) {
>>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>>
>>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>>> aperture_start and end values to see why the check fails ?
>>>>>
>>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>>
>>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>>> 0xffffffff 0xffffffff
>>>>> .....
>>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>>
>>>>> Yes, size seems to be the problem here. When the VF  device gets
>>> attached
>>>> to vfio-pci,
>>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>>> become
>>>> zero.
>>>>
>>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>>> easily enough for testing), or patch 6, implying that somehow the
>>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>>> very soon after after creation (where we originally called
>>>> of_dma_configure()) and immediately before probe (where we now call
>>> it).
>>>>
>>>> Either way I guess this is yet more motivation to write that "change the
>>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>>
>>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>>> find the
>>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>>> ...
>>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff   ---->dev_info()
>>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.981298] nfs: server 172.18.45.166 OK
>>> [  188.981593] nfs: server 172.18.45.166 OK
>>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>>> ...
>>>
>>> Applied up to patch 6, and the issue appeared,
>>>
>>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.026122] nfs: server 172.18.45.166 OK
>>> [  191.026317] nfs: server 172.18.45.166 OK
>>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>>> 0xffffffffffffffff
>>> [  269.757617] specified DMA range outside IOMMU capability
>>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>>> platform DMA ops
>>>
>>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>>> the device
>>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
>>
>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
>
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.
>

ok, i will add this fix separately and also the acpi fix that
lorenzo has suggested in patch #8 in to the series after
testing confirmation.

Regards,
  Sricharan

>
> -----8<-----
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 9933077df7b7..77d080bde52d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
> device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

>>>> On 24/03/17 09:27, Shameerali Kolothum Thodi wrote:
>>>>> Hi Sricharan,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Sricharan R [mailto:sricharan at codeaurora.org]
>>> [...]
>>>>>> Looks like this triggers the start of the bug.
>>>>>> So the below check in iommu_dma_init_domain fails,
>>>>>>
>>>>>>          if (domain->geometry.force_aperture) {
>>>>>>                  if (base > domain->geometry.aperture_end ||
>>>>>>                      base + size <= domain->geometry.aperture_start) {
>>>>>>
>>>>>> and the rest goes out of sync after that. Can you print out the base,
>>>>>> aperture_start and end values to see why the check fails ?
>>>>>
>>>>> dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size,
>>>> domain->geometry.aperture_start, domain->geometry.aperture_end,
>>>> *dev->dma_mask, dev->coherent_dma_mask);
>>>>>
>>>>> [  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>>> 0xffffffff 0xffffffff
>>>>> .....
>>>>> [  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff,
>>>> 0xffffffffffffffff 0xffffffffffffffff
>>>>>
>>>>> Yes, size seems to be the problem here. When the VF  device gets
>>> attached
>>>> to vfio-pci,
>>>>> somehow the dev->coherent_dma_mask is set to 64 bits and size
>>> become
>>>> zero.
>>>>
>>>> AFAICS, this is either down to patch 3 (which should apply on its own
>>>> easily enough for testing), or patch 6, implying that somehow the
>>>> vfio-pci device gets its DMA mask widened to 64 bits somewhere between
>>>> very soon after after creation (where we originally called
>>>> of_dma_configure()) and immediately before probe (where we now call
>>> it).
>>>>
>>>> Either way I guess this is yet more motivation to write that "change the
>>>> arch_setup_dma_ops() interface to take a mask instead of a size" patch...
>>>
>>> Just applying the patch 3 and binding the device into vfio-pci is fine. Please
>>> find the
>>> log below (with  dev_info debug added to iommu_dma_init_domain ).
>>> ...
>>> [  142.851906] iommu: Adding device 0000:81:10.0 to group 6
>>> [  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff   ---->dev_info()
>>> [  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
>>> [  142.865692] ixgbevf 0000:81:10.0: MAC: 1
>>> [  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  152.902894] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.980933] nfs: server 172.18.45.166 not responding, still trying
>>> [  188.981298] nfs: server 172.18.45.166 OK
>>> [  188.981593] nfs: server 172.18.45.166 OK
>>> [  221.755626] VFIO - User Level meta-driver version: 0.3
>>> ...
>>>
>>> Applied up to patch 6, and the issue appeared,
>>>
>>> [  145.212351] iommu: Adding device 0000:81:10.0 to group 5
>>> [  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff,
>>> 0xffffffff 0xffffffff
>>> [  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
>>> [  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
>>> [  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you
>>> may have to assign one manually
>>> [  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by
>>> administrator.
>>> [  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
>>> [  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
>>> [  145.225944] ixgbevf 0000:81:10.0: MAC: 1
>>> [  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
>>> [  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control:
>>> None
>>> [  154.947742] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.025780] nfs: server 172.18.45.166 not responding, still trying
>>> [  191.026122] nfs: server 172.18.45.166 OK
>>> [  191.026317] nfs: server 172.18.45.166 OK
>>> [  263.706402] VFIO - User Level meta-driver version: 0.3
>>> [  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff
>>> 0xffffffffffffffff
>>> [  269.757617] specified DMA range outside IOMMU capability
>>> [  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining
>>> platform DMA ops
>>>
>>>  From the logs its clear that  when ixgbevf driver originally probes and adds
>>> the device
>>>  to smmu  the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.
>>
>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]
>
> Aha, but of course it's still the same struct device getting bound to
> VFIO later, so whatever mask the first driver set is still in there when
> we go through of_dma_configure() the second time (and the fact that we
> go through more than once being the new behaviour). So yes, this is a
> legitimate problem and we really do need to be robust against size
> overflow. I reckon the below tweak of your fix is probably the way to
> go; cleaning up the arch_setup_dma_ops() interface can happen later.
>

ok, i will add this fix separately and also the acpi fix that
lorenzo has suggested in patch #8 in to the series after
testing confirmation.

Regards,
  Sricharan

>
> -----8<-----
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 9933077df7b7..77d080bde52d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct
> device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
> Sent: Tuesday, March 28, 2017 5:54 AM
> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
> will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org;
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org;
> linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
> okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi,
> 
[...]

> >>>  From the logs its clear that  when ixgbevf driver originally probes
> >>> and adds the device  to smmu  the dma mask is 32, but when it binds
> >>> to vfio-pci, it becomes 64 bit.
> >>
> >> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
> >> probe[1]
> >
> > Aha, but of course it's still the same struct device getting bound to
> > VFIO later, so whatever mask the first driver set is still in there
> > when we go through of_dma_configure() the second time (and the fact
> > that we go through more than once being the new behaviour). So yes,
> > this is a legitimate problem and we really do need to be robust
> > against size overflow. I reckon the below tweak of your fix is
> > probably the way to go; cleaning up the arch_setup_dma_ops() interface
> can happen later.
> >
> 
> ok, i will add this fix separately and also the acpi fix that lorenzo has
> suggested in patch #8 in to the series after testing confirmation.
> 
I can confirm that the patches fixes the issues reported here . Both 
DT and ACPI works now.
 
Cheers,
Shameer 

RE: [PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Sricharan R [mailto:sricharan@codeaurora.org]
> Sent: Tuesday, March 28, 2017 5:54 AM
> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
> will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com;
> iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
> linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
> okaya@codeaurora.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi,
> 
[...]

> >>>  From the logs its clear that  when ixgbevf driver originally probes
> >>> and adds the device  to smmu  the dma mask is 32, but when it binds
> >>> to vfio-pci, it becomes 64 bit.
> >>
> >> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
> >> probe[1]
> >
> > Aha, but of course it's still the same struct device getting bound to
> > VFIO later, so whatever mask the first driver set is still in there
> > when we go through of_dma_configure() the second time (and the fact
> > that we go through more than once being the new behaviour). So yes,
> > this is a legitimate problem and we really do need to be robust
> > against size overflow. I reckon the below tweak of your fix is
> > probably the way to go; cleaning up the arch_setup_dma_ops() interface
> can happen later.
> >
> 
> ok, i will add this fix separately and also the acpi fix that lorenzo has
> suggested in patch #8 in to the series after testing confirmation.
> 
I can confirm that the patches fixes the issues reported here . Both 
DT and ACPI works now.
 
Cheers,
Shameer 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Sricharan R [mailto:sricharan at codeaurora.org]
> Sent: Tuesday, March 28, 2017 5:54 AM
> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
> will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com;
> iommu at lists.linux-foundation.org; linux-arm-kernel at lists.infradead.org;
> linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
> okaya at codeaurora.org
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi,
> 
[...]

> >>>  From the logs its clear that  when ixgbevf driver originally probes
> >>> and adds the device  to smmu  the dma mask is 32, but when it binds
> >>> to vfio-pci, it becomes 64 bit.
> >>
> >> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
> >> probe[1]
> >
> > Aha, but of course it's still the same struct device getting bound to
> > VFIO later, so whatever mask the first driver set is still in there
> > when we go through of_dma_configure() the second time (and the fact
> > that we go through more than once being the new behaviour). So yes,
> > this is a legitimate problem and we really do need to be robust
> > against size overflow. I reckon the below tweak of your fix is
> > probably the way to go; cleaning up the arch_setup_dma_ops() interface
> can happen later.
> >
> 
> ok, i will add this fix separately and also the acpi fix that lorenzo has
> suggested in patch #8 in to the series after testing confirmation.
> 
I can confirm that the patches fixes the issues reported here . Both 
DT and ACPI works now.
 
Cheers,
Shameer 

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Rob Herring]

On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
> 
> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)

Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
with v8 of this series. Bisecting blames this commit which looks like it 
hasn't changeed. 

Rob

P.S. Doesn't look like you have copied the DT maintainers nor list for 
the DT changes.

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Rob Herring]

On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)

Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
with v8 of this series. Bisecting blames this commit which looks like it 
hasn't changeed. 

Rob

P.S. Doesn't look like you have copied the DT maintainers nor list for 
the DT changes.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Rob Herring]

On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)

Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
with v8 of this series. Bisecting blames this commit which looks like it 
hasn't changeed. 

Rob

P.S. Doesn't look like you have copied the DT maintainers nor list for 
the DT changes.

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 28/03/17 16:00, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
> 
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
> with v8 of this series. Bisecting blames this commit which looks like it 
> hasn't changeed. 

v8 managed to break *all* PCI devices which didn't have an associated
"iommu-map". The problem was actually in patch 2 (and is fixed in this
version), but it only hit at this point once we start propagating the
erroneous error code back to the driver probe.

Robin.

> 
> Rob
> 
> P.S. Doesn't look like you have copied the DT maintainers nor list for 
> the DT changes.
> 

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 28/03/17 16:00, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
> 
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
> with v8 of this series. Bisecting blames this commit which looks like it 
> hasn't changeed. 

v8 managed to break *all* PCI devices which didn't have an associated
"iommu-map". The problem was actually in patch 2 (and is fixed in this
version), but it only hit at this point once we start propagating the
erroneous error code back to the driver probe.

Robin.

> 
> Rob
> 
> P.S. Doesn't look like you have copied the DT maintainers nor list for 
> the DT changes.
> 

[V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 28/03/17 16:00, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
> 
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken 
> with v8 of this series. Bisecting blames this commit which looks like it 
> hasn't changeed. 

v8 managed to break *all* PCI devices which didn't have an associated
"iommu-map". The problem was actually in patch 2 (and is fixed in this
version), but it only hit at this point once we start propagating the
erroneous error code back to the driver probe.

Robin.

> 
> Rob
> 
> P.S. Doesn't look like you have copied the DT maintainers nor list for 
> the DT changes.
> 

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 3/28/2017 8:30 PM, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken
> with v8 of this series. Bisecting blames this commit which looks like it
> hasn't changeed.

Ya, as Robin mentioned it was broken at patch #2 and
comes out after this patch. I am going to repost this series with
couple of more fixes added as well.

>
> Rob
>
> P.S. Doesn't look like you have copied the DT maintainers nor list for
> the DT changes.

Ha, really sorry about that. will add it.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 3/28/2017 8:30 PM, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken
> with v8 of this series. Bisecting blames this commit which looks like it
> hasn't changeed.

Ya, as Robin mentioned it was broken at patch #2 and
comes out after this patch. I am going to repost this series with
couple of more fixes added as well.

>
> Rob
>
> P.S. Doesn't look like you have copied the DT maintainers nor list for
> the DT changes.

Ha, really sorry about that. will add it.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[V9, 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 3/28/2017 8:30 PM, Rob Herring wrote:
> On Fri, Mar 10, 2017 at 12:30:57AM +0530, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> Maybe it is the same issue reported for VFIO, but virtio-pci is broken
> with v8 of this series. Bisecting blames this commit which looks like it
> hasn't changeed.

Ya, as Robin mentioned it was broken at patch #2 and
comes out after this patch. I am going to repost this series with
couple of more fixes added as well.

>
> Rob
>
> P.S. Doesn't look like you have copied the DT maintainers nor list for
> the DT changes.

Ha, really sorry about that. will add it.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

On 3/28/2017 7:45 PM, Shameerali Kolothum Thodi wrote:
>
>
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
>> Sent: Tuesday, March 28, 2017 5:54 AM
>> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
>> will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org;
>> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org;
>> linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
>> bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
>> acpi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; tn-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org; hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org;
>> okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi,
>>
> [...]
>
>>>>>  From the logs its clear that  when ixgbevf driver originally probes
>>>>> and adds the device  to smmu  the dma mask is 32, but when it binds
>>>>> to vfio-pci, it becomes 64 bit.
>>>>
>>>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
>>>> probe[1]
>>>
>>> Aha, but of course it's still the same struct device getting bound to
>>> VFIO later, so whatever mask the first driver set is still in there
>>> when we go through of_dma_configure() the second time (and the fact
>>> that we go through more than once being the new behaviour). So yes,
>>> this is a legitimate problem and we really do need to be robust
>>> against size overflow. I reckon the below tweak of your fix is
>>> probably the way to go; cleaning up the arch_setup_dma_ops() interface
>> can happen later.
>>>
>>
>> ok, i will add this fix separately and also the acpi fix that lorenzo has
>> suggested in patch #8 in to the series after testing confirmation.
>>
> I can confirm that the patches fixes the issues reported here . Both
> DT and ACPI works now.
>

Thanks for the testing. Will repost with the fixes.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

On 3/28/2017 7:45 PM, Shameerali Kolothum Thodi wrote:
>
>
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan@codeaurora.org]
>> Sent: Tuesday, March 28, 2017 5:54 AM
>> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
>> will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com;
>> iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
>> linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com;
>> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
>> acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org;
>> okaya@codeaurora.org
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi,
>>
> [...]
>
>>>>>  From the logs its clear that  when ixgbevf driver originally probes
>>>>> and adds the device  to smmu  the dma mask is 32, but when it binds
>>>>> to vfio-pci, it becomes 64 bit.
>>>>
>>>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
>>>> probe[1]
>>>
>>> Aha, but of course it's still the same struct device getting bound to
>>> VFIO later, so whatever mask the first driver set is still in there
>>> when we go through of_dma_configure() the second time (and the fact
>>> that we go through more than once being the new behaviour). So yes,
>>> this is a legitimate problem and we really do need to be robust
>>> against size overflow. I reckon the below tweak of your fix is
>>> probably the way to go; cleaning up the arch_setup_dma_ops() interface
>> can happen later.
>>>
>>
>> ok, i will add this fix separately and also the acpi fix that lorenzo has
>> suggested in patch #8 in to the series after testing confirmation.
>>
> I can confirm that the patches fixes the issues reported here . Both
> DT and ACPI works now.
>

Thanks for the testing. Will repost with the fixes.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V9 00/11] IOMMU probe deferral support

[Sricharan R]

Hi,

On 3/28/2017 7:45 PM, Shameerali Kolothum Thodi wrote:
>
>
>> -----Original Message-----
>> From: Sricharan R [mailto:sricharan at codeaurora.org]
>> Sent: Tuesday, March 28, 2017 5:54 AM
>> To: Robin Murphy; Shameerali Kolothum Thodi; Wangzhou (B);
>> will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com;
>> iommu at lists.linux-foundation.org; linux-arm-kernel at lists.infradead.org;
>> linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com;
>> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
>> acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org;
>> okaya at codeaurora.org
>> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
>>
>> Hi,
>>
> [...]
>
>>>>>  From the logs its clear that  when ixgbevf driver originally probes
>>>>> and adds the device  to smmu  the dma mask is 32, but when it binds
>>>>> to vfio-pci, it becomes 64 bit.
>>>>
>>>> Just to add to that, the mask is set to 64 bit in the ixgebvf driver
>>>> probe[1]
>>>
>>> Aha, but of course it's still the same struct device getting bound to
>>> VFIO later, so whatever mask the first driver set is still in there
>>> when we go through of_dma_configure() the second time (and the fact
>>> that we go through more than once being the new behaviour). So yes,
>>> this is a legitimate problem and we really do need to be robust
>>> against size overflow. I reckon the below tweak of your fix is
>>> probably the way to go; cleaning up the arch_setup_dma_ops() interface
>> can happen later.
>>>
>>
>> ok, i will add this fix separately and also the acpi fix that lorenzo has
>> suggested in patch #8 in to the series after testing confirmation.
>>
> I can confirm that the patches fixes the issues reported here . Both
> DT and ACPI works now.
>

Thanks for the testing. Will repost with the fixes.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 00/12] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
against mainline and linux-next, iommu-next.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [8]. 

Please note that, i have kept the tested/acked tags intact from V8
because V9/10 were for more fixes that was added, so the original
tags that was given for the functional testing remains the same.

 [V10]
     * Rebased on top of 4.11-rc5.
 
     * Fixed coherent_dma_mask 64bit overflow issue [8]
       for OF. The fix for OF was added as a separate
       patch#6, since the issue is true even without probe deferral,
       but gets reproduced with the probe deferral series.
       Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

     * Also fixed an build issue caught by kbuild robot for
       m68k arch. The issue was dma_(de)configure was not
       getting defined for !CONFIG_HAS_DMA, so fixed that as well.

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property and deferencing
       of_iommu_table after init.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html
[8] http://patchwork.ozlabs.org/patch/743898/

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (4):
  of: device: Fix overflow of coherent_dma_mask
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |  11 ++-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  36 +++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |  12 ++++
 include/linux/of_device.h         |  10 ++-
 19 files changed, 329 insertions(+), 338 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 00/12] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
against mainline and linux-next, iommu-next.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [8]. 

Please note that, i have kept the tested/acked tags intact from V8
because V9/10 were for more fixes that was added, so the original
tags that was given for the functional testing remains the same.

 [V10]
     * Rebased on top of 4.11-rc5.
 
     * Fixed coherent_dma_mask 64bit overflow issue [8]
       for OF. The fix for OF was added as a separate
       patch#6, since the issue is true even without probe deferral,
       but gets reproduced with the probe deferral series.
       Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

     * Also fixed an build issue caught by kbuild robot for
       m68k arch. The issue was dma_(de)configure was not
       getting defined for !CONFIG_HAS_DMA, so fixed that as well.

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property and deferencing
       of_iommu_table after init.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html
[8] http://patchwork.ozlabs.org/patch/743898/

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (4):
  of: device: Fix overflow of coherent_dma_mask
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |  11 ++-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  36 +++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |  12 ++++
 include/linux/of_device.h         |  10 ++-
 19 files changed, 329 insertions(+), 338 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 00/12] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
against mainline and linux-next, iommu-next.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [8]. 

Please note that, i have kept the tested/acked tags intact from V8
because V9/10 were for more fixes that was added, so the original
tags that was given for the functional testing remains the same.

 [V10]
     * Rebased on top of 4.11-rc5.
 
     * Fixed coherent_dma_mask 64bit overflow issue [8]
       for OF. The fix for OF was added as a separate
       patch#6, since the issue is true even without probe deferral,
       but gets reproduced with the probe deferral series.
       Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

     * Also fixed an build issue caught by kbuild robot for
       m68k arch. The issue was dma_(de)configure was not
       getting defined for !CONFIG_HAS_DMA, so fixed that as well.

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property and deferencing
       of_iommu_table after init.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html
[8] http://patchwork.ozlabs.org/patch/743898/

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (4):
  of: device: Fix overflow of coherent_dma_mask
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |  11 ++-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  36 +++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |  12 ++++
 include/linux/of_device.h         |  10 ++-
 19 files changed, 329 insertions(+), 338 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 00/12] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
against mainline and linux-next, iommu-next.
  
* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

Previous post of this series [8]. 

Please note that, i have kept the tested/acked tags intact from V8
because V9/10 were for more fixes that was added, so the original
tags that was given for the functional testing remains the same.

 [V10]
     * Rebased on top of 4.11-rc5.
 
     * Fixed coherent_dma_mask 64bit overflow issue [8]
       for OF. The fix for OF was added as a separate
       patch#6, since the issue is true even without probe deferral,
       but gets reproduced with the probe deferral series.
       Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

     * Also fixed an build issue caught by kbuild robot for
       m68k arch. The issue was dma_(de)configure was not
       getting defined for !CONFIG_HAS_DMA, so fixed that as well.

 [V9]
     * Rebased on top of 4.11-rc1.

     * Merged Robin's fixes for legacy binding issue,
       pci devices with no iommu-map property and deferencing
       of_iommu_table after init.
     
 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html
[8] http://patchwork.ozlabs.org/patch/743898/

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (4):
  of: device: Fix overflow of coherent_dma_mask
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |  11 ++-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  36 +++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |  12 ++++
 include/linux/of_device.h         |  10 ++-
 19 files changed, 329 insertions(+), 338 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 01/12] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 01/12] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 01/12] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 01/12] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2683e9f..8f4e599 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 02/12] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 02/12] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 02/12] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 02/12] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 8f4e599..c8be889 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5dfcc96..5344db5 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index c12dace..af98455 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
Cc: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
Cc: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 4a5bb96..3dd9ec3 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Size of the dma-range is calculated as coherent_dma_mask + 1
and passed to arch_setup_dma_ops further. It overflows when
the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
resulting in size getting passed as 0 wrongly. Fix this by
passsing in max(mask, mask + 1). Note that in this case
when the mask is set to full 64bits, we will be passing the mask
itself to arch_setup_dma_ops instead of the size. The real fix
for this should be to make arch_setup_dma_ops receive the
mask and handle it, to be done in the future.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/of/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..c2ae6bb 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Size of the dma-range is calculated as coherent_dma_mask + 1
and passed to arch_setup_dma_ops further. It overflows when
the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
resulting in size getting passed as 0 wrongly. Fix this by
passsing in max(mask, mask + 1). Note that in this case
when the mask is set to full 64bits, we will be passing the mask
itself to arch_setup_dma_ops instead of the size. The real fix
for this should be to make arch_setup_dma_ops receive the
mask and handle it, to be done in the future.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/of/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..c2ae6bb 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Size of the dma-range is calculated as coherent_dma_mask + 1
and passed to arch_setup_dma_ops further. It overflows when
the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
resulting in size getting passed as 0 wrongly. Fix this by
passsing in max(mask, mask + 1). Note that in this case
when the mask is set to full 64bits, we will be passing the mask
itself to arch_setup_dma_ops instead of the size. The real fix
for this should be to make arch_setup_dma_ops receive the
mask and handle it, to be done in the future.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/of/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..c2ae6bb 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Bjorn Helgaas <bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---

 [V10] Added dummy dma_(de)configure functions in case
       of !CONFIG_HAS_DMA to avoid build breaks.

 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h | 12 ++++++++++++
 6 files changed, 62 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5344db5..2aa4ebb 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dfc9a27..5a8dd43 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 0977317..4f3eece 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+#ifdef CONFIG_HAS_DMA
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+#else
+static inline int dma_configure(struct device *dev)
+{
+	return 0;
+}
+
+static inline void dma_deconfigure(struct device *dev) {}
+#endif
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---

 [V10] Added dummy dma_(de)configure functions in case
       of !CONFIG_HAS_DMA to avoid build breaks.

 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h | 12 ++++++++++++
 6 files changed, 62 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5344db5..2aa4ebb 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dfc9a27..5a8dd43 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 0977317..4f3eece 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+#ifdef CONFIG_HAS_DMA
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+#else
+static inline int dma_configure(struct device *dev)
+{
+	return 0;
+}
+
+static inline void dma_deconfigure(struct device *dev) {}
+#endif
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---

 [V10] Added dummy dma_(de)configure functions in case
       of !CONFIG_HAS_DMA to avoid build breaks.

 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h | 12 ++++++++++++
 6 files changed, 62 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 5344db5..2aa4ebb 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dfc9a27..5a8dd43 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 0977317..4f3eece 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+#ifdef CONFIG_HAS_DMA
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+#else
+static inline int dma_configure(struct device *dev)
+{
+	return 0;
+}
+
+static inline void dma_deconfigure(struct device *dev) {}
+#endif
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 08/12] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c2ae6bb..bea8aec 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 08/12] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c2ae6bb..bea8aec 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 08/12] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index c8be889..9f44ee8 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c2ae6bb..bea8aec 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index af98455..2cacdd8 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
[Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
          called multiple times for same device]
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---

 [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

 drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
 drivers/acpi/scan.c        | 11 ++++++++---
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 3dd9ec3..e323ece 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
@@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2a9a5de 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 9b05886..79d06ef6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
[Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
          called multiple times for same device]
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---

 [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

 drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
 drivers/acpi/scan.c        | 11 ++++++++---
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 3dd9ec3..e323ece 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
@@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2a9a5de 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 9b05886..79d06ef6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
[Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
          called multiple times for same device]
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---

 [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
       and the fix for dma_configure getting called more than
       once for the same device.

 drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
 drivers/acpi/scan.c        | 11 ++++++++---
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 3dd9ec3..e323ece 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);
@@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..2a513cc 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2a9a5de 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 9b05886..79d06ef6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 10/12] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's iommu is probed, the notifier trick required to handle the
early setup of dma_ops before the iommu group gets created is not
required. So removing the notifier's here.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 81cdb2e..b465759 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 10/12] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's iommu is probed, the notifier trick required to handle the
early setup of dma_ops before the iommu group gets created is not
required. So removing the notifier's here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 81cdb2e..b465759 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 10/12] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's iommu is probed, the notifier trick required to handle the
early setup of dma_ops before the iommu group gets created is not
required. So removing the notifier's here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 81cdb2e..b465759 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 11/12] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 591bb96..4362139 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index b493c99..792fe7d 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2075,6 +2075,23 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
+static void arm_smmu_bus_init(void)
+{
+	/* Oh, for a proper bus abstraction */
+	if (!iommu_present(&platform_bus_type))
+		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+	if (!iommu_present(&amba_bustype))
+		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+	if (!iommu_present(&pci_bus_type)) {
+		pci_request_acs();
+		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+	}
+#endif
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2180,21 +2197,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-	}
-#endif
+	/*
+	 * For ACPI and generic DT bindings, an SMMU will be probed before
+	 * any device which might need it, so we want the bus ops in place
+	 * ready to handle default domain setup as soon as any SMMU exists.
+	 */
+	if (!using_legacy_binding)
+		arm_smmu_bus_init();
+
+	return 0;
+}
+
+/*
+ * With the legacy DT binding in play, though, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		arm_smmu_bus_init();
 	return 0;
 }
+device_initcall_sync(arm_smmu_legacy_bus_init);
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
@@ -2219,56 +2245,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 11/12] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 591bb96..4362139 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index b493c99..792fe7d 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2075,6 +2075,23 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
+static void arm_smmu_bus_init(void)
+{
+	/* Oh, for a proper bus abstraction */
+	if (!iommu_present(&platform_bus_type))
+		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+	if (!iommu_present(&amba_bustype))
+		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+	if (!iommu_present(&pci_bus_type)) {
+		pci_request_acs();
+		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+	}
+#endif
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2180,21 +2197,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-	}
-#endif
+	/*
+	 * For ACPI and generic DT bindings, an SMMU will be probed before
+	 * any device which might need it, so we want the bus ops in place
+	 * ready to handle default domain setup as soon as any SMMU exists.
+	 */
+	if (!using_legacy_binding)
+		arm_smmu_bus_init();
+
+	return 0;
+}
+
+/*
+ * With the legacy DT binding in play, though, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		arm_smmu_bus_init();
 	return 0;
 }
+device_initcall_sync(arm_smmu_legacy_bus_init);
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
@@ -2219,56 +2245,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 11/12] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 591bb96..4362139 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index b493c99..792fe7d 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2075,6 +2075,23 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	return 0;
 }
 
+static void arm_smmu_bus_init(void)
+{
+	/* Oh, for a proper bus abstraction */
+	if (!iommu_present(&platform_bus_type))
+		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+	if (!iommu_present(&amba_bustype))
+		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+	if (!iommu_present(&pci_bus_type)) {
+		pci_request_acs();
+		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
+	}
+#endif
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	struct resource *res;
@@ -2180,21 +2197,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
-	/* Oh, for a proper bus abstraction */
-	if (!iommu_present(&platform_bus_type))
-		bus_set_iommu(&platform_bus_type, &arm_smmu_ops);
-#ifdef CONFIG_ARM_AMBA
-	if (!iommu_present(&amba_bustype))
-		bus_set_iommu(&amba_bustype, &arm_smmu_ops);
-#endif
-#ifdef CONFIG_PCI
-	if (!iommu_present(&pci_bus_type)) {
-		pci_request_acs();
-		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
-	}
-#endif
+	/*
+	 * For ACPI and generic DT bindings, an SMMU will be probed before
+	 * any device which might need it, so we want the bus ops in place
+	 * ready to handle default domain setup as soon as any SMMU exists.
+	 */
+	if (!using_legacy_binding)
+		arm_smmu_bus_init();
+
+	return 0;
+}
+
+/*
+ * With the legacy DT binding in play, though, we have no guarantees about
+ * probe order, but then we're also not doing default domains, so we can
+ * delay setting bus ops until we're sure every possible SMMU is ready,
+ * and that way ensure that no add_device() calls get missed.
+ */
+static int arm_smmu_legacy_bus_init(void)
+{
+	if (using_legacy_binding)
+		arm_smmu_bus_init();
 	return 0;
 }
+device_initcall_sync(arm_smmu_legacy_bus_init);
 
 static int arm_smmu_device_remove(struct platform_device *pdev)
 {
@@ -2219,56 +2245,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
Cc: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
Cc: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e323ece..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 0968d13..9faa26c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e323ece..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 0968d13..9faa26c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e323ece..e7b1940 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 0968d13..9faa26c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Just for completeness:

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Just for completeness:

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Just for completeness:

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Just for completeness:

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

Re: [PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> 
> As part of moving DMA initializing to probe time the
> of_dma_deconfigure() function will need to be called from different
> source files. Make it public and move it to drivers/of/device.c where
> the of_dma_configure() function is.

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> ---
>  drivers/of/device.c       | 12 ++++++++++++
>  drivers/of/platform.c     |  5 -----
>  include/linux/of_device.h |  3 +++
>  3 files changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 09dedd0..c17c19d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> +/**
> + * of_dma_deconfigure - Clean up DMA configuration
> + * @dev:	Device for which to clean up DMA configuration
> + *
> + * Clean up all configuration performed by of_dma_configure_ops() and free all
> + * resources that have been allocated.
> + */
> +void of_dma_deconfigure(struct device *dev)
> +{
> +	arch_teardown_dma_ops(dev);
> +}
> +
>  int of_device_register(struct platform_device *pdev)
>  {
>  	device_initialize(&pdev->dev);
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5dfcc96..5344db5 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
>  }
>  EXPORT_SYMBOL(of_device_alloc);
>  
> -static void of_dma_deconfigure(struct device *dev)
> -{
> -	arch_teardown_dma_ops(dev);
> -}
> -
>  /**
>   * of_platform_device_create_pdata - Alloc, initialize and register an of_device
>   * @np: pointer to node to create device for
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index c12dace..af98455 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  
>  void of_dma_configure(struct device *dev, struct device_node *np);
> +void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
>  static inline int of_driver_match_device(struct device *dev,
> @@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  static inline void of_dma_configure(struct device *dev, struct device_node *np)
>  {}
> +static inline void of_dma_deconfigure(struct device *dev)
> +{}
>  #endif /* CONFIG_OF */
>  
>  #endif /* _LINUX_OF_DEVICE_H */
> 

Re: [PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> As part of moving DMA initializing to probe time the
> of_dma_deconfigure() function will need to be called from different
> source files. Make it public and move it to drivers/of/device.c where
> the of_dma_configure() function is.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/device.c       | 12 ++++++++++++
>  drivers/of/platform.c     |  5 -----
>  include/linux/of_device.h |  3 +++
>  3 files changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 09dedd0..c17c19d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> +/**
> + * of_dma_deconfigure - Clean up DMA configuration
> + * @dev:	Device for which to clean up DMA configuration
> + *
> + * Clean up all configuration performed by of_dma_configure_ops() and free all
> + * resources that have been allocated.
> + */
> +void of_dma_deconfigure(struct device *dev)
> +{
> +	arch_teardown_dma_ops(dev);
> +}
> +
>  int of_device_register(struct platform_device *pdev)
>  {
>  	device_initialize(&pdev->dev);
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5dfcc96..5344db5 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
>  }
>  EXPORT_SYMBOL(of_device_alloc);
>  
> -static void of_dma_deconfigure(struct device *dev)
> -{
> -	arch_teardown_dma_ops(dev);
> -}
> -
>  /**
>   * of_platform_device_create_pdata - Alloc, initialize and register an of_device
>   * @np: pointer to node to create device for
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index c12dace..af98455 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  
>  void of_dma_configure(struct device *dev, struct device_node *np);
> +void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
>  static inline int of_driver_match_device(struct device *dev,
> @@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  static inline void of_dma_configure(struct device *dev, struct device_node *np)
>  {}
> +static inline void of_dma_deconfigure(struct device *dev)
> +{}
>  #endif /* CONFIG_OF */
>  
>  #endif /* _LINUX_OF_DEVICE_H */
> 

Re: [PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> As part of moving DMA initializing to probe time the
> of_dma_deconfigure() function will need to be called from different
> source files. Make it public and move it to drivers/of/device.c where
> the of_dma_configure() function is.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/device.c       | 12 ++++++++++++
>  drivers/of/platform.c     |  5 -----
>  include/linux/of_device.h |  3 +++
>  3 files changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 09dedd0..c17c19d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> +/**
> + * of_dma_deconfigure - Clean up DMA configuration
> + * @dev:	Device for which to clean up DMA configuration
> + *
> + * Clean up all configuration performed by of_dma_configure_ops() and free all
> + * resources that have been allocated.
> + */
> +void of_dma_deconfigure(struct device *dev)
> +{
> +	arch_teardown_dma_ops(dev);
> +}
> +
>  int of_device_register(struct platform_device *pdev)
>  {
>  	device_initialize(&pdev->dev);
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5dfcc96..5344db5 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
>  }
>  EXPORT_SYMBOL(of_device_alloc);
>  
> -static void of_dma_deconfigure(struct device *dev)
> -{
> -	arch_teardown_dma_ops(dev);
> -}
> -
>  /**
>   * of_platform_device_create_pdata - Alloc, initialize and register an of_device
>   * @np: pointer to node to create device for
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index c12dace..af98455 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  
>  void of_dma_configure(struct device *dev, struct device_node *np);
> +void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
>  static inline int of_driver_match_device(struct device *dev,
> @@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  static inline void of_dma_configure(struct device *dev, struct device_node *np)
>  {}
> +static inline void of_dma_deconfigure(struct device *dev)
> +{}
>  #endif /* CONFIG_OF */
>  
>  #endif /* _LINUX_OF_DEVICE_H */
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 04/12] of: dma: Make of_dma_deconfigure() public

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> As part of moving DMA initializing to probe time the
> of_dma_deconfigure() function will need to be called from different
> source files. Make it public and move it to drivers/of/device.c where
> the of_dma_configure() function is.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/device.c       | 12 ++++++++++++
>  drivers/of/platform.c     |  5 -----
>  include/linux/of_device.h |  3 +++
>  3 files changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 09dedd0..c17c19d 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> +/**
> + * of_dma_deconfigure - Clean up DMA configuration
> + * @dev:	Device for which to clean up DMA configuration
> + *
> + * Clean up all configuration performed by of_dma_configure_ops() and free all
> + * resources that have been allocated.
> + */
> +void of_dma_deconfigure(struct device *dev)
> +{
> +	arch_teardown_dma_ops(dev);
> +}
> +
>  int of_device_register(struct platform_device *pdev)
>  {
>  	device_initialize(&pdev->dev);
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5dfcc96..5344db5 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
>  }
>  EXPORT_SYMBOL(of_device_alloc);
>  
> -static void of_dma_deconfigure(struct device *dev)
> -{
> -	arch_teardown_dma_ops(dev);
> -}
> -
>  /**
>   * of_platform_device_create_pdata - Alloc, initialize and register an of_device
>   * @np: pointer to node to create device for
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index c12dace..af98455 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  
>  void of_dma_configure(struct device *dev, struct device_node *np);
> +void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
>  static inline int of_driver_match_device(struct device *dev,
> @@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  }
>  static inline void of_dma_configure(struct device *dev, struct device_node *np)
>  {}
> +static inline void of_dma_deconfigure(struct device *dev)
> +{}
>  #endif /* CONFIG_OF */
>  
>  #endif /* _LINUX_OF_DEVICE_H */
> 

Re: [PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
> 
> The IOMMU probe deferral implementation requires a mechanism to detect
> if drivers for SMMU components are built-in in the kernel to detect
> whether IOMMU configuration for a given device should be deferred (ie
> SMMU drivers present but still not probed) or not (drivers not present).
> 
> Add a simple function to IORT to detect if SMMU drivers for SMMU
> components managed by IORT are built-in in the kernel.

Ah, if only DT could be this neat and tidy :D

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

> Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
> Cc: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> Cc: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/acpi/arm64/iort.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 4a5bb96..3dd9ec3 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
>  	return ret;
>  }
>  
> +static inline bool iort_iommu_driver_enabled(u8 type)
> +{
> +	switch (type) {
> +	case ACPI_IORT_NODE_SMMU_V3:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +	case ACPI_IORT_NODE_SMMU:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +	default:
> +		pr_warn("IORT node type %u does not describe an SMMU\n", type);
> +		return false;
> +	}
> +}
> +
>  static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  					struct acpi_iort_node *node,
>  					u32 streamid)
> 

Re: [PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IOMMU probe deferral implementation requires a mechanism to detect
> if drivers for SMMU components are built-in in the kernel to detect
> whether IOMMU configuration for a given device should be deferred (ie
> SMMU drivers present but still not probed) or not (drivers not present).
> 
> Add a simple function to IORT to detect if SMMU drivers for SMMU
> components managed by IORT are built-in in the kernel.

Ah, if only DT could be this neat and tidy :D

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 4a5bb96..3dd9ec3 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
>  	return ret;
>  }
>  
> +static inline bool iort_iommu_driver_enabled(u8 type)
> +{
> +	switch (type) {
> +	case ACPI_IORT_NODE_SMMU_V3:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +	case ACPI_IORT_NODE_SMMU:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +	default:
> +		pr_warn("IORT node type %u does not describe an SMMU\n", type);
> +		return false;
> +	}
> +}
> +
>  static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  					struct acpi_iort_node *node,
>  					u32 streamid)
> 

Re: [PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IOMMU probe deferral implementation requires a mechanism to detect
> if drivers for SMMU components are built-in in the kernel to detect
> whether IOMMU configuration for a given device should be deferred (ie
> SMMU drivers present but still not probed) or not (drivers not present).
> 
> Add a simple function to IORT to detect if SMMU drivers for SMMU
> components managed by IORT are built-in in the kernel.

Ah, if only DT could be this neat and tidy :D

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 4a5bb96..3dd9ec3 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
>  	return ret;
>  }
>  
> +static inline bool iort_iommu_driver_enabled(u8 type)
> +{
> +	switch (type) {
> +	case ACPI_IORT_NODE_SMMU_V3:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +	case ACPI_IORT_NODE_SMMU:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +	default:
> +		pr_warn("IORT node type %u does not describe an SMMU\n", type);
> +		return false;
> +	}
> +}
> +
>  static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  					struct acpi_iort_node *node,
>  					u32 streamid)
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 05/12] ACPI/IORT: Add function to check SMMUs drivers presence

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IOMMU probe deferral implementation requires a mechanism to detect
> if drivers for SMMU components are built-in in the kernel to detect
> whether IOMMU configuration for a given device should be deferred (ie
> SMMU drivers present but still not probed) or not (drivers not present).
> 
> Add a simple function to IORT to detect if SMMU drivers for SMMU
> components managed by IORT are built-in in the kernel.

Ah, if only DT could be this neat and tidy :D

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 4a5bb96..3dd9ec3 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
>  	return ret;
>  }
>  
> +static inline bool iort_iommu_driver_enabled(u8 type)
> +{
> +	switch (type) {
> +	case ACPI_IORT_NODE_SMMU_V3:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
> +	case ACPI_IORT_NODE_SMMU:
> +		return IS_BUILTIN(CONFIG_ARM_SMMU);
> +	default:
> +		pr_warn("IORT node type %u does not describe an SMMU\n", type);
> +		return false;
> +	}
> +}
> +
>  static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  					struct acpi_iort_node *node,
>  					u32 streamid)
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.

Agreed; in the meantime, this fix is neat and robust enough.

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.

Agreed; in the meantime, this fix is neat and robust enough.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.

Agreed; in the meantime, this fix is neat and robust enough.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.

Agreed; in the meantime, this fix is neat and robust enough.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 

Re: [PATCH V10 08/12] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 449b948..82bd45c 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>  {
>  	struct device *bridge = NULL, *dma_dev = dev;
>  	enum dev_dma_attr attr;
> +	int ret = 0;
>  
>  	if (dev_is_pci(dev)) {
>  		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>  	}
>  
>  	if (dma_dev->of_node) {
> -		of_dma_configure(dev, dma_dev->of_node);
> +		ret = of_dma_configure(dev, dma_dev->of_node);
>  	} else if (has_acpi_companion(dma_dev)) {
>  		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>  		if (attr != DEV_DMA_NOT_SUPPORTED)
> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>  	if (bridge)
>  		pci_put_host_bridge_device(bridge);
>  
> -	return 0;
> +	return ret;
>  }
>  
>  void dma_deconfigure(struct device *dev)
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index c8be889..9f44ee8 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>  			ops = ERR_PTR(err);
>  	}
>  
> -	return IS_ERR(ops) ? NULL : ops;
> +	return ops;
>  }
>  
>  static int __init of_iommu_init(void)
> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>  	for_each_matching_node_and_match(np, matches, &match) {
>  		const of_iommu_init_fn init_fn = match->data;
>  
> -		if (init_fn(np))
> +		if (init_fn && init_fn(np))
>  			pr_err("Failed to initialise IOMMU %s\n",
>  				of_node_full_name(np));
>  	}
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c2ae6bb..bea8aec 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>  	u64 dma_addr, paddr, size;
>  	int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		coherent ? " " : " not ");
>  
>  	iommu = of_iommu_configure(dev, np);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
> +
>  	dev_dbg(dev, "device is%sbehind an iommu\n",
>  		iommu ? " " : " not ");
>  
>  	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index af98455..2cacdd8 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  	return of_node_get(cpu_dev->of_node);
>  }
>  
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
> @@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  {
>  	return NULL;
>  }
> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +
> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
> +{
> +	return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */
> 

[PATCH V10 08/12] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 449b948..82bd45c 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>  {
>  	struct device *bridge = NULL, *dma_dev = dev;
>  	enum dev_dma_attr attr;
> +	int ret = 0;
>  
>  	if (dev_is_pci(dev)) {
>  		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>  	}
>  
>  	if (dma_dev->of_node) {
> -		of_dma_configure(dev, dma_dev->of_node);
> +		ret = of_dma_configure(dev, dma_dev->of_node);
>  	} else if (has_acpi_companion(dma_dev)) {
>  		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>  		if (attr != DEV_DMA_NOT_SUPPORTED)
> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>  	if (bridge)
>  		pci_put_host_bridge_device(bridge);
>  
> -	return 0;
> +	return ret;
>  }
>  
>  void dma_deconfigure(struct device *dev)
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index c8be889..9f44ee8 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>  			ops = ERR_PTR(err);
>  	}
>  
> -	return IS_ERR(ops) ? NULL : ops;
> +	return ops;
>  }
>  
>  static int __init of_iommu_init(void)
> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>  	for_each_matching_node_and_match(np, matches, &match) {
>  		const of_iommu_init_fn init_fn = match->data;
>  
> -		if (init_fn(np))
> +		if (init_fn && init_fn(np))
>  			pr_err("Failed to initialise IOMMU %s\n",
>  				of_node_full_name(np));
>  	}
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c2ae6bb..bea8aec 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>  	u64 dma_addr, paddr, size;
>  	int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		coherent ? " " : " not ");
>  
>  	iommu = of_iommu_configure(dev, np);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
> +
>  	dev_dbg(dev, "device is%sbehind an iommu\n",
>  		iommu ? " " : " not ");
>  
>  	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index af98455..2cacdd8 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  	return of_node_get(cpu_dev->of_node);
>  }
>  
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
> @@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  {
>  	return NULL;
>  }
> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +
> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
> +{
> +	return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */
> 

Re: [PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This is an equivalent to the DT's handling of the iommu master's probe
> with deferred probing when the corrsponding iommu is not probed yet.
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the firmware describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.

I think we really have caught everything now...

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> [Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
>           called multiple times for same device]
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>  drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
>  drivers/acpi/scan.c        | 11 ++++++++---
>  drivers/base/dma-mapping.c |  2 +-
>  include/acpi/acpi_bus.h    |  2 +-
>  include/linux/acpi.h       |  7 +++++--
>  5 files changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 3dd9ec3..e323ece 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  	const struct iommu_ops *ops = NULL;
>  	int ret = -ENODEV;
>  	struct fwnode_handle *iort_fwnode;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> +
> +	/*
> +	 * If we already translated the fwspec there
> +	 * is nothing left to do, return the iommu_ops.
> +	 */
> +	if (fwspec && fwspec->ops)
> +		return fwspec->ops;
>  
>  	if (node) {
>  		iort_fwnode = iort_get_fwnode(node);
> @@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  			return NULL;
>  
>  		ops = iommu_ops_from_fwnode(iort_fwnode);
> +		/*
> +		 * If the ops look-up fails, this means that either
> +		 * the SMMU drivers have not been probed yet or that
> +		 * the SMMU drivers are not built in the kernel;
> +		 * Depending on whether the SMMU drivers are built-in
> +		 * in the kernel or not, defer the IOMMU configuration
> +		 * or just abort it.
> +		 */
>  		if (!ops)
> -			return NULL;
> +			return iort_iommu_driver_enabled(node->type) ?
> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>  
>  		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>  	}
> @@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  
>  		while (parent) {
>  			ops = iort_iommu_xlate(dev, parent, streamid);
> +			if (IS_ERR_OR_NULL(ops))
> +				return ops;
>  
>  			parent = iort_node_get_id(node, &streamid,
>  						  IORT_IOMMU_TYPE, i++);
>  		}
>  	}
>  
> +	/*
> +	 * If we have reason to believe the IOMMU driver missed the initial
> +	 * add_device callback for dev, replay it to get things in order.
> +	 */
> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +	    dev->bus && !dev->iommu_group) {
> +		int err = ops->add_device(dev);
> +
> +		if (err)
> +			ops = ERR_PTR(err);
> +	}
> +
>  	return ops;
>  }
>  
> diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
> index 1926918..2a513cc 100644
> --- a/drivers/acpi/scan.c
> +++ b/drivers/acpi/scan.c
> @@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>   * @dev: The pointer to the device
>   * @attr: device dma attributes
>   */
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>  {
>  	const struct iommu_ops *iommu;
> +	u64 size;
>  
>  	iort_set_dma_mask(dev);
>  
>  	iommu = iort_iommu_configure(dev);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
>  
> +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	/*
>  	 * Assume dma valid range starts at 0 and covers the whole
>  	 * coherent_dma_mask.
>  	 */
> -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
> -			   attr == DEV_DMA_COHERENT);
> +	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dma_configure);
>  
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 82bd45c..755a2b5 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
>  	} else if (has_acpi_companion(dma_dev)) {
>  		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>  		if (attr != DEV_DMA_NOT_SUPPORTED)
> -			acpi_dma_configure(dev, attr);
> +			ret = acpi_dma_configure(dev, attr);
>  	}
>  
>  	if (bridge)
> diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
> index ef0ae8a..2a9a5de 100644
> --- a/include/acpi/acpi_bus.h
> +++ b/include/acpi/acpi_bus.h
> @@ -575,7 +575,7 @@ struct acpi_pci_root {
>  
>  bool acpi_dma_supported(struct acpi_device *adev);
>  enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
>  void acpi_dma_deconfigure(struct device *dev);
>  
>  struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index 9b05886..79d06ef6 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>  	return DEV_DMA_NOT_SUPPORTED;
>  }
>  
> -static inline void acpi_dma_configure(struct device *dev,
> -				      enum dev_dma_attr attr) { }
> +static inline int acpi_dma_configure(struct device *dev,
> +				     enum dev_dma_attr attr)
> +{
> +	return 0;
> +}
>  
>  static inline void acpi_dma_deconfigure(struct device *dev) { }
>  
> 

Re: [PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This is an equivalent to the DT's handling of the iommu master's probe
> with deferred probing when the corrsponding iommu is not probed yet.
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the firmware describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.

I think we really have caught everything now...

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> [Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
>           called multiple times for same device]
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>  drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
>  drivers/acpi/scan.c        | 11 ++++++++---
>  drivers/base/dma-mapping.c |  2 +-
>  include/acpi/acpi_bus.h    |  2 +-
>  include/linux/acpi.h       |  7 +++++--
>  5 files changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 3dd9ec3..e323ece 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  	const struct iommu_ops *ops = NULL;
>  	int ret = -ENODEV;
>  	struct fwnode_handle *iort_fwnode;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> +
> +	/*
> +	 * If we already translated the fwspec there
> +	 * is nothing left to do, return the iommu_ops.
> +	 */
> +	if (fwspec && fwspec->ops)
> +		return fwspec->ops;
>  
>  	if (node) {
>  		iort_fwnode = iort_get_fwnode(node);
> @@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  			return NULL;
>  
>  		ops = iommu_ops_from_fwnode(iort_fwnode);
> +		/*
> +		 * If the ops look-up fails, this means that either
> +		 * the SMMU drivers have not been probed yet or that
> +		 * the SMMU drivers are not built in the kernel;
> +		 * Depending on whether the SMMU drivers are built-in
> +		 * in the kernel or not, defer the IOMMU configuration
> +		 * or just abort it.
> +		 */
>  		if (!ops)
> -			return NULL;
> +			return iort_iommu_driver_enabled(node->type) ?
> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>  
>  		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>  	}
> @@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  
>  		while (parent) {
>  			ops = iort_iommu_xlate(dev, parent, streamid);
> +			if (IS_ERR_OR_NULL(ops))
> +				return ops;
>  
>  			parent = iort_node_get_id(node, &streamid,
>  						  IORT_IOMMU_TYPE, i++);
>  		}
>  	}
>  
> +	/*
> +	 * If we have reason to believe the IOMMU driver missed the initial
> +	 * add_device callback for dev, replay it to get things in order.
> +	 */
> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +	    dev->bus && !dev->iommu_group) {
> +		int err = ops->add_device(dev);
> +
> +		if (err)
> +			ops = ERR_PTR(err);
> +	}
> +
>  	return ops;
>  }
>  
> diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
> index 1926918..2a513cc 100644
> --- a/drivers/acpi/scan.c
> +++ b/drivers/acpi/scan.c
> @@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>   * @dev: The pointer to the device
>   * @attr: device dma attributes
>   */
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>  {
>  	const struct iommu_ops *iommu;
> +	u64 size;
>  
>  	iort_set_dma_mask(dev);
>  
>  	iommu = iort_iommu_configure(dev);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
>  
> +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	/*
>  	 * Assume dma valid range starts at 0 and covers the whole
>  	 * coherent_dma_mask.
>  	 */
> -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
> -			   attr == DEV_DMA_COHERENT);
> +	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dma_configure);
>  
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 82bd45c..755a2b5 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
>  	} else if (has_acpi_companion(dma_dev)) {
>  		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>  		if (attr != DEV_DMA_NOT_SUPPORTED)
> -			acpi_dma_configure(dev, attr);
> +			ret = acpi_dma_configure(dev, attr);
>  	}
>  
>  	if (bridge)
> diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
> index ef0ae8a..2a9a5de 100644
> --- a/include/acpi/acpi_bus.h
> +++ b/include/acpi/acpi_bus.h
> @@ -575,7 +575,7 @@ struct acpi_pci_root {
>  
>  bool acpi_dma_supported(struct acpi_device *adev);
>  enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
>  void acpi_dma_deconfigure(struct device *dev);
>  
>  struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index 9b05886..79d06ef6 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>  	return DEV_DMA_NOT_SUPPORTED;
>  }
>  
> -static inline void acpi_dma_configure(struct device *dev,
> -				      enum dev_dma_attr attr) { }
> +static inline int acpi_dma_configure(struct device *dev,
> +				     enum dev_dma_attr attr)
> +{
> +	return 0;
> +}
>  
>  static inline void acpi_dma_deconfigure(struct device *dev) { }
>  
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 09/12] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This is an equivalent to the DT's handling of the iommu master's probe
> with deferred probing when the corrsponding iommu is not probed yet.
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the firmware describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.

I think we really have caught everything now...

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> [Lorenzo: Added fixes for dma_coherent_mask overflow, acpi_dma_configure
>           called multiple times for same device]
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>  drivers/acpi/arm64/iort.c  | 33 ++++++++++++++++++++++++++++++++-
>  drivers/acpi/scan.c        | 11 ++++++++---
>  drivers/base/dma-mapping.c |  2 +-
>  include/acpi/acpi_bus.h    |  2 +-
>  include/linux/acpi.h       |  7 +++++--
>  5 files changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 3dd9ec3..e323ece 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  	const struct iommu_ops *ops = NULL;
>  	int ret = -ENODEV;
>  	struct fwnode_handle *iort_fwnode;
> +	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> +
> +	/*
> +	 * If we already translated the fwspec there
> +	 * is nothing left to do, return the iommu_ops.
> +	 */
> +	if (fwspec && fwspec->ops)
> +		return fwspec->ops;
>  
>  	if (node) {
>  		iort_fwnode = iort_get_fwnode(node);
> @@ -550,8 +558,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>  			return NULL;
>  
>  		ops = iommu_ops_from_fwnode(iort_fwnode);
> +		/*
> +		 * If the ops look-up fails, this means that either
> +		 * the SMMU drivers have not been probed yet or that
> +		 * the SMMU drivers are not built in the kernel;
> +		 * Depending on whether the SMMU drivers are built-in
> +		 * in the kernel or not, defer the IOMMU configuration
> +		 * or just abort it.
> +		 */
>  		if (!ops)
> -			return NULL;
> +			return iort_iommu_driver_enabled(node->type) ?
> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>  
>  		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>  	}
> @@ -625,12 +642,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  
>  		while (parent) {
>  			ops = iort_iommu_xlate(dev, parent, streamid);
> +			if (IS_ERR_OR_NULL(ops))
> +				return ops;
>  
>  			parent = iort_node_get_id(node, &streamid,
>  						  IORT_IOMMU_TYPE, i++);
>  		}
>  	}
>  
> +	/*
> +	 * If we have reason to believe the IOMMU driver missed the initial
> +	 * add_device callback for dev, replay it to get things in order.
> +	 */
> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +	    dev->bus && !dev->iommu_group) {
> +		int err = ops->add_device(dev);
> +
> +		if (err)
> +			ops = ERR_PTR(err);
> +	}
> +
>  	return ops;
>  }
>  
> diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
> index 1926918..2a513cc 100644
> --- a/drivers/acpi/scan.c
> +++ b/drivers/acpi/scan.c
> @@ -1373,20 +1373,25 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>   * @dev: The pointer to the device
>   * @attr: device dma attributes
>   */
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
>  {
>  	const struct iommu_ops *iommu;
> +	u64 size;
>  
>  	iort_set_dma_mask(dev);
>  
>  	iommu = iort_iommu_configure(dev);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
>  
> +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	/*
>  	 * Assume dma valid range starts at 0 and covers the whole
>  	 * coherent_dma_mask.
>  	 */
> -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
> -			   attr == DEV_DMA_COHERENT);
> +	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dma_configure);
>  
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 82bd45c..755a2b5 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
>  	} else if (has_acpi_companion(dma_dev)) {
>  		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>  		if (attr != DEV_DMA_NOT_SUPPORTED)
> -			acpi_dma_configure(dev, attr);
> +			ret = acpi_dma_configure(dev, attr);
>  	}
>  
>  	if (bridge)
> diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
> index ef0ae8a..2a9a5de 100644
> --- a/include/acpi/acpi_bus.h
> +++ b/include/acpi/acpi_bus.h
> @@ -575,7 +575,7 @@ struct acpi_pci_root {
>  
>  bool acpi_dma_supported(struct acpi_device *adev);
>  enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
> -void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
> +int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
>  void acpi_dma_deconfigure(struct device *dev);
>  
>  struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index 9b05886..79d06ef6 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -762,8 +762,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
>  	return DEV_DMA_NOT_SUPPORTED;
>  }
>  
> -static inline void acpi_dma_configure(struct device *dev,
> -				      enum dev_dma_attr attr) { }
> +static inline int acpi_dma_configure(struct device *dev,
> +				     enum dev_dma_attr attr)
> +{
> +	return 0;
> +}
>  
>  static inline void acpi_dma_deconfigure(struct device *dev) { }
>  
> 

Re: [PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IORT linker section introduced by commit 34ceea275f62
> ("ACPI/IORT: Introduce linker section for IORT entries probing")
> was needed to make sure SMMU drivers are registered (and therefore
> probed) in the kernel before devices using the SMMU have a chance
> to probe in turn.
> 
> Through the introduction of deferred IOMMU configuration the linker
> section based IORT probing infrastructure is not needed any longer, in
> that device/SMMU probe dependencies are managed through the probe
> deferral mechanism, making the IORT linker section infrastructure
> unused, so that it can be removed.
> 
> Remove the unused IORT linker section probing infrastructure
> from the kernel to complete the ACPI IORT IOMMU configure probe
> deferral mechanism implementation.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c         | 2 --
>  include/asm-generic/vmlinux.lds.h | 1 -
>  include/linux/acpi_iort.h         | 3 ---
>  3 files changed, 6 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index e323ece..e7b1940 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
>  	}
>  
>  	iort_init_platform_devices();
> -
> -	acpi_probe_device_table(iort);
>  }
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 0968d13..9faa26c 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -566,7 +566,6 @@
>  	IRQCHIP_OF_MATCH_TABLE()					\
>  	ACPI_PROBE_TABLE(irqchip)					\
>  	ACPI_PROBE_TABLE(clksrc)					\
> -	ACPI_PROBE_TABLE(iort)						\
>  	EARLYCON_TABLE()
>  
>  #define INIT_TEXT							\
> diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
> index 77e0809..f167e1d04 100644
> --- a/include/linux/acpi_iort.h
> +++ b/include/linux/acpi_iort.h
> @@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  { return NULL; }
>  #endif
>  
> -#define IORT_ACPI_DECLARE(name, table_id, fn)		\
> -	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
> -
>  #endif /* __ACPI_IORT_H__ */
> 

Re: [PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IORT linker section introduced by commit 34ceea275f62
> ("ACPI/IORT: Introduce linker section for IORT entries probing")
> was needed to make sure SMMU drivers are registered (and therefore
> probed) in the kernel before devices using the SMMU have a chance
> to probe in turn.
> 
> Through the introduction of deferred IOMMU configuration the linker
> section based IORT probing infrastructure is not needed any longer, in
> that device/SMMU probe dependencies are managed through the probe
> deferral mechanism, making the IORT linker section infrastructure
> unused, so that it can be removed.
> 
> Remove the unused IORT linker section probing infrastructure
> from the kernel to complete the ACPI IORT IOMMU configure probe
> deferral mechanism implementation.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c         | 2 --
>  include/asm-generic/vmlinux.lds.h | 1 -
>  include/linux/acpi_iort.h         | 3 ---
>  3 files changed, 6 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index e323ece..e7b1940 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
>  	}
>  
>  	iort_init_platform_devices();
> -
> -	acpi_probe_device_table(iort);
>  }
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 0968d13..9faa26c 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -566,7 +566,6 @@
>  	IRQCHIP_OF_MATCH_TABLE()					\
>  	ACPI_PROBE_TABLE(irqchip)					\
>  	ACPI_PROBE_TABLE(clksrc)					\
> -	ACPI_PROBE_TABLE(iort)						\
>  	EARLYCON_TABLE()
>  
>  #define INIT_TEXT							\
> diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
> index 77e0809..f167e1d04 100644
> --- a/include/linux/acpi_iort.h
> +++ b/include/linux/acpi_iort.h
> @@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  { return NULL; }
>  #endif
>  
> -#define IORT_ACPI_DECLARE(name, table_id, fn)		\
> -	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
> -
>  #endif /* __ACPI_IORT_H__ */
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 12/12] ACPI/IORT: Remove linker section for IORT entries probing

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> 
> The IORT linker section introduced by commit 34ceea275f62
> ("ACPI/IORT: Introduce linker section for IORT entries probing")
> was needed to make sure SMMU drivers are registered (and therefore
> probed) in the kernel before devices using the SMMU have a chance
> to probe in turn.
> 
> Through the introduction of deferred IOMMU configuration the linker
> section based IORT probing infrastructure is not needed any longer, in
> that device/SMMU probe dependencies are managed through the probe
> deferral mechanism, making the IORT linker section infrastructure
> unused, so that it can be removed.
> 
> Remove the unused IORT linker section probing infrastructure
> from the kernel to complete the ACPI IORT IOMMU configure probe
> deferral mechanism implementation.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c         | 2 --
>  include/asm-generic/vmlinux.lds.h | 1 -
>  include/linux/acpi_iort.h         | 3 ---
>  3 files changed, 6 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index e323ece..e7b1940 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -1000,6 +1000,4 @@ void __init acpi_iort_init(void)
>  	}
>  
>  	iort_init_platform_devices();
> -
> -	acpi_probe_device_table(iort);
>  }
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 0968d13..9faa26c 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -566,7 +566,6 @@
>  	IRQCHIP_OF_MATCH_TABLE()					\
>  	ACPI_PROBE_TABLE(irqchip)					\
>  	ACPI_PROBE_TABLE(clksrc)					\
> -	ACPI_PROBE_TABLE(iort)						\
>  	EARLYCON_TABLE()
>  
>  #define INIT_TEXT							\
> diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
> index 77e0809..f167e1d04 100644
> --- a/include/linux/acpi_iort.h
> +++ b/include/linux/acpi_iort.h
> @@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  { return NULL; }
>  #endif
>  
> -#define IORT_ACPI_DECLARE(name, table_id, fn)		\
> -	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
> -
>  #endif /* __ACPI_IORT_H__ */
> 

Re: [PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Configuring DMA ops at probe time will allow deferring device probe when
> the IOMMU isn't available yet. The dma_configure for the device is
> now called from the generic device_attach callback just before the
> bus/driver probe is called. This way, configuring the DMA ops for the
> device would be called at the same place for all bus_types, hence the
> deferred probing mechanism should work for all buses as well.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> This patch changes the dma ops configuration to probe time for
> both OF and ACPI based platform/amba/pci bus devices.

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

It's possible we could subsume {of,acpi}_dma_deconfigure() into
dma_deconfigure() entirely in future if it becomes clear that neither of
them will ever need to do anything firmware-specific, but I think for
now it's probably safer to keep the current symmetry - calling
arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
it really should be!)

Robin.

> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
> Acked-by: Bjorn Helgaas <bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> (drivers/pci part)
> Acked-by: Rafael J. Wysocki <rafael.j.wysocki-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
> 
>  [V10] Added dummy dma_(de)configure functions in case
>        of !CONFIG_HAS_DMA to avoid build breaks.
> 
>  drivers/acpi/glue.c         |  5 -----
>  drivers/base/dd.c           |  9 +++++++++
>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>  drivers/of/platform.c       |  5 +----
>  drivers/pci/probe.c         | 28 ----------------------------
>  include/linux/dma-mapping.h | 12 ++++++++++++
>  6 files changed, 62 insertions(+), 37 deletions(-)
> 
> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
> index fb19e1c..c05f241 100644
> --- a/drivers/acpi/glue.c
> +++ b/drivers/acpi/glue.c
> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	struct list_head *physnode_list;
>  	unsigned int node_id;
>  	int retval = -EINVAL;
> -	enum dev_dma_attr attr;
>  
>  	if (has_acpi_companion(dev)) {
>  		if (acpi_dev) {
> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	if (!has_acpi_companion(dev))
>  		ACPI_COMPANION_SET(dev, acpi_dev);
>  
> -	attr = acpi_get_dma_attr(acpi_dev);
> -	if (attr != DEV_DMA_NOT_SUPPORTED)
> -		acpi_dma_configure(dev, attr);
> -
>  	acpi_physnode_link_name(physical_node_name, node_id);
>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>  				   physical_node_name);
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index a1fbf55..4882f06 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -19,6 +19,7 @@
>  
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/kthread.h>
>  #include <linux/wait.h>
> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	if (ret)
>  		goto pinctrl_bind_failed;
>  
> +	ret = dma_configure(dev);
> +	if (ret)
> +		goto dma_failed;
> +
>  	if (driver_sysfs_add(dev)) {
>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>  			__func__, dev_name(dev));
> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	goto done;
>  
>  probe_failed:
> +	dma_deconfigure(dev);
> +dma_failed:
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>  			drv->remove(dev);
>  
>  		device_links_driver_cleanup(dev);
> +		dma_deconfigure(dev);
> +
>  		devres_release_all(dev);
>  		dev->driver = NULL;
>  		dev_set_drvdata(dev, NULL);
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index efd71cf..449b948 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -7,9 +7,11 @@
>   * This file is released under the GPLv2.
>   */
>  
> +#include <linux/acpi.h>
>  #include <linux/dma-mapping.h>
>  #include <linux/export.h>
>  #include <linux/gfp.h>
> +#include <linux/of_device.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>  	vunmap(cpu_addr);
>  }
>  #endif
> +
> +/*
> + * Common configuration to enable DMA API use for a device
> + */
> +#include <linux/pci.h>
> +
> +int dma_configure(struct device *dev)
> +{
> +	struct device *bridge = NULL, *dma_dev = dev;
> +	enum dev_dma_attr attr;
> +
> +	if (dev_is_pci(dev)) {
> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> +		dma_dev = bridge;
> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
> +		    dma_dev->parent->of_node)
> +			dma_dev = dma_dev->parent;
> +	}
> +
> +	if (dma_dev->of_node) {
> +		of_dma_configure(dev, dma_dev->of_node);
> +	} else if (has_acpi_companion(dma_dev)) {
> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
> +		if (attr != DEV_DMA_NOT_SUPPORTED)
> +			acpi_dma_configure(dev, attr);
> +	}
> +
> +	if (bridge)
> +		pci_put_host_bridge_device(bridge);
> +
> +	return 0;
> +}
> +
> +void dma_deconfigure(struct device *dev)
> +{
> +	of_dma_deconfigure(dev);
> +	acpi_dma_deconfigure(dev);
> +}
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5344db5..2aa4ebb 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -22,6 +22,7 @@
>  #include <linux/slab.h>
>  #include <linux/of_address.h>
>  #include <linux/of_device.h>
> +#include <linux/of_iommu.h>
>  #include <linux/of_irq.h>
>  #include <linux/of_platform.h>
>  #include <linux/platform_device.h>
> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>  
>  	dev->dev.bus = &platform_bus_type;
>  	dev->dev.platform_data = platform_data;
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>  
>  	if (of_device_add(dev) != 0) {
> -		of_dma_deconfigure(&dev->dev);
>  		platform_device_put(dev);
>  		goto err_clear_flag;
>  	}
> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>  		dev_set_name(&dev->dev, "%s", bus_id);
>  	else
>  		of_device_make_bus_id(&dev->dev);
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  
>  	/* Allow the HW Peripheral ID to be overridden */
>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>  		amba_device_unregister(to_amba_device(dev));
>  #endif
>  
> -	of_dma_deconfigure(dev);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>  	return 0;
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index dfc9a27..5a8dd43 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>  	dev_set_msi_domain(&dev->dev, d);
>  }
>  
> -/**
> - * pci_dma_configure - Setup DMA configuration
> - * @dev: ptr to pci_dev struct of the PCI device
> - *
> - * Function to update PCI devices's DMA configuration using the same
> - * info from the OF node or ACPI node of host bridge's parent (if any).
> - */
> -static void pci_dma_configure(struct pci_dev *dev)
> -{
> -	struct device *bridge = pci_get_host_bridge_device(dev);
> -
> -	if (IS_ENABLED(CONFIG_OF) &&
> -		bridge->parent && bridge->parent->of_node) {
> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> -	} else if (has_acpi_companion(bridge)) {
> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
> -
> -		if (attr == DEV_DMA_NOT_SUPPORTED)
> -			dev_warn(&dev->dev, "DMA not supported.\n");
> -		else
> -			acpi_dma_configure(&dev->dev, attr);
> -	}
> -
> -	pci_put_host_bridge_device(bridge);
> -}
> -
>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  {
>  	int ret;
> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  	dev->dev.dma_mask = &dev->dma_mask;
>  	dev->dev.dma_parms = &dev->dma_parms;
>  	dev->dev.coherent_dma_mask = 0xffffffffull;
> -	pci_dma_configure(dev);
>  
>  	pci_set_dma_max_seg_size(dev, 65536);
>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 0977317..4f3eece 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>  }
>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>  
> +#ifdef CONFIG_HAS_DMA
> +int dma_configure(struct device *dev);
> +void dma_deconfigure(struct device *dev);
> +#else
> +static inline int dma_configure(struct device *dev)
> +{
> +	return 0;
> +}
> +
> +static inline void dma_deconfigure(struct device *dev) {}
> +#endif
> +
>  /*
>   * Managed DMA API
>   */
> 

Re: [PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Configuring DMA ops at probe time will allow deferring device probe when
> the IOMMU isn't available yet. The dma_configure for the device is
> now called from the generic device_attach callback just before the
> bus/driver probe is called. This way, configuring the DMA ops for the
> device would be called at the same place for all bus_types, hence the
> deferred probing mechanism should work for all buses as well.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> This patch changes the dma ops configuration to probe time for
> both OF and ACPI based platform/amba/pci bus devices.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

It's possible we could subsume {of,acpi}_dma_deconfigure() into
dma_deconfigure() entirely in future if it becomes clear that neither of
them will ever need to do anything firmware-specific, but I think for
now it's probably safer to keep the current symmetry - calling
arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
it really should be!)

Robin.

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added dummy dma_(de)configure functions in case
>        of !CONFIG_HAS_DMA to avoid build breaks.
> 
>  drivers/acpi/glue.c         |  5 -----
>  drivers/base/dd.c           |  9 +++++++++
>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>  drivers/of/platform.c       |  5 +----
>  drivers/pci/probe.c         | 28 ----------------------------
>  include/linux/dma-mapping.h | 12 ++++++++++++
>  6 files changed, 62 insertions(+), 37 deletions(-)
> 
> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
> index fb19e1c..c05f241 100644
> --- a/drivers/acpi/glue.c
> +++ b/drivers/acpi/glue.c
> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	struct list_head *physnode_list;
>  	unsigned int node_id;
>  	int retval = -EINVAL;
> -	enum dev_dma_attr attr;
>  
>  	if (has_acpi_companion(dev)) {
>  		if (acpi_dev) {
> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	if (!has_acpi_companion(dev))
>  		ACPI_COMPANION_SET(dev, acpi_dev);
>  
> -	attr = acpi_get_dma_attr(acpi_dev);
> -	if (attr != DEV_DMA_NOT_SUPPORTED)
> -		acpi_dma_configure(dev, attr);
> -
>  	acpi_physnode_link_name(physical_node_name, node_id);
>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>  				   physical_node_name);
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index a1fbf55..4882f06 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -19,6 +19,7 @@
>  
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/kthread.h>
>  #include <linux/wait.h>
> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	if (ret)
>  		goto pinctrl_bind_failed;
>  
> +	ret = dma_configure(dev);
> +	if (ret)
> +		goto dma_failed;
> +
>  	if (driver_sysfs_add(dev)) {
>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>  			__func__, dev_name(dev));
> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	goto done;
>  
>  probe_failed:
> +	dma_deconfigure(dev);
> +dma_failed:
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>  			drv->remove(dev);
>  
>  		device_links_driver_cleanup(dev);
> +		dma_deconfigure(dev);
> +
>  		devres_release_all(dev);
>  		dev->driver = NULL;
>  		dev_set_drvdata(dev, NULL);
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index efd71cf..449b948 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -7,9 +7,11 @@
>   * This file is released under the GPLv2.
>   */
>  
> +#include <linux/acpi.h>
>  #include <linux/dma-mapping.h>
>  #include <linux/export.h>
>  #include <linux/gfp.h>
> +#include <linux/of_device.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>  	vunmap(cpu_addr);
>  }
>  #endif
> +
> +/*
> + * Common configuration to enable DMA API use for a device
> + */
> +#include <linux/pci.h>
> +
> +int dma_configure(struct device *dev)
> +{
> +	struct device *bridge = NULL, *dma_dev = dev;
> +	enum dev_dma_attr attr;
> +
> +	if (dev_is_pci(dev)) {
> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> +		dma_dev = bridge;
> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
> +		    dma_dev->parent->of_node)
> +			dma_dev = dma_dev->parent;
> +	}
> +
> +	if (dma_dev->of_node) {
> +		of_dma_configure(dev, dma_dev->of_node);
> +	} else if (has_acpi_companion(dma_dev)) {
> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
> +		if (attr != DEV_DMA_NOT_SUPPORTED)
> +			acpi_dma_configure(dev, attr);
> +	}
> +
> +	if (bridge)
> +		pci_put_host_bridge_device(bridge);
> +
> +	return 0;
> +}
> +
> +void dma_deconfigure(struct device *dev)
> +{
> +	of_dma_deconfigure(dev);
> +	acpi_dma_deconfigure(dev);
> +}
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5344db5..2aa4ebb 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -22,6 +22,7 @@
>  #include <linux/slab.h>
>  #include <linux/of_address.h>
>  #include <linux/of_device.h>
> +#include <linux/of_iommu.h>
>  #include <linux/of_irq.h>
>  #include <linux/of_platform.h>
>  #include <linux/platform_device.h>
> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>  
>  	dev->dev.bus = &platform_bus_type;
>  	dev->dev.platform_data = platform_data;
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>  
>  	if (of_device_add(dev) != 0) {
> -		of_dma_deconfigure(&dev->dev);
>  		platform_device_put(dev);
>  		goto err_clear_flag;
>  	}
> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>  		dev_set_name(&dev->dev, "%s", bus_id);
>  	else
>  		of_device_make_bus_id(&dev->dev);
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  
>  	/* Allow the HW Peripheral ID to be overridden */
>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>  		amba_device_unregister(to_amba_device(dev));
>  #endif
>  
> -	of_dma_deconfigure(dev);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>  	return 0;
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index dfc9a27..5a8dd43 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>  	dev_set_msi_domain(&dev->dev, d);
>  }
>  
> -/**
> - * pci_dma_configure - Setup DMA configuration
> - * @dev: ptr to pci_dev struct of the PCI device
> - *
> - * Function to update PCI devices's DMA configuration using the same
> - * info from the OF node or ACPI node of host bridge's parent (if any).
> - */
> -static void pci_dma_configure(struct pci_dev *dev)
> -{
> -	struct device *bridge = pci_get_host_bridge_device(dev);
> -
> -	if (IS_ENABLED(CONFIG_OF) &&
> -		bridge->parent && bridge->parent->of_node) {
> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> -	} else if (has_acpi_companion(bridge)) {
> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
> -
> -		if (attr == DEV_DMA_NOT_SUPPORTED)
> -			dev_warn(&dev->dev, "DMA not supported.\n");
> -		else
> -			acpi_dma_configure(&dev->dev, attr);
> -	}
> -
> -	pci_put_host_bridge_device(bridge);
> -}
> -
>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  {
>  	int ret;
> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  	dev->dev.dma_mask = &dev->dma_mask;
>  	dev->dev.dma_parms = &dev->dma_parms;
>  	dev->dev.coherent_dma_mask = 0xffffffffull;
> -	pci_dma_configure(dev);
>  
>  	pci_set_dma_max_seg_size(dev, 65536);
>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 0977317..4f3eece 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>  }
>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>  
> +#ifdef CONFIG_HAS_DMA
> +int dma_configure(struct device *dev);
> +void dma_deconfigure(struct device *dev);
> +#else
> +static inline int dma_configure(struct device *dev)
> +{
> +	return 0;
> +}
> +
> +static inline void dma_deconfigure(struct device *dev) {}
> +#endif
> +
>  /*
>   * Managed DMA API
>   */
> 

Re: [PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Configuring DMA ops at probe time will allow deferring device probe when
> the IOMMU isn't available yet. The dma_configure for the device is
> now called from the generic device_attach callback just before the
> bus/driver probe is called. This way, configuring the DMA ops for the
> device would be called at the same place for all bus_types, hence the
> deferred probing mechanism should work for all buses as well.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> This patch changes the dma ops configuration to probe time for
> both OF and ACPI based platform/amba/pci bus devices.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

It's possible we could subsume {of,acpi}_dma_deconfigure() into
dma_deconfigure() entirely in future if it becomes clear that neither of
them will ever need to do anything firmware-specific, but I think for
now it's probably safer to keep the current symmetry - calling
arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
it really should be!)

Robin.

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added dummy dma_(de)configure functions in case
>        of !CONFIG_HAS_DMA to avoid build breaks.
> 
>  drivers/acpi/glue.c         |  5 -----
>  drivers/base/dd.c           |  9 +++++++++
>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>  drivers/of/platform.c       |  5 +----
>  drivers/pci/probe.c         | 28 ----------------------------
>  include/linux/dma-mapping.h | 12 ++++++++++++
>  6 files changed, 62 insertions(+), 37 deletions(-)
> 
> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
> index fb19e1c..c05f241 100644
> --- a/drivers/acpi/glue.c
> +++ b/drivers/acpi/glue.c
> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	struct list_head *physnode_list;
>  	unsigned int node_id;
>  	int retval = -EINVAL;
> -	enum dev_dma_attr attr;
>  
>  	if (has_acpi_companion(dev)) {
>  		if (acpi_dev) {
> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	if (!has_acpi_companion(dev))
>  		ACPI_COMPANION_SET(dev, acpi_dev);
>  
> -	attr = acpi_get_dma_attr(acpi_dev);
> -	if (attr != DEV_DMA_NOT_SUPPORTED)
> -		acpi_dma_configure(dev, attr);
> -
>  	acpi_physnode_link_name(physical_node_name, node_id);
>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>  				   physical_node_name);
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index a1fbf55..4882f06 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -19,6 +19,7 @@
>  
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/kthread.h>
>  #include <linux/wait.h>
> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	if (ret)
>  		goto pinctrl_bind_failed;
>  
> +	ret = dma_configure(dev);
> +	if (ret)
> +		goto dma_failed;
> +
>  	if (driver_sysfs_add(dev)) {
>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>  			__func__, dev_name(dev));
> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	goto done;
>  
>  probe_failed:
> +	dma_deconfigure(dev);
> +dma_failed:
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>  			drv->remove(dev);
>  
>  		device_links_driver_cleanup(dev);
> +		dma_deconfigure(dev);
> +
>  		devres_release_all(dev);
>  		dev->driver = NULL;
>  		dev_set_drvdata(dev, NULL);
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index efd71cf..449b948 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -7,9 +7,11 @@
>   * This file is released under the GPLv2.
>   */
>  
> +#include <linux/acpi.h>
>  #include <linux/dma-mapping.h>
>  #include <linux/export.h>
>  #include <linux/gfp.h>
> +#include <linux/of_device.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>  	vunmap(cpu_addr);
>  }
>  #endif
> +
> +/*
> + * Common configuration to enable DMA API use for a device
> + */
> +#include <linux/pci.h>
> +
> +int dma_configure(struct device *dev)
> +{
> +	struct device *bridge = NULL, *dma_dev = dev;
> +	enum dev_dma_attr attr;
> +
> +	if (dev_is_pci(dev)) {
> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> +		dma_dev = bridge;
> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
> +		    dma_dev->parent->of_node)
> +			dma_dev = dma_dev->parent;
> +	}
> +
> +	if (dma_dev->of_node) {
> +		of_dma_configure(dev, dma_dev->of_node);
> +	} else if (has_acpi_companion(dma_dev)) {
> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
> +		if (attr != DEV_DMA_NOT_SUPPORTED)
> +			acpi_dma_configure(dev, attr);
> +	}
> +
> +	if (bridge)
> +		pci_put_host_bridge_device(bridge);
> +
> +	return 0;
> +}
> +
> +void dma_deconfigure(struct device *dev)
> +{
> +	of_dma_deconfigure(dev);
> +	acpi_dma_deconfigure(dev);
> +}
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5344db5..2aa4ebb 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -22,6 +22,7 @@
>  #include <linux/slab.h>
>  #include <linux/of_address.h>
>  #include <linux/of_device.h>
> +#include <linux/of_iommu.h>
>  #include <linux/of_irq.h>
>  #include <linux/of_platform.h>
>  #include <linux/platform_device.h>
> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>  
>  	dev->dev.bus = &platform_bus_type;
>  	dev->dev.platform_data = platform_data;
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>  
>  	if (of_device_add(dev) != 0) {
> -		of_dma_deconfigure(&dev->dev);
>  		platform_device_put(dev);
>  		goto err_clear_flag;
>  	}
> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>  		dev_set_name(&dev->dev, "%s", bus_id);
>  	else
>  		of_device_make_bus_id(&dev->dev);
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  
>  	/* Allow the HW Peripheral ID to be overridden */
>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>  		amba_device_unregister(to_amba_device(dev));
>  #endif
>  
> -	of_dma_deconfigure(dev);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>  	return 0;
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index dfc9a27..5a8dd43 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>  	dev_set_msi_domain(&dev->dev, d);
>  }
>  
> -/**
> - * pci_dma_configure - Setup DMA configuration
> - * @dev: ptr to pci_dev struct of the PCI device
> - *
> - * Function to update PCI devices's DMA configuration using the same
> - * info from the OF node or ACPI node of host bridge's parent (if any).
> - */
> -static void pci_dma_configure(struct pci_dev *dev)
> -{
> -	struct device *bridge = pci_get_host_bridge_device(dev);
> -
> -	if (IS_ENABLED(CONFIG_OF) &&
> -		bridge->parent && bridge->parent->of_node) {
> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> -	} else if (has_acpi_companion(bridge)) {
> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
> -
> -		if (attr == DEV_DMA_NOT_SUPPORTED)
> -			dev_warn(&dev->dev, "DMA not supported.\n");
> -		else
> -			acpi_dma_configure(&dev->dev, attr);
> -	}
> -
> -	pci_put_host_bridge_device(bridge);
> -}
> -
>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  {
>  	int ret;
> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  	dev->dev.dma_mask = &dev->dma_mask;
>  	dev->dev.dma_parms = &dev->dma_parms;
>  	dev->dev.coherent_dma_mask = 0xffffffffull;
> -	pci_dma_configure(dev);
>  
>  	pci_set_dma_max_seg_size(dev, 65536);
>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 0977317..4f3eece 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>  }
>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>  
> +#ifdef CONFIG_HAS_DMA
> +int dma_configure(struct device *dev);
> +void dma_deconfigure(struct device *dev);
> +#else
> +static inline int dma_configure(struct device *dev)
> +{
> +	return 0;
> +}
> +
> +static inline void dma_deconfigure(struct device *dev) {}
> +#endif
> +
>  /*
>   * Managed DMA API
>   */
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> Configuring DMA ops at probe time will allow deferring device probe when
> the IOMMU isn't available yet. The dma_configure for the device is
> now called from the generic device_attach callback just before the
> bus/driver probe is called. This way, configuring the DMA ops for the
> device would be called at the same place for all bus_types, hence the
> deferred probing mechanism should work for all buses as well.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> This patch changes the dma ops configuration to probe time for
> both OF and ACPI based platform/amba/pci bus devices.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

It's possible we could subsume {of,acpi}_dma_deconfigure() into
dma_deconfigure() entirely in future if it becomes clear that neither of
them will ever need to do anything firmware-specific, but I think for
now it's probably safer to keep the current symmetry - calling
arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
it really should be!)

Robin.

> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
> 
>  [V10] Added dummy dma_(de)configure functions in case
>        of !CONFIG_HAS_DMA to avoid build breaks.
> 
>  drivers/acpi/glue.c         |  5 -----
>  drivers/base/dd.c           |  9 +++++++++
>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>  drivers/of/platform.c       |  5 +----
>  drivers/pci/probe.c         | 28 ----------------------------
>  include/linux/dma-mapping.h | 12 ++++++++++++
>  6 files changed, 62 insertions(+), 37 deletions(-)
> 
> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
> index fb19e1c..c05f241 100644
> --- a/drivers/acpi/glue.c
> +++ b/drivers/acpi/glue.c
> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	struct list_head *physnode_list;
>  	unsigned int node_id;
>  	int retval = -EINVAL;
> -	enum dev_dma_attr attr;
>  
>  	if (has_acpi_companion(dev)) {
>  		if (acpi_dev) {
> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>  	if (!has_acpi_companion(dev))
>  		ACPI_COMPANION_SET(dev, acpi_dev);
>  
> -	attr = acpi_get_dma_attr(acpi_dev);
> -	if (attr != DEV_DMA_NOT_SUPPORTED)
> -		acpi_dma_configure(dev, attr);
> -
>  	acpi_physnode_link_name(physical_node_name, node_id);
>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>  				   physical_node_name);
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index a1fbf55..4882f06 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -19,6 +19,7 @@
>  
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/kthread.h>
>  #include <linux/wait.h>
> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	if (ret)
>  		goto pinctrl_bind_failed;
>  
> +	ret = dma_configure(dev);
> +	if (ret)
> +		goto dma_failed;
> +
>  	if (driver_sysfs_add(dev)) {
>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>  			__func__, dev_name(dev));
> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	goto done;
>  
>  probe_failed:
> +	dma_deconfigure(dev);
> +dma_failed:
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>  			drv->remove(dev);
>  
>  		device_links_driver_cleanup(dev);
> +		dma_deconfigure(dev);
> +
>  		devres_release_all(dev);
>  		dev->driver = NULL;
>  		dev_set_drvdata(dev, NULL);
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index efd71cf..449b948 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -7,9 +7,11 @@
>   * This file is released under the GPLv2.
>   */
>  
> +#include <linux/acpi.h>
>  #include <linux/dma-mapping.h>
>  #include <linux/export.h>
>  #include <linux/gfp.h>
> +#include <linux/of_device.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>  	vunmap(cpu_addr);
>  }
>  #endif
> +
> +/*
> + * Common configuration to enable DMA API use for a device
> + */
> +#include <linux/pci.h>
> +
> +int dma_configure(struct device *dev)
> +{
> +	struct device *bridge = NULL, *dma_dev = dev;
> +	enum dev_dma_attr attr;
> +
> +	if (dev_is_pci(dev)) {
> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> +		dma_dev = bridge;
> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
> +		    dma_dev->parent->of_node)
> +			dma_dev = dma_dev->parent;
> +	}
> +
> +	if (dma_dev->of_node) {
> +		of_dma_configure(dev, dma_dev->of_node);
> +	} else if (has_acpi_companion(dma_dev)) {
> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
> +		if (attr != DEV_DMA_NOT_SUPPORTED)
> +			acpi_dma_configure(dev, attr);
> +	}
> +
> +	if (bridge)
> +		pci_put_host_bridge_device(bridge);
> +
> +	return 0;
> +}
> +
> +void dma_deconfigure(struct device *dev)
> +{
> +	of_dma_deconfigure(dev);
> +	acpi_dma_deconfigure(dev);
> +}
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 5344db5..2aa4ebb 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -22,6 +22,7 @@
>  #include <linux/slab.h>
>  #include <linux/of_address.h>
>  #include <linux/of_device.h>
> +#include <linux/of_iommu.h>
>  #include <linux/of_irq.h>
>  #include <linux/of_platform.h>
>  #include <linux/platform_device.h>
> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>  
>  	dev->dev.bus = &platform_bus_type;
>  	dev->dev.platform_data = platform_data;
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>  
>  	if (of_device_add(dev) != 0) {
> -		of_dma_deconfigure(&dev->dev);
>  		platform_device_put(dev);
>  		goto err_clear_flag;
>  	}
> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>  		dev_set_name(&dev->dev, "%s", bus_id);
>  	else
>  		of_device_make_bus_id(&dev->dev);
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  
>  	/* Allow the HW Peripheral ID to be overridden */
>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>  		amba_device_unregister(to_amba_device(dev));
>  #endif
>  
> -	of_dma_deconfigure(dev);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>  	return 0;
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index dfc9a27..5a8dd43 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>  	dev_set_msi_domain(&dev->dev, d);
>  }
>  
> -/**
> - * pci_dma_configure - Setup DMA configuration
> - * @dev: ptr to pci_dev struct of the PCI device
> - *
> - * Function to update PCI devices's DMA configuration using the same
> - * info from the OF node or ACPI node of host bridge's parent (if any).
> - */
> -static void pci_dma_configure(struct pci_dev *dev)
> -{
> -	struct device *bridge = pci_get_host_bridge_device(dev);
> -
> -	if (IS_ENABLED(CONFIG_OF) &&
> -		bridge->parent && bridge->parent->of_node) {
> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> -	} else if (has_acpi_companion(bridge)) {
> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
> -
> -		if (attr == DEV_DMA_NOT_SUPPORTED)
> -			dev_warn(&dev->dev, "DMA not supported.\n");
> -		else
> -			acpi_dma_configure(&dev->dev, attr);
> -	}
> -
> -	pci_put_host_bridge_device(bridge);
> -}
> -
>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  {
>  	int ret;
> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>  	dev->dev.dma_mask = &dev->dma_mask;
>  	dev->dev.dma_parms = &dev->dma_parms;
>  	dev->dev.coherent_dma_mask = 0xffffffffull;
> -	pci_dma_configure(dev);
>  
>  	pci_set_dma_max_seg_size(dev, 65536);
>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 0977317..4f3eece 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>  }
>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>  
> +#ifdef CONFIG_HAS_DMA
> +int dma_configure(struct device *dev);
> +void dma_deconfigure(struct device *dev);
> +#else
> +static inline int dma_configure(struct device *dev)
> +{
> +	return 0;
> +}
> +
> +static inline void dma_deconfigure(struct device *dev) {}
> +#endif
> +
>  /*
>   * Managed DMA API
>   */
> 

Re: [PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Hi Robin,

On 4/4/2017 5:47 PM, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
>> Configuring DMA ops at probe time will allow deferring device probe when
>> the IOMMU isn't available yet. The dma_configure for the device is
>> now called from the generic device_attach callback just before the
>> bus/driver probe is called. This way, configuring the DMA ops for the
>> device would be called at the same place for all bus_types, hence the
>> deferred probing mechanism should work for all buses as well.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> This patch changes the dma ops configuration to probe time for
>> both OF and ACPI based platform/amba/pci bus devices.
>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>
> It's possible we could subsume {of,acpi}_dma_deconfigure() into
> dma_deconfigure() entirely in future if it becomes clear that neither of
> them will ever need to do anything firmware-specific, but I think for
> now it's probably safer to keep the current symmetry - calling
> arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
> it really should be!)
>

Ya, calling it twice should cause no harm. If nothing firmware specific
gets added this could simply be put in deconfigure itself in future.
Thanks for all the reviews !!

Regards,
  Sricharan



> Robin.
>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
>> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>
>>  [V10] Added dummy dma_(de)configure functions in case
>>        of !CONFIG_HAS_DMA to avoid build breaks.
>>
>>  drivers/acpi/glue.c         |  5 -----
>>  drivers/base/dd.c           |  9 +++++++++
>>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>>  drivers/of/platform.c       |  5 +----
>>  drivers/pci/probe.c         | 28 ----------------------------
>>  include/linux/dma-mapping.h | 12 ++++++++++++
>>  6 files changed, 62 insertions(+), 37 deletions(-)
>>
>> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
>> index fb19e1c..c05f241 100644
>> --- a/drivers/acpi/glue.c
>> +++ b/drivers/acpi/glue.c
>> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	struct list_head *physnode_list;
>>  	unsigned int node_id;
>>  	int retval = -EINVAL;
>> -	enum dev_dma_attr attr;
>>
>>  	if (has_acpi_companion(dev)) {
>>  		if (acpi_dev) {
>> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	if (!has_acpi_companion(dev))
>>  		ACPI_COMPANION_SET(dev, acpi_dev);
>>
>> -	attr = acpi_get_dma_attr(acpi_dev);
>> -	if (attr != DEV_DMA_NOT_SUPPORTED)
>> -		acpi_dma_configure(dev, attr);
>> -
>>  	acpi_physnode_link_name(physical_node_name, node_id);
>>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>>  				   physical_node_name);
>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> index a1fbf55..4882f06 100644
>> --- a/drivers/base/dd.c
>> +++ b/drivers/base/dd.c
>> @@ -19,6 +19,7 @@
>>
>>  #include <linux/device.h>
>>  #include <linux/delay.h>
>> +#include <linux/dma-mapping.h>
>>  #include <linux/module.h>
>>  #include <linux/kthread.h>
>>  #include <linux/wait.h>
>> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	if (ret)
>>  		goto pinctrl_bind_failed;
>>
>> +	ret = dma_configure(dev);
>> +	if (ret)
>> +		goto dma_failed;
>> +
>>  	if (driver_sysfs_add(dev)) {
>>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>>  			__func__, dev_name(dev));
>> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	goto done;
>>
>>  probe_failed:
>> +	dma_deconfigure(dev);
>> +dma_failed:
>>  	if (dev->bus)
>>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>>  			drv->remove(dev);
>>
>>  		device_links_driver_cleanup(dev);
>> +		dma_deconfigure(dev);
>> +
>>  		devres_release_all(dev);
>>  		dev->driver = NULL;
>>  		dev_set_drvdata(dev, NULL);
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index efd71cf..449b948 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -7,9 +7,11 @@
>>   * This file is released under the GPLv2.
>>   */
>>
>> +#include <linux/acpi.h>
>>  #include <linux/dma-mapping.h>
>>  #include <linux/export.h>
>>  #include <linux/gfp.h>
>> +#include <linux/of_device.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>
>> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>>  	vunmap(cpu_addr);
>>  }
>>  #endif
>> +
>> +/*
>> + * Common configuration to enable DMA API use for a device
>> + */
>> +#include <linux/pci.h>
>> +
>> +int dma_configure(struct device *dev)
>> +{
>> +	struct device *bridge = NULL, *dma_dev = dev;
>> +	enum dev_dma_attr attr;
>> +
>> +	if (dev_is_pci(dev)) {
>> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>> +		dma_dev = bridge;
>> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
>> +		    dma_dev->parent->of_node)
>> +			dma_dev = dma_dev->parent;
>> +	}
>> +
>> +	if (dma_dev->of_node) {
>> +		of_dma_configure(dev, dma_dev->of_node);
>> +	} else if (has_acpi_companion(dma_dev)) {
>> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>> +		if (attr != DEV_DMA_NOT_SUPPORTED)
>> +			acpi_dma_configure(dev, attr);
>> +	}
>> +
>> +	if (bridge)
>> +		pci_put_host_bridge_device(bridge);
>> +
>> +	return 0;
>> +}
>> +
>> +void dma_deconfigure(struct device *dev)
>> +{
>> +	of_dma_deconfigure(dev);
>> +	acpi_dma_deconfigure(dev);
>> +}
>> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
>> index 5344db5..2aa4ebb 100644
>> --- a/drivers/of/platform.c
>> +++ b/drivers/of/platform.c
>> @@ -22,6 +22,7 @@
>>  #include <linux/slab.h>
>>  #include <linux/of_address.h>
>>  #include <linux/of_device.h>
>> +#include <linux/of_iommu.h>
>>  #include <linux/of_irq.h>
>>  #include <linux/of_platform.h>
>>  #include <linux/platform_device.h>
>> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>>
>>  	dev->dev.bus = &platform_bus_type;
>>  	dev->dev.platform_data = platform_data;
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>>
>>  	if (of_device_add(dev) != 0) {
>> -		of_dma_deconfigure(&dev->dev);
>>  		platform_device_put(dev);
>>  		goto err_clear_flag;
>>  	}
>> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>>  		dev_set_name(&dev->dev, "%s", bus_id);
>>  	else
>>  		of_device_make_bus_id(&dev->dev);
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>
>>  	/* Allow the HW Peripheral ID to be overridden */
>>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
>> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>>  		amba_device_unregister(to_amba_device(dev));
>>  #endif
>>
>> -	of_dma_deconfigure(dev);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>>  	return 0;
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index dfc9a27..5a8dd43 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>>  	dev_set_msi_domain(&dev->dev, d);
>>  }
>>
>> -/**
>> - * pci_dma_configure - Setup DMA configuration
>> - * @dev: ptr to pci_dev struct of the PCI device
>> - *
>> - * Function to update PCI devices's DMA configuration using the same
>> - * info from the OF node or ACPI node of host bridge's parent (if any).
>> - */
>> -static void pci_dma_configure(struct pci_dev *dev)
>> -{
>> -	struct device *bridge = pci_get_host_bridge_device(dev);
>> -
>> -	if (IS_ENABLED(CONFIG_OF) &&
>> -		bridge->parent && bridge->parent->of_node) {
>> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
>> -	} else if (has_acpi_companion(bridge)) {
>> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
>> -
>> -		if (attr == DEV_DMA_NOT_SUPPORTED)
>> -			dev_warn(&dev->dev, "DMA not supported.\n");
>> -		else
>> -			acpi_dma_configure(&dev->dev, attr);
>> -	}
>> -
>> -	pci_put_host_bridge_device(bridge);
>> -}
>> -
>>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  {
>>  	int ret;
>> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  	dev->dev.dma_mask = &dev->dma_mask;
>>  	dev->dev.dma_parms = &dev->dma_parms;
>>  	dev->dev.coherent_dma_mask = 0xffffffffull;
>> -	pci_dma_configure(dev);
>>
>>  	pci_set_dma_max_seg_size(dev, 65536);
>>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
>> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
>> index 0977317..4f3eece 100644
>> --- a/include/linux/dma-mapping.h
>> +++ b/include/linux/dma-mapping.h
>> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>>  }
>>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>>
>> +#ifdef CONFIG_HAS_DMA
>> +int dma_configure(struct device *dev);
>> +void dma_deconfigure(struct device *dev);
>> +#else
>> +static inline int dma_configure(struct device *dev)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void dma_deconfigure(struct device *dev) {}
>> +#endif
>> +
>>  /*
>>   * Managed DMA API
>>   */
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Hi Robin,

On 4/4/2017 5:47 PM, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
>> Configuring DMA ops at probe time will allow deferring device probe when
>> the IOMMU isn't available yet. The dma_configure for the device is
>> now called from the generic device_attach callback just before the
>> bus/driver probe is called. This way, configuring the DMA ops for the
>> device would be called at the same place for all bus_types, hence the
>> deferred probing mechanism should work for all buses as well.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> This patch changes the dma ops configuration to probe time for
>> both OF and ACPI based platform/amba/pci bus devices.
>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>
> It's possible we could subsume {of,acpi}_dma_deconfigure() into
> dma_deconfigure() entirely in future if it becomes clear that neither of
> them will ever need to do anything firmware-specific, but I think for
> now it's probably safer to keep the current symmetry - calling
> arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
> it really should be!)
>

Ya, calling it twice should cause no harm. If nothing firmware specific
gets added this could simply be put in deconfigure itself in future.
Thanks for all the reviews !!

Regards,
  Sricharan



> Robin.
>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
>> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>
>>  [V10] Added dummy dma_(de)configure functions in case
>>        of !CONFIG_HAS_DMA to avoid build breaks.
>>
>>  drivers/acpi/glue.c         |  5 -----
>>  drivers/base/dd.c           |  9 +++++++++
>>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>>  drivers/of/platform.c       |  5 +----
>>  drivers/pci/probe.c         | 28 ----------------------------
>>  include/linux/dma-mapping.h | 12 ++++++++++++
>>  6 files changed, 62 insertions(+), 37 deletions(-)
>>
>> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
>> index fb19e1c..c05f241 100644
>> --- a/drivers/acpi/glue.c
>> +++ b/drivers/acpi/glue.c
>> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	struct list_head *physnode_list;
>>  	unsigned int node_id;
>>  	int retval = -EINVAL;
>> -	enum dev_dma_attr attr;
>>
>>  	if (has_acpi_companion(dev)) {
>>  		if (acpi_dev) {
>> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	if (!has_acpi_companion(dev))
>>  		ACPI_COMPANION_SET(dev, acpi_dev);
>>
>> -	attr = acpi_get_dma_attr(acpi_dev);
>> -	if (attr != DEV_DMA_NOT_SUPPORTED)
>> -		acpi_dma_configure(dev, attr);
>> -
>>  	acpi_physnode_link_name(physical_node_name, node_id);
>>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>>  				   physical_node_name);
>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> index a1fbf55..4882f06 100644
>> --- a/drivers/base/dd.c
>> +++ b/drivers/base/dd.c
>> @@ -19,6 +19,7 @@
>>
>>  #include <linux/device.h>
>>  #include <linux/delay.h>
>> +#include <linux/dma-mapping.h>
>>  #include <linux/module.h>
>>  #include <linux/kthread.h>
>>  #include <linux/wait.h>
>> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	if (ret)
>>  		goto pinctrl_bind_failed;
>>
>> +	ret = dma_configure(dev);
>> +	if (ret)
>> +		goto dma_failed;
>> +
>>  	if (driver_sysfs_add(dev)) {
>>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>>  			__func__, dev_name(dev));
>> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	goto done;
>>
>>  probe_failed:
>> +	dma_deconfigure(dev);
>> +dma_failed:
>>  	if (dev->bus)
>>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>>  			drv->remove(dev);
>>
>>  		device_links_driver_cleanup(dev);
>> +		dma_deconfigure(dev);
>> +
>>  		devres_release_all(dev);
>>  		dev->driver = NULL;
>>  		dev_set_drvdata(dev, NULL);
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index efd71cf..449b948 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -7,9 +7,11 @@
>>   * This file is released under the GPLv2.
>>   */
>>
>> +#include <linux/acpi.h>
>>  #include <linux/dma-mapping.h>
>>  #include <linux/export.h>
>>  #include <linux/gfp.h>
>> +#include <linux/of_device.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>
>> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>>  	vunmap(cpu_addr);
>>  }
>>  #endif
>> +
>> +/*
>> + * Common configuration to enable DMA API use for a device
>> + */
>> +#include <linux/pci.h>
>> +
>> +int dma_configure(struct device *dev)
>> +{
>> +	struct device *bridge = NULL, *dma_dev = dev;
>> +	enum dev_dma_attr attr;
>> +
>> +	if (dev_is_pci(dev)) {
>> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>> +		dma_dev = bridge;
>> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
>> +		    dma_dev->parent->of_node)
>> +			dma_dev = dma_dev->parent;
>> +	}
>> +
>> +	if (dma_dev->of_node) {
>> +		of_dma_configure(dev, dma_dev->of_node);
>> +	} else if (has_acpi_companion(dma_dev)) {
>> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>> +		if (attr != DEV_DMA_NOT_SUPPORTED)
>> +			acpi_dma_configure(dev, attr);
>> +	}
>> +
>> +	if (bridge)
>> +		pci_put_host_bridge_device(bridge);
>> +
>> +	return 0;
>> +}
>> +
>> +void dma_deconfigure(struct device *dev)
>> +{
>> +	of_dma_deconfigure(dev);
>> +	acpi_dma_deconfigure(dev);
>> +}
>> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
>> index 5344db5..2aa4ebb 100644
>> --- a/drivers/of/platform.c
>> +++ b/drivers/of/platform.c
>> @@ -22,6 +22,7 @@
>>  #include <linux/slab.h>
>>  #include <linux/of_address.h>
>>  #include <linux/of_device.h>
>> +#include <linux/of_iommu.h>
>>  #include <linux/of_irq.h>
>>  #include <linux/of_platform.h>
>>  #include <linux/platform_device.h>
>> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>>
>>  	dev->dev.bus = &platform_bus_type;
>>  	dev->dev.platform_data = platform_data;
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>>
>>  	if (of_device_add(dev) != 0) {
>> -		of_dma_deconfigure(&dev->dev);
>>  		platform_device_put(dev);
>>  		goto err_clear_flag;
>>  	}
>> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>>  		dev_set_name(&dev->dev, "%s", bus_id);
>>  	else
>>  		of_device_make_bus_id(&dev->dev);
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>
>>  	/* Allow the HW Peripheral ID to be overridden */
>>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
>> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>>  		amba_device_unregister(to_amba_device(dev));
>>  #endif
>>
>> -	of_dma_deconfigure(dev);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>>  	return 0;
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index dfc9a27..5a8dd43 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>>  	dev_set_msi_domain(&dev->dev, d);
>>  }
>>
>> -/**
>> - * pci_dma_configure - Setup DMA configuration
>> - * @dev: ptr to pci_dev struct of the PCI device
>> - *
>> - * Function to update PCI devices's DMA configuration using the same
>> - * info from the OF node or ACPI node of host bridge's parent (if any).
>> - */
>> -static void pci_dma_configure(struct pci_dev *dev)
>> -{
>> -	struct device *bridge = pci_get_host_bridge_device(dev);
>> -
>> -	if (IS_ENABLED(CONFIG_OF) &&
>> -		bridge->parent && bridge->parent->of_node) {
>> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
>> -	} else if (has_acpi_companion(bridge)) {
>> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
>> -
>> -		if (attr == DEV_DMA_NOT_SUPPORTED)
>> -			dev_warn(&dev->dev, "DMA not supported.\n");
>> -		else
>> -			acpi_dma_configure(&dev->dev, attr);
>> -	}
>> -
>> -	pci_put_host_bridge_device(bridge);
>> -}
>> -
>>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  {
>>  	int ret;
>> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  	dev->dev.dma_mask = &dev->dma_mask;
>>  	dev->dev.dma_parms = &dev->dma_parms;
>>  	dev->dev.coherent_dma_mask = 0xffffffffull;
>> -	pci_dma_configure(dev);
>>
>>  	pci_set_dma_max_seg_size(dev, 65536);
>>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
>> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
>> index 0977317..4f3eece 100644
>> --- a/include/linux/dma-mapping.h
>> +++ b/include/linux/dma-mapping.h
>> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>>  }
>>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>>
>> +#ifdef CONFIG_HAS_DMA
>> +int dma_configure(struct device *dev);
>> +void dma_deconfigure(struct device *dev);
>> +#else
>> +static inline int dma_configure(struct device *dev)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void dma_deconfigure(struct device *dev) {}
>> +#endif
>> +
>>  /*
>>   * Managed DMA API
>>   */
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 07/12] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Hi Robin,

On 4/4/2017 5:47 PM, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
>> Configuring DMA ops at probe time will allow deferring device probe when
>> the IOMMU isn't available yet. The dma_configure for the device is
>> now called from the generic device_attach callback just before the
>> bus/driver probe is called. This way, configuring the DMA ops for the
>> device would be called at the same place for all bus_types, hence the
>> deferred probing mechanism should work for all buses as well.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>> Similarly on the device/driver_unregister path __device_release_driver is
>> called which inturn calls dma_deconfigure.
>>
>> This patch changes the dma ops configuration to probe time for
>> both OF and ACPI based platform/amba/pci bus devices.
>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>
> It's possible we could subsume {of,acpi}_dma_deconfigure() into
> dma_deconfigure() entirely in future if it becomes clear that neither of
> them will ever need to do anything firmware-specific, but I think for
> now it's probably safer to keep the current symmetry - calling
> arch_teardown_dma_ops() twice is benign (and even if it weren't, I'd say
> it really should be!)
>

Ya, calling it twice should cause no harm. If nothing firmware specific
gets added this could simply be put in deconfigure itself in future.
Thanks for all the reviews !!

Regards,
  Sricharan



> Robin.
>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
>> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>
>>  [V10] Added dummy dma_(de)configure functions in case
>>        of !CONFIG_HAS_DMA to avoid build breaks.
>>
>>  drivers/acpi/glue.c         |  5 -----
>>  drivers/base/dd.c           |  9 +++++++++
>>  drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
>>  drivers/of/platform.c       |  5 +----
>>  drivers/pci/probe.c         | 28 ----------------------------
>>  include/linux/dma-mapping.h | 12 ++++++++++++
>>  6 files changed, 62 insertions(+), 37 deletions(-)
>>
>> diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
>> index fb19e1c..c05f241 100644
>> --- a/drivers/acpi/glue.c
>> +++ b/drivers/acpi/glue.c
>> @@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	struct list_head *physnode_list;
>>  	unsigned int node_id;
>>  	int retval = -EINVAL;
>> -	enum dev_dma_attr attr;
>>
>>  	if (has_acpi_companion(dev)) {
>>  		if (acpi_dev) {
>> @@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
>>  	if (!has_acpi_companion(dev))
>>  		ACPI_COMPANION_SET(dev, acpi_dev);
>>
>> -	attr = acpi_get_dma_attr(acpi_dev);
>> -	if (attr != DEV_DMA_NOT_SUPPORTED)
>> -		acpi_dma_configure(dev, attr);
>> -
>>  	acpi_physnode_link_name(physical_node_name, node_id);
>>  	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
>>  				   physical_node_name);
>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> index a1fbf55..4882f06 100644
>> --- a/drivers/base/dd.c
>> +++ b/drivers/base/dd.c
>> @@ -19,6 +19,7 @@
>>
>>  #include <linux/device.h>
>>  #include <linux/delay.h>
>> +#include <linux/dma-mapping.h>
>>  #include <linux/module.h>
>>  #include <linux/kthread.h>
>>  #include <linux/wait.h>
>> @@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	if (ret)
>>  		goto pinctrl_bind_failed;
>>
>> +	ret = dma_configure(dev);
>> +	if (ret)
>> +		goto dma_failed;
>> +
>>  	if (driver_sysfs_add(dev)) {
>>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>>  			__func__, dev_name(dev));
>> @@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	goto done;
>>
>>  probe_failed:
>> +	dma_deconfigure(dev);
>> +dma_failed:
>>  	if (dev->bus)
>>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>> @@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
>>  			drv->remove(dev);
>>
>>  		device_links_driver_cleanup(dev);
>> +		dma_deconfigure(dev);
>> +
>>  		devres_release_all(dev);
>>  		dev->driver = NULL;
>>  		dev_set_drvdata(dev, NULL);
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index efd71cf..449b948 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -7,9 +7,11 @@
>>   * This file is released under the GPLv2.
>>   */
>>
>> +#include <linux/acpi.h>
>>  #include <linux/dma-mapping.h>
>>  #include <linux/export.h>
>>  #include <linux/gfp.h>
>> +#include <linux/of_device.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>
>> @@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
>>  	vunmap(cpu_addr);
>>  }
>>  #endif
>> +
>> +/*
>> + * Common configuration to enable DMA API use for a device
>> + */
>> +#include <linux/pci.h>
>> +
>> +int dma_configure(struct device *dev)
>> +{
>> +	struct device *bridge = NULL, *dma_dev = dev;
>> +	enum dev_dma_attr attr;
>> +
>> +	if (dev_is_pci(dev)) {
>> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>> +		dma_dev = bridge;
>> +		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
>> +		    dma_dev->parent->of_node)
>> +			dma_dev = dma_dev->parent;
>> +	}
>> +
>> +	if (dma_dev->of_node) {
>> +		of_dma_configure(dev, dma_dev->of_node);
>> +	} else if (has_acpi_companion(dma_dev)) {
>> +		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>> +		if (attr != DEV_DMA_NOT_SUPPORTED)
>> +			acpi_dma_configure(dev, attr);
>> +	}
>> +
>> +	if (bridge)
>> +		pci_put_host_bridge_device(bridge);
>> +
>> +	return 0;
>> +}
>> +
>> +void dma_deconfigure(struct device *dev)
>> +{
>> +	of_dma_deconfigure(dev);
>> +	acpi_dma_deconfigure(dev);
>> +}
>> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
>> index 5344db5..2aa4ebb 100644
>> --- a/drivers/of/platform.c
>> +++ b/drivers/of/platform.c
>> @@ -22,6 +22,7 @@
>>  #include <linux/slab.h>
>>  #include <linux/of_address.h>
>>  #include <linux/of_device.h>
>> +#include <linux/of_iommu.h>
>>  #include <linux/of_irq.h>
>>  #include <linux/of_platform.h>
>>  #include <linux/platform_device.h>
>> @@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
>>
>>  	dev->dev.bus = &platform_bus_type;
>>  	dev->dev.platform_data = platform_data;
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>>
>>  	if (of_device_add(dev) != 0) {
>> -		of_dma_deconfigure(&dev->dev);
>>  		platform_device_put(dev);
>>  		goto err_clear_flag;
>>  	}
>> @@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>>  		dev_set_name(&dev->dev, "%s", bus_id);
>>  	else
>>  		of_device_make_bus_id(&dev->dev);
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>
>>  	/* Allow the HW Peripheral ID to be overridden */
>>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
>> @@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>>  		amba_device_unregister(to_amba_device(dev));
>>  #endif
>>
>> -	of_dma_deconfigure(dev);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>>  	return 0;
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index dfc9a27..5a8dd43 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
>>  	dev_set_msi_domain(&dev->dev, d);
>>  }
>>
>> -/**
>> - * pci_dma_configure - Setup DMA configuration
>> - * @dev: ptr to pci_dev struct of the PCI device
>> - *
>> - * Function to update PCI devices's DMA configuration using the same
>> - * info from the OF node or ACPI node of host bridge's parent (if any).
>> - */
>> -static void pci_dma_configure(struct pci_dev *dev)
>> -{
>> -	struct device *bridge = pci_get_host_bridge_device(dev);
>> -
>> -	if (IS_ENABLED(CONFIG_OF) &&
>> -		bridge->parent && bridge->parent->of_node) {
>> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
>> -	} else if (has_acpi_companion(bridge)) {
>> -		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>> -		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
>> -
>> -		if (attr == DEV_DMA_NOT_SUPPORTED)
>> -			dev_warn(&dev->dev, "DMA not supported.\n");
>> -		else
>> -			acpi_dma_configure(&dev->dev, attr);
>> -	}
>> -
>> -	pci_put_host_bridge_device(bridge);
>> -}
>> -
>>  void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  {
>>  	int ret;
>> @@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
>>  	dev->dev.dma_mask = &dev->dma_mask;
>>  	dev->dev.dma_parms = &dev->dma_parms;
>>  	dev->dev.coherent_dma_mask = 0xffffffffull;
>> -	pci_dma_configure(dev);
>>
>>  	pci_set_dma_max_seg_size(dev, 65536);
>>  	pci_set_dma_seg_boundary(dev, 0xffffffff);
>> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
>> index 0977317..4f3eece 100644
>> --- a/include/linux/dma-mapping.h
>> +++ b/include/linux/dma-mapping.h
>> @@ -728,6 +728,18 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
>>  }
>>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>>
>> +#ifdef CONFIG_HAS_DMA
>> +int dma_configure(struct device *dev);
>> +void dma_deconfigure(struct device *dev);
>> +#else
>> +static inline int dma_configure(struct device *dev)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void dma_deconfigure(struct device *dev) {}
>> +#endif
>> +
>>  /*
>>   * Managed DMA API
>>   */
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> against mainline and linux-next, iommu-next.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Sricharan, thanks for keeping this going - I really think we're there now :)

Joerg, I realise that at -rc5 time is getting on a bit already, but even
the non-vintage parts of the series are pretty mature now so it would be
nice to at least give it a spin in -next. If you don't quite share my
confidence for landing it in 4.12, please consider it for early next
cycle to get a full workout.

Thanks,
Robin.

> Previous post of this series [8]. 
> 
> Please note that, i have kept the tested/acked tags intact from V8
> because V9/10 were for more fixes that was added, so the original
> tags that was given for the functional testing remains the same.
> 
>  [V10]
>      * Rebased on top of 4.11-rc5.
>  
>      * Fixed coherent_dma_mask 64bit overflow issue [8]
>        for OF. The fix for OF was added as a separate
>        patch#6, since the issue is true even without probe deferral,
>        but gets reproduced with the probe deferral series.
>        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>      * Also fixed an build issue caught by kbuild robot for
>        m68k arch. The issue was dma_(de)configure was not
>        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property and deferencing
>        of_iommu_table after init.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> [8] http://patchwork.ozlabs.org/patch/743898/
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (4):
>   of: device: Fix overflow of coherent_dma_mask
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |  11 ++-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  36 +++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |  12 ++++
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 329 insertions(+), 338 deletions(-)
> 

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> against mainline and linux-next, iommu-next.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Sricharan, thanks for keeping this going - I really think we're there now :)

Joerg, I realise that at -rc5 time is getting on a bit already, but even
the non-vintage parts of the series are pretty mature now so it would be
nice to at least give it a spin in -next. If you don't quite share my
confidence for landing it in 4.12, please consider it for early next
cycle to get a full workout.

Thanks,
Robin.

> Previous post of this series [8]. 
> 
> Please note that, i have kept the tested/acked tags intact from V8
> because V9/10 were for more fixes that was added, so the original
> tags that was given for the functional testing remains the same.
> 
>  [V10]
>      * Rebased on top of 4.11-rc5.
>  
>      * Fixed coherent_dma_mask 64bit overflow issue [8]
>        for OF. The fix for OF was added as a separate
>        patch#6, since the issue is true even without probe deferral,
>        but gets reproduced with the probe deferral series.
>        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>      * Also fixed an build issue caught by kbuild robot for
>        m68k arch. The issue was dma_(de)configure was not
>        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property and deferencing
>        of_iommu_table after init.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> [8] http://patchwork.ozlabs.org/patch/743898/
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (4):
>   of: device: Fix overflow of coherent_dma_mask
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |  11 ++-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  36 +++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |  12 ++++
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 329 insertions(+), 338 deletions(-)
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 00/12] IOMMU probe deferral support

[Robin Murphy]

On 04/04/17 11:18, Sricharan R wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
> Similarly on the device/driver_unregister path __device_release_driver is
> called which inturn calls dma_deconfigure.
> 
> Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> against mainline and linux-next, iommu-next.
>   
> * Tested with platform and pci devices for probe deferral
>   and reprobe on arm64 based platform.

Sricharan, thanks for keeping this going - I really think we're there now :)

Joerg, I realise that at -rc5 time is getting on a bit already, but even
the non-vintage parts of the series are pretty mature now so it would be
nice to at least give it a spin in -next. If you don't quite share my
confidence for landing it in 4.12, please consider it for early next
cycle to get a full workout.

Thanks,
Robin.

> Previous post of this series [8]. 
> 
> Please note that, i have kept the tested/acked tags intact from V8
> because V9/10 were for more fixes that was added, so the original
> tags that was given for the functional testing remains the same.
> 
>  [V10]
>      * Rebased on top of 4.11-rc5.
>  
>      * Fixed coherent_dma_mask 64bit overflow issue [8]
>        for OF. The fix for OF was added as a separate
>        patch#6, since the issue is true even without probe deferral,
>        but gets reproduced with the probe deferral series.
>        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
>        and the fix for dma_configure getting called more than
>        once for the same device.
> 
>      * Also fixed an build issue caught by kbuild robot for
>        m68k arch. The issue was dma_(de)configure was not
>        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> 
>  [V9]
>      * Rebased on top of 4.11-rc1.
> 
>      * Merged Robin's fixes for legacy binding issue,
>        pci devices with no iommu-map property and deferencing
>        of_iommu_table after init.
>      
>  [V8]
>      * Picked up all the acks and tested tags from Marek and
>        Hanjun for DT and ACPI patches respectively, since
>        no functional changes was done.
> 
>      * Addressed Minor comments Sinan and Bjorn.
> 
>      * Added Robin's fix for fixing the deferencing NULL for
>        of_iommu_table after init in patch #2.
> 
>      * Rebased it on top of linux-next
> 
>  [V7]
>      * Updated the subject and commit log for patch #6 as per
>        comments from Lorenzo. No functional changes.
> 
>  [V6]
>      * Fixed a bug in dma_configure function pointed out by
>        Robin.
>      * Reordered the patches as per comments from Robin and
>        Lorenzo.
>      * Added Tags.
> 
>  [V5]
>      * Reworked the pci configuration code hanging outside and
>        pushed it to dma_configure as in PATCH#5,6,7.
>        Also added a couple of patches that Lorenzo provided for
>        correcting the Probe deferring mechanism in case of
>        ACPI devices from here [5].
> 
>  [V4]
>      * Took the reworked patches [2] from Robin's branch and
>        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> 
>      * Added the patches for moving the dma ops configuration of
>        acpi based devices to probe time as well.
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration
>        separately based on review comments that both masks and ops are
>        required only during the device probe time.
> 
>      * Reworked the series based on Generic DT bindings series.
> 
>      * Added call to iommu's remove_device in the cleanup path for arm and
>        arm64.
> 
>      * Removed the notifier trick in arm64 to handle early device
>        registration.
> 
>      * Added reset of dma_ops in cleanup path for arm based on comments.
> 
>      * Fixed the pci_iommu_configure path and tested with PCI device as
>        well.
>  
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
>        last post.
> 
>      * Fixed few other cosmetic comments.
>   
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from
>        generic code
>  
>      * Added iommu add_device callback from of_iommu_configure path
> 
>  [V1]
>      * Initial post from Laurent Pinchart [1]
> 
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> [3] https://lkml.org/lkml/2016/11/21/141
> [4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> [8] http://patchwork.ozlabs.org/patch/743898/
> 
> Laurent Pinchart (3):
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> 
> Lorenzo Pieralisi (2):
>   ACPI/IORT: Add function to check SMMUs drivers presence
>   ACPI/IORT: Remove linker section for IORT entries probing
> 
> Robin Murphy (3):
>   iommu/of: Refactor of_iommu_configure() for error handling
>   iommu/of: Prepare for deferred IOMMU configuration
>   iommu/arm-smmu: Clean up early-probing workarounds
> 
> Sricharan R (4):
>   of: device: Fix overflow of coherent_dma_mask
>   of/acpi: Configure dma operations at probe time for platform/amba/pci
>     bus devices
>   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
>     error
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
> 
>  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
>  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
>  drivers/acpi/glue.c               |   5 --
>  drivers/acpi/scan.c               |  11 ++-
>  drivers/base/dd.c                 |   9 +++
>  drivers/base/dma-mapping.c        |  41 +++++++++++
>  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
>  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
>  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
>  drivers/of/address.c              |  20 +++++-
>  drivers/of/device.c               |  36 +++++-----
>  drivers/of/platform.c             |  10 +--
>  drivers/pci/probe.c               |  28 --------
>  include/acpi/acpi_bus.h           |   2 +-
>  include/asm-generic/vmlinux.lds.h |   1 -
>  include/linux/acpi.h              |   7 +-
>  include/linux/acpi_iort.h         |   3 -
>  include/linux/dma-mapping.h       |  12 ++++
>  include/linux/of_device.h         |  10 ++-
>  19 files changed, 329 insertions(+), 338 deletions(-)
> 

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Rob Herring]

On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.

For patches 3, 4, 6, 7, 8:

Acked-by: Rob Herring <robh@kernel.org>

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Rob Herring]

On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.

For patches 3, 4, 6, 7, 8:

Acked-by: Rob Herring <robh@kernel.org>

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Rob Herring]

On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.

For patches 3, 4, 6, 7, 8:

Acked-by: Rob Herring <robh@kernel.org>

[PATCH V10 00/12] IOMMU probe deferral support

[Rob Herring]

On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
> This series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.

For patches 3, 4, 6, 7, 8:

Acked-by: Rob Herring <robh@kernel.org>

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Tue, Apr 04, 2017 at 01:49:29PM +0100, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
> > This series calls the dma ops configuration for the devices
> > at a generic place so that it works for all busses.
> > The dma_configure_ops for a device is now called during
> > the device_attach callback just before the probe of the
> > bus/driver is called. Similarly dma_deconfigure is called during
> > device/driver_detach path.
> > 
> > pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >        |                         |
> > pci_bus_add_device     (device_add/driver_register)
> >        |                         |
> > device_attach           device_initial_probe
> >        |                         |
> > __device_attach_driver    __device_attach_driver
> >        |
> > driver_probe_device
> >        |
> > really_probe
> >        |
> > dma_configure
> > 
> > Similarly on the device/driver_unregister path __device_release_driver is
> > called which inturn calls dma_deconfigure.
> > 
> > Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> > against mainline and linux-next, iommu-next.
> >   
> > * Tested with platform and pci devices for probe deferral
> >   and reprobe on arm64 based platform.
> 
> Sricharan, thanks for keeping this going - I really think we're there now :)

FYI, I re-tested this series with ACPI and tried a merge with IORT
patches queued for 4.12 (via arm64) and there does not seem to be any
merge conflicts so it should be ready to go.

Thanks,
Lorenzo

> Joerg, I realise that at -rc5 time is getting on a bit already, but even
> the non-vintage parts of the series are pretty mature now so it would be
> nice to at least give it a spin in -next. If you don't quite share my
> confidence for landing it in 4.12, please consider it for early next
> cycle to get a full workout.
> 
> Thanks,
> Robin.
> 
> > Previous post of this series [8]. 
> > 
> > Please note that, i have kept the tested/acked tags intact from V8
> > because V9/10 were for more fixes that was added, so the original
> > tags that was given for the functional testing remains the same.
> > 
> >  [V10]
> >      * Rebased on top of 4.11-rc5.
> >  
> >      * Fixed coherent_dma_mask 64bit overflow issue [8]
> >        for OF. The fix for OF was added as a separate
> >        patch#6, since the issue is true even without probe deferral,
> >        but gets reproduced with the probe deferral series.
> >        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
> >        and the fix for dma_configure getting called more than
> >        once for the same device.
> > 
> >      * Also fixed an build issue caught by kbuild robot for
> >        m68k arch. The issue was dma_(de)configure was not
> >        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> > 
> >  [V9]
> >      * Rebased on top of 4.11-rc1.
> > 
> >      * Merged Robin's fixes for legacy binding issue,
> >        pci devices with no iommu-map property and deferencing
> >        of_iommu_table after init.
> >      
> >  [V8]
> >      * Picked up all the acks and tested tags from Marek and
> >        Hanjun for DT and ACPI patches respectively, since
> >        no functional changes was done.
> > 
> >      * Addressed Minor comments Sinan and Bjorn.
> > 
> >      * Added Robin's fix for fixing the deferencing NULL for
> >        of_iommu_table after init in patch #2.
> > 
> >      * Rebased it on top of linux-next
> > 
> >  [V7]
> >      * Updated the subject and commit log for patch #6 as per
> >        comments from Lorenzo. No functional changes.
> > 
> >  [V6]
> >      * Fixed a bug in dma_configure function pointed out by
> >        Robin.
> >      * Reordered the patches as per comments from Robin and
> >        Lorenzo.
> >      * Added Tags.
> > 
> >  [V5]
> >      * Reworked the pci configuration code hanging outside and
> >        pushed it to dma_configure as in PATCH#5,6,7.
> >        Also added a couple of patches that Lorenzo provided for
> >        correcting the Probe deferring mechanism in case of
> >        ACPI devices from here [5].
> > 
> >  [V4]
> >      * Took the reworked patches [2] from Robin's branch and
> >        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> > 
> >      * Added the patches for moving the dma ops configuration of
> >        acpi based devices to probe time as well.
> >  [V3]
> >      * Removed the patch to split dma_masks/dma_ops configuration
> >        separately based on review comments that both masks and ops are
> >        required only during the device probe time.
> > 
> >      * Reworked the series based on Generic DT bindings series.
> > 
> >      * Added call to iommu's remove_device in the cleanup path for arm and
> >        arm64.
> > 
> >      * Removed the notifier trick in arm64 to handle early device
> >        registration.
> > 
> >      * Added reset of dma_ops in cleanup path for arm based on comments.
> > 
> >      * Fixed the pci_iommu_configure path and tested with PCI device as
> >        well.
> >  
> >      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
> >        last post.
> > 
> >      * Fixed few other cosmetic comments.
> >   
> >  [V2]
> >      * Updated the Initial post to call dma_configure/deconfigure from
> >        generic code
> >  
> >      * Added iommu add_device callback from of_iommu_configure path
> > 
> >  [V1]
> >      * Initial post from Laurent Pinchart [1]
> > 
> > [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> > [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> > [3] https://lkml.org/lkml/2016/11/21/141
> > [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> > [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> > [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> > [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> > [8] http://patchwork.ozlabs.org/patch/743898/
> > 
> > Laurent Pinchart (3):
> >   of: dma: Move range size workaround to of_dma_get_range()
> >   of: dma: Make of_dma_deconfigure() public
> >   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> > 
> > Lorenzo Pieralisi (2):
> >   ACPI/IORT: Add function to check SMMUs drivers presence
> >   ACPI/IORT: Remove linker section for IORT entries probing
> > 
> > Robin Murphy (3):
> >   iommu/of: Refactor of_iommu_configure() for error handling
> >   iommu/of: Prepare for deferred IOMMU configuration
> >   iommu/arm-smmu: Clean up early-probing workarounds
> > 
> > Sricharan R (4):
> >   of: device: Fix overflow of coherent_dma_mask
> >   of/acpi: Configure dma operations at probe time for platform/amba/pci
> >     bus devices
> >   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
> >     error
> >   arm64: dma-mapping: Remove the notifier trick to handle early setting
> >     of dma_ops
> > 
> >  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
> >  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
> >  drivers/acpi/glue.c               |   5 --
> >  drivers/acpi/scan.c               |  11 ++-
> >  drivers/base/dd.c                 |   9 +++
> >  drivers/base/dma-mapping.c        |  41 +++++++++++
> >  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
> >  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
> >  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
> >  drivers/of/address.c              |  20 +++++-
> >  drivers/of/device.c               |  36 +++++-----
> >  drivers/of/platform.c             |  10 +--
> >  drivers/pci/probe.c               |  28 --------
> >  include/acpi/acpi_bus.h           |   2 +-
> >  include/asm-generic/vmlinux.lds.h |   1 -
> >  include/linux/acpi.h              |   7 +-
> >  include/linux/acpi_iort.h         |   3 -
> >  include/linux/dma-mapping.h       |  12 ++++
> >  include/linux/of_device.h         |  10 ++-
> >  19 files changed, 329 insertions(+), 338 deletions(-)
> > 
> 

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Tue, Apr 04, 2017 at 01:49:29PM +0100, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
> > This series calls the dma ops configuration for the devices
> > at a generic place so that it works for all busses.
> > The dma_configure_ops for a device is now called during
> > the device_attach callback just before the probe of the
> > bus/driver is called. Similarly dma_deconfigure is called during
> > device/driver_detach path.
> > 
> > pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >        |                         |
> > pci_bus_add_device     (device_add/driver_register)
> >        |                         |
> > device_attach           device_initial_probe
> >        |                         |
> > __device_attach_driver    __device_attach_driver
> >        |
> > driver_probe_device
> >        |
> > really_probe
> >        |
> > dma_configure
> > 
> > Similarly on the device/driver_unregister path __device_release_driver is
> > called which inturn calls dma_deconfigure.
> > 
> > Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> > against mainline and linux-next, iommu-next.
> >   
> > * Tested with platform and pci devices for probe deferral
> >   and reprobe on arm64 based platform.
> 
> Sricharan, thanks for keeping this going - I really think we're there now :)

FYI, I re-tested this series with ACPI and tried a merge with IORT
patches queued for 4.12 (via arm64) and there does not seem to be any
merge conflicts so it should be ready to go.

Thanks,
Lorenzo

> Joerg, I realise that at -rc5 time is getting on a bit already, but even
> the non-vintage parts of the series are pretty mature now so it would be
> nice to at least give it a spin in -next. If you don't quite share my
> confidence for landing it in 4.12, please consider it for early next
> cycle to get a full workout.
> 
> Thanks,
> Robin.
> 
> > Previous post of this series [8]. 
> > 
> > Please note that, i have kept the tested/acked tags intact from V8
> > because V9/10 were for more fixes that was added, so the original
> > tags that was given for the functional testing remains the same.
> > 
> >  [V10]
> >      * Rebased on top of 4.11-rc5.
> >  
> >      * Fixed coherent_dma_mask 64bit overflow issue [8]
> >        for OF. The fix for OF was added as a separate
> >        patch#6, since the issue is true even without probe deferral,
> >        but gets reproduced with the probe deferral series.
> >        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
> >        and the fix for dma_configure getting called more than
> >        once for the same device.
> > 
> >      * Also fixed an build issue caught by kbuild robot for
> >        m68k arch. The issue was dma_(de)configure was not
> >        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> > 
> >  [V9]
> >      * Rebased on top of 4.11-rc1.
> > 
> >      * Merged Robin's fixes for legacy binding issue,
> >        pci devices with no iommu-map property and deferencing
> >        of_iommu_table after init.
> >      
> >  [V8]
> >      * Picked up all the acks and tested tags from Marek and
> >        Hanjun for DT and ACPI patches respectively, since
> >        no functional changes was done.
> > 
> >      * Addressed Minor comments Sinan and Bjorn.
> > 
> >      * Added Robin's fix for fixing the deferencing NULL for
> >        of_iommu_table after init in patch #2.
> > 
> >      * Rebased it on top of linux-next
> > 
> >  [V7]
> >      * Updated the subject and commit log for patch #6 as per
> >        comments from Lorenzo. No functional changes.
> > 
> >  [V6]
> >      * Fixed a bug in dma_configure function pointed out by
> >        Robin.
> >      * Reordered the patches as per comments from Robin and
> >        Lorenzo.
> >      * Added Tags.
> > 
> >  [V5]
> >      * Reworked the pci configuration code hanging outside and
> >        pushed it to dma_configure as in PATCH#5,6,7.
> >        Also added a couple of patches that Lorenzo provided for
> >        correcting the Probe deferring mechanism in case of
> >        ACPI devices from here [5].
> > 
> >  [V4]
> >      * Took the reworked patches [2] from Robin's branch and
> >        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> > 
> >      * Added the patches for moving the dma ops configuration of
> >        acpi based devices to probe time as well.
> >  [V3]
> >      * Removed the patch to split dma_masks/dma_ops configuration
> >        separately based on review comments that both masks and ops are
> >        required only during the device probe time.
> > 
> >      * Reworked the series based on Generic DT bindings series.
> > 
> >      * Added call to iommu's remove_device in the cleanup path for arm and
> >        arm64.
> > 
> >      * Removed the notifier trick in arm64 to handle early device
> >        registration.
> > 
> >      * Added reset of dma_ops in cleanup path for arm based on comments.
> > 
> >      * Fixed the pci_iommu_configure path and tested with PCI device as
> >        well.
> >  
> >      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
> >        last post.
> > 
> >      * Fixed few other cosmetic comments.
> >   
> >  [V2]
> >      * Updated the Initial post to call dma_configure/deconfigure from
> >        generic code
> >  
> >      * Added iommu add_device callback from of_iommu_configure path
> > 
> >  [V1]
> >      * Initial post from Laurent Pinchart [1]
> > 
> > [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> > [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> > [3] https://lkml.org/lkml/2016/11/21/141
> > [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> > [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> > [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> > [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> > [8] http://patchwork.ozlabs.org/patch/743898/
> > 
> > Laurent Pinchart (3):
> >   of: dma: Move range size workaround to of_dma_get_range()
> >   of: dma: Make of_dma_deconfigure() public
> >   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> > 
> > Lorenzo Pieralisi (2):
> >   ACPI/IORT: Add function to check SMMUs drivers presence
> >   ACPI/IORT: Remove linker section for IORT entries probing
> > 
> > Robin Murphy (3):
> >   iommu/of: Refactor of_iommu_configure() for error handling
> >   iommu/of: Prepare for deferred IOMMU configuration
> >   iommu/arm-smmu: Clean up early-probing workarounds
> > 
> > Sricharan R (4):
> >   of: device: Fix overflow of coherent_dma_mask
> >   of/acpi: Configure dma operations at probe time for platform/amba/pci
> >     bus devices
> >   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
> >     error
> >   arm64: dma-mapping: Remove the notifier trick to handle early setting
> >     of dma_ops
> > 
> >  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
> >  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
> >  drivers/acpi/glue.c               |   5 --
> >  drivers/acpi/scan.c               |  11 ++-
> >  drivers/base/dd.c                 |   9 +++
> >  drivers/base/dma-mapping.c        |  41 +++++++++++
> >  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
> >  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
> >  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
> >  drivers/of/address.c              |  20 +++++-
> >  drivers/of/device.c               |  36 +++++-----
> >  drivers/of/platform.c             |  10 +--
> >  drivers/pci/probe.c               |  28 --------
> >  include/acpi/acpi_bus.h           |   2 +-
> >  include/asm-generic/vmlinux.lds.h |   1 -
> >  include/linux/acpi.h              |   7 +-
> >  include/linux/acpi_iort.h         |   3 -
> >  include/linux/dma-mapping.h       |  12 ++++
> >  include/linux/of_device.h         |  10 ++-
> >  19 files changed, 329 insertions(+), 338 deletions(-)
> > 
> 

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Tue, Apr 04, 2017 at 01:49:29PM +0100, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
> > This series calls the dma ops configuration for the devices
> > at a generic place so that it works for all busses.
> > The dma_configure_ops for a device is now called during
> > the device_attach callback just before the probe of the
> > bus/driver is called. Similarly dma_deconfigure is called during
> > device/driver_detach path.
> > 
> > pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >        |                         |
> > pci_bus_add_device     (device_add/driver_register)
> >        |                         |
> > device_attach           device_initial_probe
> >        |                         |
> > __device_attach_driver    __device_attach_driver
> >        |
> > driver_probe_device
> >        |
> > really_probe
> >        |
> > dma_configure
> > 
> > Similarly on the device/driver_unregister path __device_release_driver is
> > called which inturn calls dma_deconfigure.
> > 
> > Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> > against mainline and linux-next, iommu-next.
> >   
> > * Tested with platform and pci devices for probe deferral
> >   and reprobe on arm64 based platform.
> 
> Sricharan, thanks for keeping this going - I really think we're there now :)

FYI, I re-tested this series with ACPI and tried a merge with IORT
patches queued for 4.12 (via arm64) and there does not seem to be any
merge conflicts so it should be ready to go.

Thanks,
Lorenzo

> Joerg, I realise that at -rc5 time is getting on a bit already, but even
> the non-vintage parts of the series are pretty mature now so it would be
> nice to at least give it a spin in -next. If you don't quite share my
> confidence for landing it in 4.12, please consider it for early next
> cycle to get a full workout.
> 
> Thanks,
> Robin.
> 
> > Previous post of this series [8]. 
> > 
> > Please note that, i have kept the tested/acked tags intact from V8
> > because V9/10 were for more fixes that was added, so the original
> > tags that was given for the functional testing remains the same.
> > 
> >  [V10]
> >      * Rebased on top of 4.11-rc5.
> >  
> >      * Fixed coherent_dma_mask 64bit overflow issue [8]
> >        for OF. The fix for OF was added as a separate
> >        patch#6, since the issue is true even without probe deferral,
> >        but gets reproduced with the probe deferral series.
> >        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
> >        and the fix for dma_configure getting called more than
> >        once for the same device.
> > 
> >      * Also fixed an build issue caught by kbuild robot for
> >        m68k arch. The issue was dma_(de)configure was not
> >        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> > 
> >  [V9]
> >      * Rebased on top of 4.11-rc1.
> > 
> >      * Merged Robin's fixes for legacy binding issue,
> >        pci devices with no iommu-map property and deferencing
> >        of_iommu_table after init.
> >      
> >  [V8]
> >      * Picked up all the acks and tested tags from Marek and
> >        Hanjun for DT and ACPI patches respectively, since
> >        no functional changes was done.
> > 
> >      * Addressed Minor comments Sinan and Bjorn.
> > 
> >      * Added Robin's fix for fixing the deferencing NULL for
> >        of_iommu_table after init in patch #2.
> > 
> >      * Rebased it on top of linux-next
> > 
> >  [V7]
> >      * Updated the subject and commit log for patch #6 as per
> >        comments from Lorenzo. No functional changes.
> > 
> >  [V6]
> >      * Fixed a bug in dma_configure function pointed out by
> >        Robin.
> >      * Reordered the patches as per comments from Robin and
> >        Lorenzo.
> >      * Added Tags.
> > 
> >  [V5]
> >      * Reworked the pci configuration code hanging outside and
> >        pushed it to dma_configure as in PATCH#5,6,7.
> >        Also added a couple of patches that Lorenzo provided for
> >        correcting the Probe deferring mechanism in case of
> >        ACPI devices from here [5].
> > 
> >  [V4]
> >      * Took the reworked patches [2] from Robin's branch and
> >        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> > 
> >      * Added the patches for moving the dma ops configuration of
> >        acpi based devices to probe time as well.
> >  [V3]
> >      * Removed the patch to split dma_masks/dma_ops configuration
> >        separately based on review comments that both masks and ops are
> >        required only during the device probe time.
> > 
> >      * Reworked the series based on Generic DT bindings series.
> > 
> >      * Added call to iommu's remove_device in the cleanup path for arm and
> >        arm64.
> > 
> >      * Removed the notifier trick in arm64 to handle early device
> >        registration.
> > 
> >      * Added reset of dma_ops in cleanup path for arm based on comments.
> > 
> >      * Fixed the pci_iommu_configure path and tested with PCI device as
> >        well.
> >  
> >      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
> >        last post.
> > 
> >      * Fixed few other cosmetic comments.
> >   
> >  [V2]
> >      * Updated the Initial post to call dma_configure/deconfigure from
> >        generic code
> >  
> >      * Added iommu add_device callback from of_iommu_configure path
> > 
> >  [V1]
> >      * Initial post from Laurent Pinchart [1]
> > 
> > [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> > [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> > [3] https://lkml.org/lkml/2016/11/21/141
> > [4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> > [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> > [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> > [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> > [8] http://patchwork.ozlabs.org/patch/743898/
> > 
> > Laurent Pinchart (3):
> >   of: dma: Move range size workaround to of_dma_get_range()
> >   of: dma: Make of_dma_deconfigure() public
> >   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> > 
> > Lorenzo Pieralisi (2):
> >   ACPI/IORT: Add function to check SMMUs drivers presence
> >   ACPI/IORT: Remove linker section for IORT entries probing
> > 
> > Robin Murphy (3):
> >   iommu/of: Refactor of_iommu_configure() for error handling
> >   iommu/of: Prepare for deferred IOMMU configuration
> >   iommu/arm-smmu: Clean up early-probing workarounds
> > 
> > Sricharan R (4):
> >   of: device: Fix overflow of coherent_dma_mask
> >   of/acpi: Configure dma operations at probe time for platform/amba/pci
> >     bus devices
> >   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
> >     error
> >   arm64: dma-mapping: Remove the notifier trick to handle early setting
> >     of dma_ops
> > 
> >  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
> >  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
> >  drivers/acpi/glue.c               |   5 --
> >  drivers/acpi/scan.c               |  11 ++-
> >  drivers/base/dd.c                 |   9 +++
> >  drivers/base/dma-mapping.c        |  41 +++++++++++
> >  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
> >  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
> >  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
> >  drivers/of/address.c              |  20 +++++-
> >  drivers/of/device.c               |  36 +++++-----
> >  drivers/of/platform.c             |  10 +--
> >  drivers/pci/probe.c               |  28 --------
> >  include/acpi/acpi_bus.h           |   2 +-
> >  include/asm-generic/vmlinux.lds.h |   1 -
> >  include/linux/acpi.h              |   7 +-
> >  include/linux/acpi_iort.h         |   3 -
> >  include/linux/dma-mapping.h       |  12 ++++
> >  include/linux/of_device.h         |  10 ++-
> >  19 files changed, 329 insertions(+), 338 deletions(-)
> > 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 00/12] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Tue, Apr 04, 2017 at 01:49:29PM +0100, Robin Murphy wrote:
> On 04/04/17 11:18, Sricharan R wrote:
> > This series calls the dma ops configuration for the devices
> > at a generic place so that it works for all busses.
> > The dma_configure_ops for a device is now called during
> > the device_attach callback just before the probe of the
> > bus/driver is called. Similarly dma_deconfigure is called during
> > device/driver_detach path.
> > 
> > pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >        |                         |
> > pci_bus_add_device     (device_add/driver_register)
> >        |                         |
> > device_attach           device_initial_probe
> >        |                         |
> > __device_attach_driver    __device_attach_driver
> >        |
> > driver_probe_device
> >        |
> > really_probe
> >        |
> > dma_configure
> > 
> > Similarly on the device/driver_unregister path __device_release_driver is
> > called which inturn calls dma_deconfigure.
> > 
> > Rebased the series against mainline 4.11-rc5. Applies and builds cleanly
> > against mainline and linux-next, iommu-next.
> >   
> > * Tested with platform and pci devices for probe deferral
> >   and reprobe on arm64 based platform.
> 
> Sricharan, thanks for keeping this going - I really think we're there now :)

FYI, I re-tested this series with ACPI and tried a merge with IORT
patches queued for 4.12 (via arm64) and there does not seem to be any
merge conflicts so it should be ready to go.

Thanks,
Lorenzo

> Joerg, I realise that at -rc5 time is getting on a bit already, but even
> the non-vintage parts of the series are pretty mature now so it would be
> nice to at least give it a spin in -next. If you don't quite share my
> confidence for landing it in 4.12, please consider it for early next
> cycle to get a full workout.
> 
> Thanks,
> Robin.
> 
> > Previous post of this series [8]. 
> > 
> > Please note that, i have kept the tested/acked tags intact from V8
> > because V9/10 were for more fixes that was added, so the original
> > tags that was given for the functional testing remains the same.
> > 
> >  [V10]
> >      * Rebased on top of 4.11-rc5.
> >  
> >      * Fixed coherent_dma_mask 64bit overflow issue [8]
> >        for OF. The fix for OF was added as a separate
> >        patch#6, since the issue is true even without probe deferral,
> >        but gets reproduced with the probe deferral series.
> >        Added Lorenzo's ACPI fix for coherent_dma_mask overflow
> >        and the fix for dma_configure getting called more than
> >        once for the same device.
> > 
> >      * Also fixed an build issue caught by kbuild robot for
> >        m68k arch. The issue was dma_(de)configure was not
> >        getting defined for !CONFIG_HAS_DMA, so fixed that as well.
> > 
> >  [V9]
> >      * Rebased on top of 4.11-rc1.
> > 
> >      * Merged Robin's fixes for legacy binding issue,
> >        pci devices with no iommu-map property and deferencing
> >        of_iommu_table after init.
> >      
> >  [V8]
> >      * Picked up all the acks and tested tags from Marek and
> >        Hanjun for DT and ACPI patches respectively, since
> >        no functional changes was done.
> > 
> >      * Addressed Minor comments Sinan and Bjorn.
> > 
> >      * Added Robin's fix for fixing the deferencing NULL for
> >        of_iommu_table after init in patch #2.
> > 
> >      * Rebased it on top of linux-next
> > 
> >  [V7]
> >      * Updated the subject and commit log for patch #6 as per
> >        comments from Lorenzo. No functional changes.
> > 
> >  [V6]
> >      * Fixed a bug in dma_configure function pointed out by
> >        Robin.
> >      * Reordered the patches as per comments from Robin and
> >        Lorenzo.
> >      * Added Tags.
> > 
> >  [V5]
> >      * Reworked the pci configuration code hanging outside and
> >        pushed it to dma_configure as in PATCH#5,6,7.
> >        Also added a couple of patches that Lorenzo provided for
> >        correcting the Probe deferring mechanism in case of
> >        ACPI devices from here [5].
> > 
> >  [V4]
> >      * Took the reworked patches [2] from Robin's branch and
> >        rebased on top of Lorenzo's ACPI IORT ARM support series [3].
> > 
> >      * Added the patches for moving the dma ops configuration of
> >        acpi based devices to probe time as well.
> >  [V3]
> >      * Removed the patch to split dma_masks/dma_ops configuration
> >        separately based on review comments that both masks and ops are
> >        required only during the device probe time.
> > 
> >      * Reworked the series based on Generic DT bindings series.
> > 
> >      * Added call to iommu's remove_device in the cleanup path for arm and
> >        arm64.
> > 
> >      * Removed the notifier trick in arm64 to handle early device
> >        registration.
> > 
> >      * Added reset of dma_ops in cleanup path for arm based on comments.
> > 
> >      * Fixed the pci_iommu_configure path and tested with PCI device as
> >        well.
> >  
> >      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
> >        last post.
> > 
> >      * Fixed few other cosmetic comments.
> >   
> >  [V2]
> >      * Updated the Initial post to call dma_configure/deconfigure from
> >        generic code
> >  
> >      * Added iommu add_device callback from of_iommu_configure path
> > 
> >  [V1]
> >      * Initial post from Laurent Pinchart [1]
> > 
> > [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> > [2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
> > [3] https://lkml.org/lkml/2016/11/21/141
> > [4] https://www.mail-archive.com/iommu at xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
> > [5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
> > [6] http://www.spinics.net/lists/linux-pci/msg57992.html
> > [7] https://www.spinics.net/lists/arm-kernel/msg556209.html
> > [8] http://patchwork.ozlabs.org/patch/743898/
> > 
> > Laurent Pinchart (3):
> >   of: dma: Move range size workaround to of_dma_get_range()
> >   of: dma: Make of_dma_deconfigure() public
> >   iommu: of: Handle IOMMU lookup failure with deferred probing or error
> > 
> > Lorenzo Pieralisi (2):
> >   ACPI/IORT: Add function to check SMMUs drivers presence
> >   ACPI/IORT: Remove linker section for IORT entries probing
> > 
> > Robin Murphy (3):
> >   iommu/of: Refactor of_iommu_configure() for error handling
> >   iommu/of: Prepare for deferred IOMMU configuration
> >   iommu/arm-smmu: Clean up early-probing workarounds
> > 
> > Sricharan R (4):
> >   of: device: Fix overflow of coherent_dma_mask
> >   of/acpi: Configure dma operations at probe time for platform/amba/pci
> >     bus devices
> >   drivers: acpi: Handle IOMMU lookup failure with deferred probing or
> >     error
> >   arm64: dma-mapping: Remove the notifier trick to handle early setting
> >     of dma_ops
> > 
> >  arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
> >  drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
> >  drivers/acpi/glue.c               |   5 --
> >  drivers/acpi/scan.c               |  11 ++-
> >  drivers/base/dd.c                 |   9 +++
> >  drivers/base/dma-mapping.c        |  41 +++++++++++
> >  drivers/iommu/arm-smmu-v3.c       |  46 +-----------
> >  drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
> >  drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
> >  drivers/of/address.c              |  20 +++++-
> >  drivers/of/device.c               |  36 +++++-----
> >  drivers/of/platform.c             |  10 +--
> >  drivers/pci/probe.c               |  28 --------
> >  include/acpi/acpi_bus.h           |   2 +-
> >  include/asm-generic/vmlinux.lds.h |   1 -
> >  include/linux/acpi.h              |   7 +-
> >  include/linux/acpi_iort.h         |   3 -
> >  include/linux/dma-mapping.h       |  12 ++++
> >  include/linux/of_device.h         |  10 ++-
> >  19 files changed, 329 insertions(+), 338 deletions(-)
> > 
> 

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Frank Rowand]

On 04/04/17 03:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Please drop this patch from the series.  I just now sent Rob an
alternative that reflects other changes that have occurred since
this patch series was first created many, many moons ago
(https://lkml.org/lkml/2017/4/6/53).

Thanks,

Frank

> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Frank Rowand]

On 04/04/17 03:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Please drop this patch from the series.  I just now sent Rob an
alternative that reflects other changes that have occurred since
this patch series was first created many, many moons ago
(https://lkml.org/lkml/2017/4/6/53).

Thanks,

Frank

> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Frank Rowand]

On 04/04/17 03:18, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Invalid dma-ranges values should be worked around when retrieving the
> DMA range in of_dma_get_range(), not by all callers of the function.
> This isn't much of a problem now that we have a single caller, but that
> situation will change when moving DMA configuration to device probe
> time.

Please drop this patch from the series.  I just now sent Rob an
alternative that reflects other changes that have occurred since
this patch series was first created many, many moons ago
(https://lkml.org/lkml/2017/4/6/53).

Thanks,

Frank

> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  drivers/of/address.c | 20 ++++++++++++++++++--
>  drivers/of/device.c  | 15 ---------------
>  2 files changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 02b2903..6aeb816 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>   *	CPU addr (phys_addr_t)	: pna cells
>   *	size			: nsize cells
>   *
> - * It returns -ENODEV if "dma-ranges" property was not found
> - * for this device in DT.
> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>   */
>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>  {
> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>  	*dma_addr = dmaaddr;
>  
>  	*size = of_read_number(ranges + naddr + pna, nsize);
> +	/*
> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
> +	 * those incorrect DT by computing the size as mask + 1.
> +	 */
> +	if (*size & 1) {
> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
> +			__func__, *size, np->full_name);
> +		*size = *size + 1;
> +	}
> +
> +	if (!*size) {
> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
> +		       __func__, np->full_name);
> +		ret = -EINVAL;
> +		goto out;
> +	}
>  
>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>  		 *dma_addr, *paddr, *size);
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index b1e6beb..09dedd0 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		size = dev->coherent_dma_mask + 1;
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
> -
> -		/*
> -		 * Add a work around to treat the size as mask + 1 in case
> -		 * it is defined in DT as a mask.
> -		 */
> -		if (size & 1) {
> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
> -				 size);
> -			size = size + 1;
> -		}
> -
> -		if (!size) {
> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
> -			return;
> -		}
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>  	}
>  
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/04/17 03:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 

NACK.

Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
size is also used in of_dma_configure() before calling arch_setup_dma_ops():

        dev->coherent_dma_mask = min(dev->coherent_dma_mask,
                                     DMA_BIT_MASK(ilog2(dma_addr + size)));
        *dev->dma_mask = min((*dev->dma_mask),
                             DMA_BIT_MASK(ilog2(dma_addr + size)));

which would be incorrect for size == 0xffffffffffffffffULL when
dma_addr != 0.  So the proposed fix really is not papering over
the base problem very well.

I agree that the proper solution involves passing a mask instead
of a size to arch_setup_dma_ops().

-Frank

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/04/17 03:18, Sricharan R wrote:
> Size of the dma-range is calculated as coherent_dma_mask + 1
> and passed to arch_setup_dma_ops further. It overflows when
> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
> resulting in size getting passed as 0 wrongly. Fix this by
> passsing in max(mask, mask + 1). Note that in this case
> when the mask is set to full 64bits, we will be passing the mask
> itself to arch_setup_dma_ops instead of the size. The real fix
> for this should be to make arch_setup_dma_ops receive the
> mask and handle it, to be done in the future.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/of/device.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..c2ae6bb 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>  	if (ret < 0) {
>  		dma_addr = offset = 0;
> -		size = dev->coherent_dma_mask + 1;
> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>  	} else {
>  		offset = PFN_DOWN(paddr - dma_addr);
>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
> 

NACK.

Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
size is also used in of_dma_configure() before calling arch_setup_dma_ops():

        dev->coherent_dma_mask = min(dev->coherent_dma_mask,
                                     DMA_BIT_MASK(ilog2(dma_addr + size)));
        *dev->dma_mask = min((*dev->dma_mask),
                             DMA_BIT_MASK(ilog2(dma_addr + size)));

which would be incorrect for size == 0xffffffffffffffffULL when
dma_addr != 0.  So the proposed fix really is not papering over
the base problem very well.

I agree that the proper solution involves passing a mask instead
of a size to arch_setup_dma_ops().

-Frank

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

Hi Frank,

On 4/6/2017 11:54 AM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
>
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

ok, would drop this.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

Hi Frank,

On 4/6/2017 11:54 AM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
>
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

ok, would drop this.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

Hi Frank,

On 4/6/2017 11:54 AM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
>
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

ok, would drop this.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

Hi Frank,

On 4/6/2017 11:54 AM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
>
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

ok, would drop this.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

Hi Frank,

On 06/04/17 07:24, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
> 
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

Fine by me too - if we can get rid of that dubious fixup altogether, all
the better. Feel free to transfer my Reviewed-by to your patch if you like.

Thanks,
Robin.

> Thanks,
> 
> Frank
> 
>>
>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>> ---
>>  drivers/of/address.c | 20 ++++++++++++++++++--
>>  drivers/of/device.c  | 15 ---------------
>>  2 files changed, 18 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/of/address.c b/drivers/of/address.c
>> index 02b2903..6aeb816 100644
>> --- a/drivers/of/address.c
>> +++ b/drivers/of/address.c
>> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>>   *	CPU addr (phys_addr_t)	: pna cells
>>   *	size			: nsize cells
>>   *
>> - * It returns -ENODEV if "dma-ranges" property was not found
>> - * for this device in DT.
>> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
>> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>>   */
>>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>>  {
>> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>>  	*dma_addr = dmaaddr;
>>  
>>  	*size = of_read_number(ranges + naddr + pna, nsize);
>> +	/*
>> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
>> +	 * those incorrect DT by computing the size as mask + 1.
>> +	 */
>> +	if (*size & 1) {
>> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
>> +			__func__, *size, np->full_name);
>> +		*size = *size + 1;
>> +	}
>> +
>> +	if (!*size) {
>> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
>> +		       __func__, np->full_name);
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
>>  
>>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>>  		 *dma_addr, *paddr, *size);
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index b1e6beb..09dedd0 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  		size = dev->coherent_dma_mask + 1;
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>> -
>> -		/*
>> -		 * Add a work around to treat the size as mask + 1 in case
>> -		 * it is defined in DT as a mask.
>> -		 */
>> -		if (size & 1) {
>> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
>> -				 size);
>> -			size = size + 1;
>> -		}
>> -
>> -		if (!size) {
>> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
>> -			return;
>> -		}
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>  	}
>>  
>>

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

Hi Frank,

On 06/04/17 07:24, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
> 
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

Fine by me too - if we can get rid of that dubious fixup altogether, all
the better. Feel free to transfer my Reviewed-by to your patch if you like.

Thanks,
Robin.

> Thanks,
> 
> Frank
> 
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> ---
>>  drivers/of/address.c | 20 ++++++++++++++++++--
>>  drivers/of/device.c  | 15 ---------------
>>  2 files changed, 18 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/of/address.c b/drivers/of/address.c
>> index 02b2903..6aeb816 100644
>> --- a/drivers/of/address.c
>> +++ b/drivers/of/address.c
>> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>>   *	CPU addr (phys_addr_t)	: pna cells
>>   *	size			: nsize cells
>>   *
>> - * It returns -ENODEV if "dma-ranges" property was not found
>> - * for this device in DT.
>> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
>> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>>   */
>>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>>  {
>> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>>  	*dma_addr = dmaaddr;
>>  
>>  	*size = of_read_number(ranges + naddr + pna, nsize);
>> +	/*
>> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
>> +	 * those incorrect DT by computing the size as mask + 1.
>> +	 */
>> +	if (*size & 1) {
>> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
>> +			__func__, *size, np->full_name);
>> +		*size = *size + 1;
>> +	}
>> +
>> +	if (!*size) {
>> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
>> +		       __func__, np->full_name);
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
>>  
>>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>>  		 *dma_addr, *paddr, *size);
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index b1e6beb..09dedd0 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  		size = dev->coherent_dma_mask + 1;
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>> -
>> -		/*
>> -		 * Add a work around to treat the size as mask + 1 in case
>> -		 * it is defined in DT as a mask.
>> -		 */
>> -		if (size & 1) {
>> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
>> -				 size);
>> -			size = size + 1;
>> -		}
>> -
>> -		if (!size) {
>> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
>> -			return;
>> -		}
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>  	}
>>  
>>

Re: [PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

Hi Frank,

On 06/04/17 07:24, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
> 
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

Fine by me too - if we can get rid of that dubious fixup altogether, all
the better. Feel free to transfer my Reviewed-by to your patch if you like.

Thanks,
Robin.

> Thanks,
> 
> Frank
> 
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> ---
>>  drivers/of/address.c | 20 ++++++++++++++++++--
>>  drivers/of/device.c  | 15 ---------------
>>  2 files changed, 18 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/of/address.c b/drivers/of/address.c
>> index 02b2903..6aeb816 100644
>> --- a/drivers/of/address.c
>> +++ b/drivers/of/address.c
>> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>>   *	CPU addr (phys_addr_t)	: pna cells
>>   *	size			: nsize cells
>>   *
>> - * It returns -ENODEV if "dma-ranges" property was not found
>> - * for this device in DT.
>> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
>> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>>   */
>>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>>  {
>> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>>  	*dma_addr = dmaaddr;
>>  
>>  	*size = of_read_number(ranges + naddr + pna, nsize);
>> +	/*
>> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
>> +	 * those incorrect DT by computing the size as mask + 1.
>> +	 */
>> +	if (*size & 1) {
>> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
>> +			__func__, *size, np->full_name);
>> +		*size = *size + 1;
>> +	}
>> +
>> +	if (!*size) {
>> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
>> +		       __func__, np->full_name);
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
>>  
>>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>>  		 *dma_addr, *paddr, *size);
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index b1e6beb..09dedd0 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  		size = dev->coherent_dma_mask + 1;
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>> -
>> -		/*
>> -		 * Add a work around to treat the size as mask + 1 in case
>> -		 * it is defined in DT as a mask.
>> -		 */
>> -		if (size & 1) {
>> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
>> -				 size);
>> -			size = size + 1;
>> -		}
>> -
>> -		if (!size) {
>> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
>> -			return;
>> -		}
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>  	}
>>  
>>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 03/12] of: dma: Move range size workaround to of_dma_get_range()

[Robin Murphy]

Hi Frank,

On 06/04/17 07:24, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Invalid dma-ranges values should be worked around when retrieving the
>> DMA range in of_dma_get_range(), not by all callers of the function.
>> This isn't much of a problem now that we have a single caller, but that
>> situation will change when moving DMA configuration to device probe
>> time.
> 
> Please drop this patch from the series.  I just now sent Rob an
> alternative that reflects other changes that have occurred since
> this patch series was first created many, many moons ago
> (https://lkml.org/lkml/2017/4/6/53).

Fine by me too - if we can get rid of that dubious fixup altogether, all
the better. Feel free to transfer my Reviewed-by to your patch if you like.

Thanks,
Robin.

> Thanks,
> 
> Frank
> 
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> ---
>>  drivers/of/address.c | 20 ++++++++++++++++++--
>>  drivers/of/device.c  | 15 ---------------
>>  2 files changed, 18 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/of/address.c b/drivers/of/address.c
>> index 02b2903..6aeb816 100644
>> --- a/drivers/of/address.c
>> +++ b/drivers/of/address.c
>> @@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
>>   *	CPU addr (phys_addr_t)	: pna cells
>>   *	size			: nsize cells
>>   *
>> - * It returns -ENODEV if "dma-ranges" property was not found
>> - * for this device in DT.
>> + * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
>> + * this device in DT, or -EINVAL if the CPU address or size is invalid.
>>   */
>>  int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
>>  {
>> @@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
>>  	*dma_addr = dmaaddr;
>>  
>>  	*size = of_read_number(ranges + naddr + pna, nsize);
>> +	/*
>> +	 * DT nodes sometimes incorrectly set the size as a mask. Work around
>> +	 * those incorrect DT by computing the size as mask + 1.
>> +	 */
>> +	if (*size & 1) {
>> +		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
>> +			__func__, *size, np->full_name);
>> +		*size = *size + 1;
>> +	}
>> +
>> +	if (!*size) {
>> +		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
>> +		       __func__, np->full_name);
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
>>  
>>  	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
>>  		 *dma_addr, *paddr, *size);
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index b1e6beb..09dedd0 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  		size = dev->coherent_dma_mask + 1;
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>> -
>> -		/*
>> -		 * Add a work around to treat the size as mask + 1 in case
>> -		 * it is defined in DT as a mask.
>> -		 */
>> -		if (size & 1) {
>> -			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
>> -				 size);
>> -			size = size + 1;
>> -		}
>> -
>> -		if (!size) {
>> -			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
>> -			return;
>> -		}
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>  	}
>>  
>>

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 08:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.

I'm not sure I agree there. Granted, there exist many more problematic
aspects than are dealt with here (I've got more patches cooking to sort
out some of the other issues we have with dma-ranges), but considering
size specifically:

- It is not possible to explicitly specify a range with a size of 2^64
in DT. If someone does specify a size of 0, they've done a silly thing
and should not be surprised that it ends badly.

- It *is* perfectly legitimate for bus code (or a previous device
driver, once we start coming here at probe time) to have set a device's
DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
and infers an invalid size of 0 from that, breaking things in the
process, that is this code's fault alone. It just so happens that
nothing managed to trigger the latent problem until patch #7 here shakes
up the callsites.

Yes, wacky impossible base + size combinations in DT were a theoretical
problem before, and remain a theoretical problem, but also fall into the
"how did you ever expect this to work?" category. There's certainly
plenty more we can do to improve the DT parsing/validation, but that
still doesn't apply to this path where the information is *not* coming
from the DT at all.

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().

Having started writing that patch too, I can tell you it's a big bugger
touching multiple architectures and fixing up various drivers doing
stupid things, hence why I'm happy with this point fix being the lesser
of two evils in terms of not holding up this mostly-orthogonal series.

Robin.

> 
> -Frank
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 08:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.

I'm not sure I agree there. Granted, there exist many more problematic
aspects than are dealt with here (I've got more patches cooking to sort
out some of the other issues we have with dma-ranges), but considering
size specifically:

- It is not possible to explicitly specify a range with a size of 2^64
in DT. If someone does specify a size of 0, they've done a silly thing
and should not be surprised that it ends badly.

- It *is* perfectly legitimate for bus code (or a previous device
driver, once we start coming here at probe time) to have set a device's
DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
and infers an invalid size of 0 from that, breaking things in the
process, that is this code's fault alone. It just so happens that
nothing managed to trigger the latent problem until patch #7 here shakes
up the callsites.

Yes, wacky impossible base + size combinations in DT were a theoretical
problem before, and remain a theoretical problem, but also fall into the
"how did you ever expect this to work?" category. There's certainly
plenty more we can do to improve the DT parsing/validation, but that
still doesn't apply to this path where the information is *not* coming
from the DT at all.

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().

Having started writing that patch too, I can tell you it's a big bugger
touching multiple architectures and fixing up various drivers doing
stupid things, hence why I'm happy with this point fix being the lesser
of two evils in terms of not holding up this mostly-orthogonal series.

Robin.

> 
> -Frank
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 08:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.

I'm not sure I agree there. Granted, there exist many more problematic
aspects than are dealt with here (I've got more patches cooking to sort
out some of the other issues we have with dma-ranges), but considering
size specifically:

- It is not possible to explicitly specify a range with a size of 2^64
in DT. If someone does specify a size of 0, they've done a silly thing
and should not be surprised that it ends badly.

- It *is* perfectly legitimate for bus code (or a previous device
driver, once we start coming here at probe time) to have set a device's
DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
and infers an invalid size of 0 from that, breaking things in the
process, that is this code's fault alone. It just so happens that
nothing managed to trigger the latent problem until patch #7 here shakes
up the callsites.

Yes, wacky impossible base + size combinations in DT were a theoretical
problem before, and remain a theoretical problem, but also fall into the
"how did you ever expect this to work?" category. There's certainly
plenty more we can do to improve the DT parsing/validation, but that
still doesn't apply to this path where the information is *not* coming
from the DT at all.

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().

Having started writing that patch too, I can tell you it's a big bugger
touching multiple architectures and fixing up various drivers doing
stupid things, hence why I'm happy with this point fix being the lesser
of two evils in terms of not holding up this mostly-orthogonal series.

Robin.

> 
> -Frank
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 08:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.

I'm not sure I agree there. Granted, there exist many more problematic
aspects than are dealt with here (I've got more patches cooking to sort
out some of the other issues we have with dma-ranges), but considering
size specifically:

- It is not possible to explicitly specify a range with a size of 2^64
in DT. If someone does specify a size of 0, they've done a silly thing
and should not be surprised that it ends badly.

- It *is* perfectly legitimate for bus code (or a previous device
driver, once we start coming here at probe time) to have set a device's
DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
and infers an invalid size of 0 from that, breaking things in the
process, that is this code's fault alone. It just so happens that
nothing managed to trigger the latent problem until patch #7 here shakes
up the callsites.

Yes, wacky impossible base + size combinations in DT were a theoretical
problem before, and remain a theoretical problem, but also fall into the
"how did you ever expect this to work?" category. There's certainly
plenty more we can do to improve the DT parsing/validation, but that
still doesn't apply to this path where the information is *not* coming
from the DT at all.

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().

Having started writing that patch too, I can tell you it's a big bugger
touching multiple architectures and fixing up various drivers doing
stupid things, hence why I'm happy with this point fix being the lesser
of two evils in terms of not holding up this mostly-orthogonal series.

Robin.

> 
> -Frank
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/6/2017 12:31 PM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
>
> NACK.
>
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.
>

Ok, but with your fix for of_dma_get_range and the above fix,
dma_addr will be '0' when size = 0xffffffffffffffffULL,
but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Regards,
  Sricharan

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/6/2017 12:31 PM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
>
> NACK.
>
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.
>

Ok, but with your fix for of_dma_get_range and the above fix,
dma_addr will be '0' when size = 0xffffffffffffffffULL,
but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Regards,
  Sricharan

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/6/2017 12:31 PM, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
>
> NACK.
>
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over
> the base problem very well.
>

Ok, but with your fix for of_dma_get_range and the above fix,
dma_addr will be '0' when size = 0xffffffffffffffffULL,
but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Regards,
  Sricharan

> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().
>

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Rob Herring]

On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>              dma_addr = offset = 0;
>>> -            size = dev->coherent_dma_mask + 1;
>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
>
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
and fix these up as 2^32 and 2^64 sizes.

Rob

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Rob Herring]

On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>              dma_addr = offset = 0;
>>> -            size = dev->coherent_dma_mask + 1;
>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
>
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
and fix these up as 2^32 and 2^64 sizes.

Rob

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Rob Herring]

On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>              dma_addr = offset = 0;
>>> -            size = dev->coherent_dma_mask + 1;
>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
>
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
and fix these up as 2^32 and 2^64 sizes.

Rob

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Rob Herring]

On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>              dma_addr = offset = 0;
>>> -            size = dev->coherent_dma_mask + 1;
>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
>
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
and fix these up as 2^32 and 2^64 sizes.

Rob

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Rob Herring]

On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>              dma_addr = offset = 0;
>>> -            size = dev->coherent_dma_mask + 1;
>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
>
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
and fix these up as 2^32 and 2^64 sizes.

Rob

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 14:56, Rob Herring wrote:
> On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:
>> On 06/04/17 08:01, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>              dma_addr = offset = 0;
>>>> -            size = dev->coherent_dma_mask + 1;
>>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>
>> I'm not sure I agree there. Granted, there exist many more problematic
>> aspects than are dealt with here (I've got more patches cooking to sort
>> out some of the other issues we have with dma-ranges), but considering
>> size specifically:
>>
>> - It is not possible to explicitly specify a range with a size of 2^64
>> in DT. If someone does specify a size of 0, they've done a silly thing
>> and should not be surprised that it ends badly.
> 
> And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
> and fix these up as 2^32 and 2^64 sizes.

...which is what Frank's patch gets rid of.

Robin.

> 
> Rob
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 14:56, Rob Herring wrote:
> On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
>> On 06/04/17 08:01, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>              dma_addr = offset = 0;
>>>> -            size = dev->coherent_dma_mask + 1;
>>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>
>> I'm not sure I agree there. Granted, there exist many more problematic
>> aspects than are dealt with here (I've got more patches cooking to sort
>> out some of the other issues we have with dma-ranges), but considering
>> size specifically:
>>
>> - It is not possible to explicitly specify a range with a size of 2^64
>> in DT. If someone does specify a size of 0, they've done a silly thing
>> and should not be surprised that it ends badly.
> 
> And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
> and fix these up as 2^32 and 2^64 sizes.

...which is what Frank's patch gets rid of.

Robin.

> 
> Rob
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 14:56, Rob Herring wrote:
> On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
>> On 06/04/17 08:01, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>              dma_addr = offset = 0;
>>>> -            size = dev->coherent_dma_mask + 1;
>>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>
>> I'm not sure I agree there. Granted, there exist many more problematic
>> aspects than are dealt with here (I've got more patches cooking to sort
>> out some of the other issues we have with dma-ranges), but considering
>> size specifically:
>>
>> - It is not possible to explicitly specify a range with a size of 2^64
>> in DT. If someone does specify a size of 0, they've done a silly thing
>> and should not be surprised that it ends badly.
> 
> And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
> and fix these up as 2^32 and 2^64 sizes.

...which is what Frank's patch gets rid of.

Robin.

> 
> Rob
> 

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 14:56, Rob Herring wrote:
> On Thu, Apr 6, 2017 at 5:24 AM, Robin Murphy <robin.murphy@arm.com> wrote:
>> On 06/04/17 08:01, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>              dma_addr = offset = 0;
>>>> -            size = dev->coherent_dma_mask + 1;
>>>> +            size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>              offset = PFN_DOWN(paddr - dma_addr);
>>>>              dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>
>> I'm not sure I agree there. Granted, there exist many more problematic
>> aspects than are dealt with here (I've got more patches cooking to sort
>> out some of the other issues we have with dma-ranges), but considering
>> size specifically:
>>
>> - It is not possible to explicitly specify a range with a size of 2^64
>> in DT. If someone does specify a size of 0, they've done a silly thing
>> and should not be surprised that it ends badly.
> 
> And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges
> and fix these up as 2^32 and 2^64 sizes.

...which is what Frank's patch gets rid of.

Robin.

> 
> Rob
> 

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Frank Rowand]

On 04/04/17 18:23, Rob Herring wrote:
> On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
> 
> For patches 3, 4, 6, 7, 8:
> 
> Acked-by: Rob Herring <robh@kernel.org>
> .
> 

And a few hours later, I had the opposite opinion on a couple of
patches, so there is some ongoing discussion working this out.

-Frank

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Frank Rowand]

On 04/04/17 18:23, Rob Herring wrote:
> On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
> 
> For patches 3, 4, 6, 7, 8:
> 
> Acked-by: Rob Herring <robh@kernel.org>
> .
> 

And a few hours later, I had the opposite opinion on a couple of
patches, so there is some ongoing discussion working this out.

-Frank

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Frank Rowand]

On 04/04/17 18:23, Rob Herring wrote:
> On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
> 
> For patches 3, 4, 6, 7, 8:
> 
> Acked-by: Rob Herring <robh@kernel.org>
> .
> 

And a few hours later, I had the opposite opinion on a couple of
patches, so there is some ongoing discussion working this out.

-Frank

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V10 00/12] IOMMU probe deferral support

[Frank Rowand]

On 04/04/17 18:23, Rob Herring wrote:
> On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
> 
> For patches 3, 4, 6, 7, 8:
> 
> Acked-by: Rob Herring <robh@kernel.org>
> .
> 

And a few hours later, I had the opposite opinion on a couple of
patches, so there is some ongoing discussion working this out.

-Frank

[PATCH V10 00/12] IOMMU probe deferral support

[Frank Rowand]

On 04/04/17 18:23, Rob Herring wrote:
> On Tue, Apr 4, 2017 at 5:18 AM, Sricharan R <sricharan@codeaurora.org> wrote:
>> This series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
> 
> For patches 3, 4, 6, 7, 8:
> 
> Acked-by: Rob Herring <robh@kernel.org>
> .
> 

And a few hours later, I had the opposite opinion on a couple of
patches, so there is some ongoing discussion working this out.

-Frank

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 03:24, Robin Murphy wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>  	if (ret < 0) {
>>>  		dma_addr = offset = 0;
>>> -		size = dev->coherent_dma_mask + 1;
>>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>  	} else {
>>>  		offset = PFN_DOWN(paddr - dma_addr);
>>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
> 
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
> 
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.
> 
> - It *is* perfectly legitimate for bus code (or a previous device
> driver, once we start coming here at probe time) to have set a device's
> DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
> and infers an invalid size of 0 from that, breaking things in the
> process, that is this code's fault alone. It just so happens that
> nothing managed to trigger the latent problem until patch #7 here shakes
> up the callsites.

The existing code that uses size does not appear capable of dealing with
the case of DMA mask of 0xffffffffffffffffULL since 2^64 does not fit
into size.

The code affected by the DMA mask is not within my area of knowledge, so
take the following with a grain of salt.  If a DMA mask of
0xffffffffffffffffULL is provided, would the code still work without error
(though with reduced capability) if the mask was changed to
0xefffffffffffffffULL?  I would guess that the location to do so would
be where dev->coherent_dma_mask is set, or some other location that
is not of_dma_configure().  This would just be a temporary workaround.


> Yes, wacky impossible base + size combinations in DT were a theoretical
> problem before, and remain a theoretical problem, but also fall into the
> "how did you ever expect this to work?" category. There's certainly
> plenty more we can do to improve the DT parsing/validation, but that
> still doesn't apply to this path where the information is *not* coming
> from the DT at all.
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
> 
> Having started writing that patch too, I can tell you it's a big bugger
> touching multiple architectures and fixing up various drivers doing
> stupid things, hence why I'm happy with this point fix being the lesser
> of two evils in terms of not holding up this mostly-orthogonal series.
> 
> Robin.
> 
>>
>> -Frank
>>
> 
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 03:24, Robin Murphy wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>  	if (ret < 0) {
>>>  		dma_addr = offset = 0;
>>> -		size = dev->coherent_dma_mask + 1;
>>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>  	} else {
>>>  		offset = PFN_DOWN(paddr - dma_addr);
>>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
> 
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
> 
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.
> 
> - It *is* perfectly legitimate for bus code (or a previous device
> driver, once we start coming here at probe time) to have set a device's
> DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
> and infers an invalid size of 0 from that, breaking things in the
> process, that is this code's fault alone. It just so happens that
> nothing managed to trigger the latent problem until patch #7 here shakes
> up the callsites.

The existing code that uses size does not appear capable of dealing with
the case of DMA mask of 0xffffffffffffffffULL since 2^64 does not fit
into size.

The code affected by the DMA mask is not within my area of knowledge, so
take the following with a grain of salt.  If a DMA mask of
0xffffffffffffffffULL is provided, would the code still work without error
(though with reduced capability) if the mask was changed to
0xefffffffffffffffULL?  I would guess that the location to do so would
be where dev->coherent_dma_mask is set, or some other location that
is not of_dma_configure().  This would just be a temporary workaround.


> Yes, wacky impossible base + size combinations in DT were a theoretical
> problem before, and remain a theoretical problem, but also fall into the
> "how did you ever expect this to work?" category. There's certainly
> plenty more we can do to improve the DT parsing/validation, but that
> still doesn't apply to this path where the information is *not* coming
> from the DT at all.
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
> 
> Having started writing that patch too, I can tell you it's a big bugger
> touching multiple architectures and fixing up various drivers doing
> stupid things, hence why I'm happy with this point fix being the lesser
> of two evils in terms of not holding up this mostly-orthogonal series.
> 
> Robin.
> 
>>
>> -Frank
>>
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 03:24, Robin Murphy wrote:
> On 06/04/17 08:01, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>  	if (ret < 0) {
>>>  		dma_addr = offset = 0;
>>> -		size = dev->coherent_dma_mask + 1;
>>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>  	} else {
>>>  		offset = PFN_DOWN(paddr - dma_addr);
>>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
> 
> I'm not sure I agree there. Granted, there exist many more problematic
> aspects than are dealt with here (I've got more patches cooking to sort
> out some of the other issues we have with dma-ranges), but considering
> size specifically:
> 
> - It is not possible to explicitly specify a range with a size of 2^64
> in DT. If someone does specify a size of 0, they've done a silly thing
> and should not be surprised that it ends badly.
> 
> - It *is* perfectly legitimate for bus code (or a previous device
> driver, once we start coming here at probe time) to have set a device's
> DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows
> and infers an invalid size of 0 from that, breaking things in the
> process, that is this code's fault alone. It just so happens that
> nothing managed to trigger the latent problem until patch #7 here shakes
> up the callsites.

The existing code that uses size does not appear capable of dealing with
the case of DMA mask of 0xffffffffffffffffULL since 2^64 does not fit
into size.

The code affected by the DMA mask is not within my area of knowledge, so
take the following with a grain of salt.  If a DMA mask of
0xffffffffffffffffULL is provided, would the code still work without error
(though with reduced capability) if the mask was changed to
0xefffffffffffffffULL?  I would guess that the location to do so would
be where dev->coherent_dma_mask is set, or some other location that
is not of_dma_configure().  This would just be a temporary workaround.


> Yes, wacky impossible base + size combinations in DT were a theoretical
> problem before, and remain a theoretical problem, but also fall into the
> "how did you ever expect this to work?" category. There's certainly
> plenty more we can do to improve the DT parsing/validation, but that
> still doesn't apply to this path where the information is *not* coming
> from the DT at all.
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
> 
> Having started writing that patch too, I can tell you it's a big bugger
> touching multiple architectures and fixing up various drivers doing
> stupid things, hence why I'm happy with this point fix being the lesser
> of two evils in terms of not holding up this mostly-orthogonal series.
> 
> Robin.
> 
>>
>> -Frank
>>
> 
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 04:01, Sricharan R wrote:
> Hi Frank,
> 
> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>          dma_addr = offset = 0;
>>> -        size = dev->coherent_dma_mask + 1;
>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>>
> 
> Ok, but with your fix for of_dma_get_range and the above fix,
> dma_addr will be '0' when size = 0xffffffffffffffffULL,
> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
affects several places.  Another potential location (based only
on the function header comment, not from reading the code) is
iommu_dma_init_domain().  The header comment says:

    * @base and @size should be exact multiples of IOMMU page granularity to
    * avoid rounding surprises.

I have not read enough context to really understand of_dma_configure(), but
it seems there is yet another issue in how the error return case from
of_dma_get_range() is handled (with the existing code, as well as if
my patch gets accepted).  An error return value can mean _either_
there is no dma-ranges property _or_ "an other problem occurred".  Should
the "an other problem occurred" case be handled by defaulting size to
a value based on dev->coherent_dma_mask (the current case) or should the
attempt to set up the DMA configuration just fail?

> 
> Regards,
>  Sricharan
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 04:01, Sricharan R wrote:
> Hi Frank,
> 
> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>          dma_addr = offset = 0;
>>> -        size = dev->coherent_dma_mask + 1;
>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>>
> 
> Ok, but with your fix for of_dma_get_range and the above fix,
> dma_addr will be '0' when size = 0xffffffffffffffffULL,
> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
affects several places.  Another potential location (based only
on the function header comment, not from reading the code) is
iommu_dma_init_domain().  The header comment says:

    * @base and @size should be exact multiples of IOMMU page granularity to
    * avoid rounding surprises.

I have not read enough context to really understand of_dma_configure(), but
it seems there is yet another issue in how the error return case from
of_dma_get_range() is handled (with the existing code, as well as if
my patch gets accepted).  An error return value can mean _either_
there is no dma-ranges property _or_ "an other problem occurred".  Should
the "an other problem occurred" case be handled by defaulting size to
a value based on dev->coherent_dma_mask (the current case) or should the
attempt to set up the DMA configuration just fail?

> 
> Regards,
>  Sricharan
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 04:01, Sricharan R wrote:
> Hi Frank,
> 
> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>          dma_addr = offset = 0;
>>> -        size = dev->coherent_dma_mask + 1;
>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>>
> 
> Ok, but with your fix for of_dma_get_range and the above fix,
> dma_addr will be '0' when size = 0xffffffffffffffffULL,
> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
affects several places.  Another potential location (based only
on the function header comment, not from reading the code) is
iommu_dma_init_domain().  The header comment says:

    * @base and @size should be exact multiples of IOMMU page granularity to
    * avoid rounding surprises.

I have not read enough context to really understand of_dma_configure(), but
it seems there is yet another issue in how the error return case from
of_dma_get_range() is handled (with the existing code, as well as if
my patch gets accepted).  An error return value can mean _either_
there is no dma-ranges property _or_ "an other problem occurred".  Should
the "an other problem occurred" case be handled by defaulting size to
a value based on dev->coherent_dma_mask (the current case) or should the
attempt to set up the DMA configuration just fail?

> 
> Regards,
>  Sricharan
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
>>
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 04:01, Sricharan R wrote:
> Hi Frank,
> 
> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>> On 04/04/17 03:18, Sricharan R wrote:
>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>> and passed to arch_setup_dma_ops further. It overflows when
>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>> resulting in size getting passed as 0 wrongly. Fix this by
>>> passsing in max(mask, mask + 1). Note that in this case
>>> when the mask is set to full 64bits, we will be passing the mask
>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>> for this should be to make arch_setup_dma_ops receive the
>>> mask and handle it, to be done in the future.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>>  drivers/of/device.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..c2ae6bb 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>      if (ret < 0) {
>>>          dma_addr = offset = 0;
>>> -        size = dev->coherent_dma_mask + 1;
>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>      } else {
>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>
>>
>> NACK.
>>
>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>
>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>         *dev->dma_mask = min((*dev->dma_mask),
>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>
>> which would be incorrect for size == 0xffffffffffffffffULL when
>> dma_addr != 0.  So the proposed fix really is not papering over
>> the base problem very well.
>>
> 
> Ok, but with your fix for of_dma_get_range and the above fix,
> dma_addr will be '0' when size = 0xffffffffffffffffULL,
> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.

Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
affects several places.  Another potential location (based only
on the function header comment, not from reading the code) is
iommu_dma_init_domain().  The header comment says:

    * @base and @size should be exact multiples of IOMMU page granularity to
    * avoid rounding surprises.

I have not read enough context to really understand of_dma_configure(), but
it seems there is yet another issue in how the error return case from
of_dma_get_range() is handled (with the existing code, as well as if
my patch gets accepted).  An error return value can mean _either_
there is no dma-ranges property _or_ "an other problem occurred".  Should
the "an other problem occurred" case be handled by defaulting size to
a value based on dev->coherent_dma_mask (the current case) or should the
attempt to set up the DMA configuration just fail?

> 
> Regards,
>  Sricharan
> 
>> I agree that the proper solution involves passing a mask instead
>> of a size to arch_setup_dma_ops().
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/7/2017 1:04 AM, Frank Rowand wrote:
> On 04/06/17 04:01, Sricharan R wrote:
>> Hi Frank,
>>
>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>          dma_addr = offset = 0;
>>>> -        size = dev->coherent_dma_mask + 1;
>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>>
>>
>> Ok, but with your fix for of_dma_get_range and the above fix,
>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>
> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
> affects several places.  Another potential location (based only
> on the function header comment, not from reading the code) is
> iommu_dma_init_domain().  The header comment says:
>
>     * @base and @size should be exact multiples of IOMMU page granularity to
>     * avoid rounding surprises.
>

ok, this is the same problem that should get solved when arch_setup_dma_ops
is prepared to take mask instead of size. It would still work as said above
with a smaller mask than specified.

> I have not read enough context to really understand of_dma_configure(), but
> it seems there is yet another issue in how the error return case from
> of_dma_get_range() is handled (with the existing code, as well as if
> my patch gets accepted).  An error return value can mean _either_
> there is no dma-ranges property _or_ "an other problem occurred".  Should
> the "an other problem occurred" case be handled by defaulting size to
> a value based on dev->coherent_dma_mask (the current case) or should the
> attempt to set up the DMA configuration just fail?

The handling of return error value looks like separate item, but looks
like its correct with what is there now. (ie) when of_dma_get_range fails
either because 'dma-ranges property' populated in DT (or) because of some
erroneous DT setting, better to set the mask which the driver has specified
and ignore DT. So the above patch just corrects a mistake in that path.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/7/2017 1:04 AM, Frank Rowand wrote:
> On 04/06/17 04:01, Sricharan R wrote:
>> Hi Frank,
>>
>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>          dma_addr = offset = 0;
>>>> -        size = dev->coherent_dma_mask + 1;
>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>>
>>
>> Ok, but with your fix for of_dma_get_range and the above fix,
>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>
> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
> affects several places.  Another potential location (based only
> on the function header comment, not from reading the code) is
> iommu_dma_init_domain().  The header comment says:
>
>     * @base and @size should be exact multiples of IOMMU page granularity to
>     * avoid rounding surprises.
>

ok, this is the same problem that should get solved when arch_setup_dma_ops
is prepared to take mask instead of size. It would still work as said above
with a smaller mask than specified.

> I have not read enough context to really understand of_dma_configure(), but
> it seems there is yet another issue in how the error return case from
> of_dma_get_range() is handled (with the existing code, as well as if
> my patch gets accepted).  An error return value can mean _either_
> there is no dma-ranges property _or_ "an other problem occurred".  Should
> the "an other problem occurred" case be handled by defaulting size to
> a value based on dev->coherent_dma_mask (the current case) or should the
> attempt to set up the DMA configuration just fail?

The handling of return error value looks like separate item, but looks
like its correct with what is there now. (ie) when of_dma_get_range fails
either because 'dma-ranges property' populated in DT (or) because of some
erroneous DT setting, better to set the mask which the driver has specified
and ignore DT. So the above patch just corrects a mistake in that path.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Sricharan R]

Hi Frank,

On 4/7/2017 1:04 AM, Frank Rowand wrote:
> On 04/06/17 04:01, Sricharan R wrote:
>> Hi Frank,
>>
>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>          dma_addr = offset = 0;
>>>> -        size = dev->coherent_dma_mask + 1;
>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>>
>>
>> Ok, but with your fix for of_dma_get_range and the above fix,
>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>
> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
> affects several places.  Another potential location (based only
> on the function header comment, not from reading the code) is
> iommu_dma_init_domain().  The header comment says:
>
>     * @base and @size should be exact multiples of IOMMU page granularity to
>     * avoid rounding surprises.
>

ok, this is the same problem that should get solved when arch_setup_dma_ops
is prepared to take mask instead of size. It would still work as said above
with a smaller mask than specified.

> I have not read enough context to really understand of_dma_configure(), but
> it seems there is yet another issue in how the error return case from
> of_dma_get_range() is handled (with the existing code, as well as if
> my patch gets accepted).  An error return value can mean _either_
> there is no dma-ranges property _or_ "an other problem occurred".  Should
> the "an other problem occurred" case be handled by defaulting size to
> a value based on dev->coherent_dma_mask (the current case) or should the
> attempt to set up the DMA configuration just fail?

The handling of return error value looks like separate item, but looks
like its correct with what is there now. (ie) when of_dma_get_range fails
either because 'dma-ranges property' populated in DT (or) because of some
erroneous DT setting, better to set the mask which the driver has specified
and ignore DT. So the above patch just corrects a mistake in that path.

Regards,
  Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 20:34, Frank Rowand wrote:
> On 04/06/17 04:01, Sricharan R wrote:
>> Hi Frank,
>>
>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>          dma_addr = offset = 0;
>>>> -        size = dev->coherent_dma_mask + 1;
>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>>
>>
>> Ok, but with your fix for of_dma_get_range and the above fix,
>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
> 
> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
> affects several places.  Another potential location (based only
> on the function header comment, not from reading the code) is
> iommu_dma_init_domain().  The header comment says:
> 
>     * @base and @size should be exact multiples of IOMMU page granularity to
>     * avoid rounding surprises.

That is really only referring to the fact that some of the work done
therein involves truncation to PFNs, so anyone passing in non-exact
values expecting them to round a particular way may get things off by a
page one way or the other. It's not going to have much practical
significance for real devices (in particular since size is used more as
a sanity check than any kind of actual limit there).

> I have not read enough context to really understand of_dma_configure(), but
> it seems there is yet another issue in how the error return case from
> of_dma_get_range() is handled (with the existing code, as well as if
> my patch gets accepted).  An error return value can mean _either_
> there is no dma-ranges property _or_ "an other problem occurred".  Should
> the "an other problem occurred" case be handled by defaulting size to
> a value based on dev->coherent_dma_mask (the current case) or should the
> attempt to set up the DMA configuration just fail?

There is indeed a lot wrong with of_dma_configure() and
arch_setup_dma_ops(), but fixing those is beyond the scope of this
series. This is just working around a latent bug in the one specific
case where a value is *not* derived from DT. Any DT which worked before
still works; any DT which made of_dma_configure() go wrong before still
makes of_dma_configure() go wrong exactly the same.

Whilst it's not ideal, since a DMA mask basically represents the maximum
size of address that that particular device can be given, I can't see it
making any practical difference for a full 64-bit DMA mask to be trimmed
down to 63 bits upon re-probing - no system is likely to have that many
physical address bits anyway, and I don't think any IOMMUs support that
large an IOVA space either, so as long as it's still big enough to cover
"everything", it'll be OK.

Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
thing to do in the first place is yet another matter, as there are
plenty of cases where it results in something which can't reach the
given range at all, but again, this isn't the place. Much as I'm keen to
get the behaviour of of_dma_configure() sorted out properly, it doesn't
seem reasonable that that should suddenly block this
almost-entirely-orthogonal series that various other work has been
waiting on for some time now. The WIP patch I have for
arch_setup_dma_ops() already touches 3 architectures and 4 other
subsystems...

Robin.

> 
>>
>> Regards,
>>  Sricharan
>>
>>> I agree that the proper solution involves passing a mask instead
>>> of a size to arch_setup_dma_ops().
>>>
>>
> 

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 06/04/17 20:34, Frank Rowand wrote:
> On 04/06/17 04:01, Sricharan R wrote:
>> Hi Frank,
>>
>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>> On 04/04/17 03:18, Sricharan R wrote:
>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>> passsing in max(mask, mask + 1). Note that in this case
>>>> when the mask is set to full 64bits, we will be passing the mask
>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>> for this should be to make arch_setup_dma_ops receive the
>>>> mask and handle it, to be done in the future.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> ---
>>>>  drivers/of/device.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>> index c17c19d..c2ae6bb 100644
>>>> --- a/drivers/of/device.c
>>>> +++ b/drivers/of/device.c
>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>      if (ret < 0) {
>>>>          dma_addr = offset = 0;
>>>> -        size = dev->coherent_dma_mask + 1;
>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>      } else {
>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>
>>>
>>> NACK.
>>>
>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>
>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>
>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>> dma_addr != 0.  So the proposed fix really is not papering over
>>> the base problem very well.
>>>
>>
>> Ok, but with your fix for of_dma_get_range and the above fix,
>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
> 
> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
> affects several places.  Another potential location (based only
> on the function header comment, not from reading the code) is
> iommu_dma_init_domain().  The header comment says:
> 
>     * @base and @size should be exact multiples of IOMMU page granularity to
>     * avoid rounding surprises.

That is really only referring to the fact that some of the work done
therein involves truncation to PFNs, so anyone passing in non-exact
values expecting them to round a particular way may get things off by a
page one way or the other. It's not going to have much practical
significance for real devices (in particular since size is used more as
a sanity check than any kind of actual limit there).

> I have not read enough context to really understand of_dma_configure(), but
> it seems there is yet another issue in how the error return case from
> of_dma_get_range() is handled (with the existing code, as well as if
> my patch gets accepted).  An error return value can mean _either_
> there is no dma-ranges property _or_ "an other problem occurred".  Should
> the "an other problem occurred" case be handled by defaulting size to
> a value based on dev->coherent_dma_mask (the current case) or should the
> attempt to set up the DMA configuration just fail?

There is indeed a lot wrong with of_dma_configure() and
arch_setup_dma_ops(), but fixing those is beyond the scope of this
series. This is just working around a latent bug in the one specific
case where a value is *not* derived from DT. Any DT which worked before
still works; any DT which made of_dma_configure() go wrong before still
makes of_dma_configure() go wrong exactly the same.

Whilst it's not ideal, since a DMA mask basically represents the maximum
size of address that that particular device can be given, I can't see it
making any practical difference for a full 64-bit DMA mask to be trimmed
down to 63 bits upon re-probing - no system is likely to have that many
physical address bits anyway, and I don't think any IOMMUs support that
large an IOVA space either, so as long as it's still big enough to cover
"everything", it'll be OK.

Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
thing to do in the first place is yet another matter, as there are
plenty of cases where it results in something which can't reach the
given range at all, but again, this isn't the place. Much as I'm keen to
get the behaviour of of_dma_configure() sorted out properly, it doesn't
seem reasonable that that should suddenly block this
almost-entirely-orthogonal series that various other work has been
waiting on for some time now. The WIP patch I have for
arch_setup_dma_ops() already touches 3 architectures and 4 other
subsystems...

Robin.

> 
>>
>> Regards,
>>  Sricharan
>>
>>> I agree that the proper solution involves passing a mask instead
>>> of a size to arch_setup_dma_ops().
>>>
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 00:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

NACK withdrawn below.

However, I would prefer a change to this line for readability. Using max() results
in the correct result, but obscures the reason behind the algorithm, where the
intent is to avoid an overflow.  How about something like:

	size = (dev->coherent_dma_mask == 0xffffffffffffffffULL)
		? 0xffffffffffffffffULL : dev->coherent_dma_mask + 1;


>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over

  ^^^^^^^^^^^^^  This is the flaw in my objection.  When in the
(ret < 0) path, dma_addr is set to zero.  So my worry about dma_addr != 0
is baseless.

I withdraw my NACK because my analysis was flawed.

-Frank

> the base problem very well.
> 
> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().
> 
> -Frank
> 

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/06/17 00:01, Frank Rowand wrote:
> On 04/04/17 03:18, Sricharan R wrote:
>> Size of the dma-range is calculated as coherent_dma_mask + 1
>> and passed to arch_setup_dma_ops further. It overflows when
>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>> resulting in size getting passed as 0 wrongly. Fix this by
>> passsing in max(mask, mask + 1). Note that in this case
>> when the mask is set to full 64bits, we will be passing the mask
>> itself to arch_setup_dma_ops instead of the size. The real fix
>> for this should be to make arch_setup_dma_ops receive the
>> mask and handle it, to be done in the future.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/of/device.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..c2ae6bb 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>  	if (ret < 0) {
>>  		dma_addr = offset = 0;
>> -		size = dev->coherent_dma_mask + 1;
>> +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

NACK withdrawn below.

However, I would prefer a change to this line for readability. Using max() results
in the correct result, but obscures the reason behind the algorithm, where the
intent is to avoid an overflow.  How about something like:

	size = (dev->coherent_dma_mask == 0xffffffffffffffffULL)
		? 0xffffffffffffffffULL : dev->coherent_dma_mask + 1;


>>  	} else {
>>  		offset = PFN_DOWN(paddr - dma_addr);
>>  		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>
> 
> NACK.
> 
> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
> 
>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>         *dev->dma_mask = min((*dev->dma_mask),
>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
> 
> which would be incorrect for size == 0xffffffffffffffffULL when
> dma_addr != 0.  So the proposed fix really is not papering over

  ^^^^^^^^^^^^^  This is the flaw in my objection.  When in the
(ret < 0) path, dma_addr is set to zero.  So my worry about dma_addr != 0
is baseless.

I withdraw my NACK because my analysis was flawed.

-Frank

> the base problem very well.
> 
> I agree that the proper solution involves passing a mask instead
> of a size to arch_setup_dma_ops().
> 
> -Frank
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/07/17 07:46, Robin Murphy wrote:
> On 06/04/17 20:34, Frank Rowand wrote:
>> On 04/06/17 04:01, Sricharan R wrote:
>>> Hi Frank,
>>>
>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>> mask and handle it, to be done in the future.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> ---
>>>>>  drivers/of/device.c | 2 +-
>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>> index c17c19d..c2ae6bb 100644
>>>>> --- a/drivers/of/device.c
>>>>> +++ b/drivers/of/device.c
>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>      if (ret < 0) {
>>>>>          dma_addr = offset = 0;
>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>      } else {
>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>
>>>>
>>>> NACK.
>>>>
>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>
>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>
>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>> the base problem very well.
>>>>
>>>
>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>
>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>> affects several places.  Another potential location (based only
>> on the function header comment, not from reading the code) is
>> iommu_dma_init_domain().  The header comment says:
>>
>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>     * avoid rounding surprises.
> 
> That is really only referring to the fact that some of the work done
> therein involves truncation to PFNs, so anyone passing in non-exact
> values expecting them to round a particular way may get things off by a
> page one way or the other. It's not going to have much practical
> significance for real devices (in particular since size is used more as
> a sanity check than any kind of actual limit there).
> 
>> I have not read enough context to really understand of_dma_configure(), but
>> it seems there is yet another issue in how the error return case from
>> of_dma_get_range() is handled (with the existing code, as well as if
>> my patch gets accepted).  An error return value can mean _either_
>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>> the "an other problem occurred" case be handled by defaulting size to
>> a value based on dev->coherent_dma_mask (the current case) or should the
>> attempt to set up the DMA configuration just fail?
> 
> There is indeed a lot wrong with of_dma_configure() and
> arch_setup_dma_ops(), but fixing those is beyond the scope of this
> series. This is just working around a latent bug in the one specific
> case where a value is *not* derived from DT. Any DT which worked before
> still works; any DT which made of_dma_configure() go wrong before still
> makes of_dma_configure() go wrong exactly the same.
> 
> Whilst it's not ideal, since a DMA mask basically represents the maximum
> size of address that that particular device can be given, I can't see it
> making any practical difference for a full 64-bit DMA mask to be trimmed
> down to 63 bits upon re-probing - no system is likely to have that many
> physical address bits anyway, and I don't think any IOMMUs support that
> large an IOVA space either, so as long as it's still big enough to cover
> "everything", it'll be OK.
> 
> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
> thing to do in the first place is yet another matter, as there are
> plenty of cases where it results in something which can't reach the
> given range at all, but again, this isn't the place. Much as I'm keen to
> get the behaviour of of_dma_configure() sorted out properly, it doesn't
> seem reasonable that that should suddenly block this
> almost-entirely-orthogonal series that various other work has been
> waiting on for some time now. The WIP patch I have for
> arch_setup_dma_ops() already touches 3 architectures and 4 other
> subsystems...

In a reply to my original NACK email, I just now retracted the NACK,
but with a requested change for readability.

I buy your analysis and argument here.  The patch will improve things
a little, but it will be good to revisit of_dma_configure() in the
future to further clean things up.

-Frank

> 
> Robin.
> 
>>
>>>
>>> Regards,
>>>  Sricharan
>>>
>>>> I agree that the proper solution involves passing a mask instead
>>>> of a size to arch_setup_dma_ops().
>>>>
>>>
>>
> 
> 

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Frank Rowand]

On 04/07/17 07:46, Robin Murphy wrote:
> On 06/04/17 20:34, Frank Rowand wrote:
>> On 04/06/17 04:01, Sricharan R wrote:
>>> Hi Frank,
>>>
>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>> mask and handle it, to be done in the future.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> ---
>>>>>  drivers/of/device.c | 2 +-
>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>> index c17c19d..c2ae6bb 100644
>>>>> --- a/drivers/of/device.c
>>>>> +++ b/drivers/of/device.c
>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>      if (ret < 0) {
>>>>>          dma_addr = offset = 0;
>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>      } else {
>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>
>>>>
>>>> NACK.
>>>>
>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>
>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>
>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>> the base problem very well.
>>>>
>>>
>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>
>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>> affects several places.  Another potential location (based only
>> on the function header comment, not from reading the code) is
>> iommu_dma_init_domain().  The header comment says:
>>
>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>     * avoid rounding surprises.
> 
> That is really only referring to the fact that some of the work done
> therein involves truncation to PFNs, so anyone passing in non-exact
> values expecting them to round a particular way may get things off by a
> page one way or the other. It's not going to have much practical
> significance for real devices (in particular since size is used more as
> a sanity check than any kind of actual limit there).
> 
>> I have not read enough context to really understand of_dma_configure(), but
>> it seems there is yet another issue in how the error return case from
>> of_dma_get_range() is handled (with the existing code, as well as if
>> my patch gets accepted).  An error return value can mean _either_
>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>> the "an other problem occurred" case be handled by defaulting size to
>> a value based on dev->coherent_dma_mask (the current case) or should the
>> attempt to set up the DMA configuration just fail?
> 
> There is indeed a lot wrong with of_dma_configure() and
> arch_setup_dma_ops(), but fixing those is beyond the scope of this
> series. This is just working around a latent bug in the one specific
> case where a value is *not* derived from DT. Any DT which worked before
> still works; any DT which made of_dma_configure() go wrong before still
> makes of_dma_configure() go wrong exactly the same.
> 
> Whilst it's not ideal, since a DMA mask basically represents the maximum
> size of address that that particular device can be given, I can't see it
> making any practical difference for a full 64-bit DMA mask to be trimmed
> down to 63 bits upon re-probing - no system is likely to have that many
> physical address bits anyway, and I don't think any IOMMUs support that
> large an IOVA space either, so as long as it's still big enough to cover
> "everything", it'll be OK.
> 
> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
> thing to do in the first place is yet another matter, as there are
> plenty of cases where it results in something which can't reach the
> given range at all, but again, this isn't the place. Much as I'm keen to
> get the behaviour of of_dma_configure() sorted out properly, it doesn't
> seem reasonable that that should suddenly block this
> almost-entirely-orthogonal series that various other work has been
> waiting on for some time now. The WIP patch I have for
> arch_setup_dma_ops() already touches 3 architectures and 4 other
> subsystems...

In a reply to my original NACK email, I just now retracted the NACK,
but with a requested change for readability.

I buy your analysis and argument here.  The patch will improve things
a little, but it will be good to revisit of_dma_configure() in the
future to further clean things up.

-Frank

> 
> Robin.
> 
>>
>>>
>>> Regards,
>>>  Sricharan
>>>
>>>> I agree that the proper solution involves passing a mask instead
>>>> of a size to arch_setup_dma_ops().
>>>>
>>>
>>
> 
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 08/04/17 00:13, Frank Rowand wrote:
> On 04/07/17 07:46, Robin Murphy wrote:
>> On 06/04/17 20:34, Frank Rowand wrote:
>>> On 04/06/17 04:01, Sricharan R wrote:
>>>> Hi Frank,
>>>>
>>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>>> mask and handle it, to be done in the future.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>>>> ---
>>>>>>  drivers/of/device.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>>> index c17c19d..c2ae6bb 100644
>>>>>> --- a/drivers/of/device.c
>>>>>> +++ b/drivers/of/device.c
>>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>>      if (ret < 0) {
>>>>>>          dma_addr = offset = 0;
>>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>>      } else {
>>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>>
>>>>>
>>>>> NACK.
>>>>>
>>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>>
>>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>
>>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>>> the base problem very well.
>>>>>
>>>>
>>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>>
>>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>>> affects several places.  Another potential location (based only
>>> on the function header comment, not from reading the code) is
>>> iommu_dma_init_domain().  The header comment says:
>>>
>>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>>     * avoid rounding surprises.
>>
>> That is really only referring to the fact that some of the work done
>> therein involves truncation to PFNs, so anyone passing in non-exact
>> values expecting them to round a particular way may get things off by a
>> page one way or the other. It's not going to have much practical
>> significance for real devices (in particular since size is used more as
>> a sanity check than any kind of actual limit there).
>>
>>> I have not read enough context to really understand of_dma_configure(), but
>>> it seems there is yet another issue in how the error return case from
>>> of_dma_get_range() is handled (with the existing code, as well as if
>>> my patch gets accepted).  An error return value can mean _either_
>>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>>> the "an other problem occurred" case be handled by defaulting size to
>>> a value based on dev->coherent_dma_mask (the current case) or should the
>>> attempt to set up the DMA configuration just fail?
>>
>> There is indeed a lot wrong with of_dma_configure() and
>> arch_setup_dma_ops(), but fixing those is beyond the scope of this
>> series. This is just working around a latent bug in the one specific
>> case where a value is *not* derived from DT. Any DT which worked before
>> still works; any DT which made of_dma_configure() go wrong before still
>> makes of_dma_configure() go wrong exactly the same.
>>
>> Whilst it's not ideal, since a DMA mask basically represents the maximum
>> size of address that that particular device can be given, I can't see it
>> making any practical difference for a full 64-bit DMA mask to be trimmed
>> down to 63 bits upon re-probing - no system is likely to have that many
>> physical address bits anyway, and I don't think any IOMMUs support that
>> large an IOVA space either, so as long as it's still big enough to cover
>> "everything", it'll be OK.
>>
>> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
>> thing to do in the first place is yet another matter, as there are
>> plenty of cases where it results in something which can't reach the
>> given range at all, but again, this isn't the place. Much as I'm keen to
>> get the behaviour of of_dma_configure() sorted out properly, it doesn't
>> seem reasonable that that should suddenly block this
>> almost-entirely-orthogonal series that various other work has been
>> waiting on for some time now. The WIP patch I have for
>> arch_setup_dma_ops() already touches 3 architectures and 4 other
>> subsystems...
> 
> In a reply to my original NACK email, I just now retracted the NACK,
> but with a requested change for readability.
> 
> I buy your analysis and argument here.  The patch will improve things
> a little, but it will be good to revisit of_dma_configure() in the
> future to further clean things up.

Thanks! As mentioned, I have some patches brewing in this area which
should hopefully help somewhat; I'll try to get them posted this week.

Robin.

> 
> -Frank
> 
>>
>> Robin.
>>
>>>
>>>>
>>>> Regards,
>>>>  Sricharan
>>>>
>>>>> I agree that the proper solution involves passing a mask instead
>>>>> of a size to arch_setup_dma_ops().
>>>>>
>>>>
>>>
>>
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 08/04/17 00:13, Frank Rowand wrote:
> On 04/07/17 07:46, Robin Murphy wrote:
>> On 06/04/17 20:34, Frank Rowand wrote:
>>> On 04/06/17 04:01, Sricharan R wrote:
>>>> Hi Frank,
>>>>
>>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>>> mask and handle it, to be done in the future.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>> ---
>>>>>>  drivers/of/device.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>>> index c17c19d..c2ae6bb 100644
>>>>>> --- a/drivers/of/device.c
>>>>>> +++ b/drivers/of/device.c
>>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>>      if (ret < 0) {
>>>>>>          dma_addr = offset = 0;
>>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>>      } else {
>>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>>
>>>>>
>>>>> NACK.
>>>>>
>>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>>
>>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>
>>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>>> the base problem very well.
>>>>>
>>>>
>>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>>
>>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>>> affects several places.  Another potential location (based only
>>> on the function header comment, not from reading the code) is
>>> iommu_dma_init_domain().  The header comment says:
>>>
>>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>>     * avoid rounding surprises.
>>
>> That is really only referring to the fact that some of the work done
>> therein involves truncation to PFNs, so anyone passing in non-exact
>> values expecting them to round a particular way may get things off by a
>> page one way or the other. It's not going to have much practical
>> significance for real devices (in particular since size is used more as
>> a sanity check than any kind of actual limit there).
>>
>>> I have not read enough context to really understand of_dma_configure(), but
>>> it seems there is yet another issue in how the error return case from
>>> of_dma_get_range() is handled (with the existing code, as well as if
>>> my patch gets accepted).  An error return value can mean _either_
>>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>>> the "an other problem occurred" case be handled by defaulting size to
>>> a value based on dev->coherent_dma_mask (the current case) or should the
>>> attempt to set up the DMA configuration just fail?
>>
>> There is indeed a lot wrong with of_dma_configure() and
>> arch_setup_dma_ops(), but fixing those is beyond the scope of this
>> series. This is just working around a latent bug in the one specific
>> case where a value is *not* derived from DT. Any DT which worked before
>> still works; any DT which made of_dma_configure() go wrong before still
>> makes of_dma_configure() go wrong exactly the same.
>>
>> Whilst it's not ideal, since a DMA mask basically represents the maximum
>> size of address that that particular device can be given, I can't see it
>> making any practical difference for a full 64-bit DMA mask to be trimmed
>> down to 63 bits upon re-probing - no system is likely to have that many
>> physical address bits anyway, and I don't think any IOMMUs support that
>> large an IOVA space either, so as long as it's still big enough to cover
>> "everything", it'll be OK.
>>
>> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
>> thing to do in the first place is yet another matter, as there are
>> plenty of cases where it results in something which can't reach the
>> given range at all, but again, this isn't the place. Much as I'm keen to
>> get the behaviour of of_dma_configure() sorted out properly, it doesn't
>> seem reasonable that that should suddenly block this
>> almost-entirely-orthogonal series that various other work has been
>> waiting on for some time now. The WIP patch I have for
>> arch_setup_dma_ops() already touches 3 architectures and 4 other
>> subsystems...
> 
> In a reply to my original NACK email, I just now retracted the NACK,
> but with a requested change for readability.
> 
> I buy your analysis and argument here.  The patch will improve things
> a little, but it will be good to revisit of_dma_configure() in the
> future to further clean things up.

Thanks! As mentioned, I have some patches brewing in this area which
should hopefully help somewhat; I'll try to get them posted this week.

Robin.

> 
> -Frank
> 
>>
>> Robin.
>>
>>>
>>>>
>>>> Regards,
>>>>  Sricharan
>>>>
>>>>> I agree that the proper solution involves passing a mask instead
>>>>> of a size to arch_setup_dma_ops().
>>>>>
>>>>
>>>
>>
>>
> 

Re: [PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 08/04/17 00:13, Frank Rowand wrote:
> On 04/07/17 07:46, Robin Murphy wrote:
>> On 06/04/17 20:34, Frank Rowand wrote:
>>> On 04/06/17 04:01, Sricharan R wrote:
>>>> Hi Frank,
>>>>
>>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>>> mask and handle it, to be done in the future.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>> ---
>>>>>>  drivers/of/device.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>>> index c17c19d..c2ae6bb 100644
>>>>>> --- a/drivers/of/device.c
>>>>>> +++ b/drivers/of/device.c
>>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>>      if (ret < 0) {
>>>>>>          dma_addr = offset = 0;
>>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>>      } else {
>>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>>
>>>>>
>>>>> NACK.
>>>>>
>>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>>
>>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>
>>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>>> the base problem very well.
>>>>>
>>>>
>>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>>
>>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>>> affects several places.  Another potential location (based only
>>> on the function header comment, not from reading the code) is
>>> iommu_dma_init_domain().  The header comment says:
>>>
>>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>>     * avoid rounding surprises.
>>
>> That is really only referring to the fact that some of the work done
>> therein involves truncation to PFNs, so anyone passing in non-exact
>> values expecting them to round a particular way may get things off by a
>> page one way or the other. It's not going to have much practical
>> significance for real devices (in particular since size is used more as
>> a sanity check than any kind of actual limit there).
>>
>>> I have not read enough context to really understand of_dma_configure(), but
>>> it seems there is yet another issue in how the error return case from
>>> of_dma_get_range() is handled (with the existing code, as well as if
>>> my patch gets accepted).  An error return value can mean _either_
>>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>>> the "an other problem occurred" case be handled by defaulting size to
>>> a value based on dev->coherent_dma_mask (the current case) or should the
>>> attempt to set up the DMA configuration just fail?
>>
>> There is indeed a lot wrong with of_dma_configure() and
>> arch_setup_dma_ops(), but fixing those is beyond the scope of this
>> series. This is just working around a latent bug in the one specific
>> case where a value is *not* derived from DT. Any DT which worked before
>> still works; any DT which made of_dma_configure() go wrong before still
>> makes of_dma_configure() go wrong exactly the same.
>>
>> Whilst it's not ideal, since a DMA mask basically represents the maximum
>> size of address that that particular device can be given, I can't see it
>> making any practical difference for a full 64-bit DMA mask to be trimmed
>> down to 63 bits upon re-probing - no system is likely to have that many
>> physical address bits anyway, and I don't think any IOMMUs support that
>> large an IOVA space either, so as long as it's still big enough to cover
>> "everything", it'll be OK.
>>
>> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
>> thing to do in the first place is yet another matter, as there are
>> plenty of cases where it results in something which can't reach the
>> given range at all, but again, this isn't the place. Much as I'm keen to
>> get the behaviour of of_dma_configure() sorted out properly, it doesn't
>> seem reasonable that that should suddenly block this
>> almost-entirely-orthogonal series that various other work has been
>> waiting on for some time now. The WIP patch I have for
>> arch_setup_dma_ops() already touches 3 architectures and 4 other
>> subsystems...
> 
> In a reply to my original NACK email, I just now retracted the NACK,
> but with a requested change for readability.
> 
> I buy your analysis and argument here.  The patch will improve things
> a little, but it will be good to revisit of_dma_configure() in the
> future to further clean things up.

Thanks! As mentioned, I have some patches brewing in this area which
should hopefully help somewhat; I'll try to get them posted this week.

Robin.

> 
> -Frank
> 
>>
>> Robin.
>>
>>>
>>>>
>>>> Regards,
>>>>  Sricharan
>>>>
>>>>> I agree that the proper solution involves passing a mask instead
>>>>> of a size to arch_setup_dma_ops().
>>>>>
>>>>
>>>
>>
>>
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V10 06/12] of: device: Fix overflow of coherent_dma_mask

[Robin Murphy]

On 08/04/17 00:13, Frank Rowand wrote:
> On 04/07/17 07:46, Robin Murphy wrote:
>> On 06/04/17 20:34, Frank Rowand wrote:
>>> On 04/06/17 04:01, Sricharan R wrote:
>>>> Hi Frank,
>>>>
>>>> On 4/6/2017 12:31 PM, Frank Rowand wrote:
>>>>> On 04/04/17 03:18, Sricharan R wrote:
>>>>>> Size of the dma-range is calculated as coherent_dma_mask + 1
>>>>>> and passed to arch_setup_dma_ops further. It overflows when
>>>>>> the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF,
>>>>>> resulting in size getting passed as 0 wrongly. Fix this by
>>>>>> passsing in max(mask, mask + 1). Note that in this case
>>>>>> when the mask is set to full 64bits, we will be passing the mask
>>>>>> itself to arch_setup_dma_ops instead of the size. The real fix
>>>>>> for this should be to make arch_setup_dma_ops receive the
>>>>>> mask and handle it, to be done in the future.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>> ---
>>>>>>  drivers/of/device.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>>>>> index c17c19d..c2ae6bb 100644
>>>>>> --- a/drivers/of/device.c
>>>>>> +++ b/drivers/of/device.c
>>>>>> @@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>>>>      ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
>>>>>>      if (ret < 0) {
>>>>>>          dma_addr = offset = 0;
>>>>>> -        size = dev->coherent_dma_mask + 1;
>>>>>> +        size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
>>>>>>      } else {
>>>>>>          offset = PFN_DOWN(paddr - dma_addr);
>>>>>>          dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
>>>>>>
>>>>>
>>>>> NACK.
>>>>>
>>>>> Passing an invalid size to arch_setup_dma_ops() is only part of the problem.
>>>>> size is also used in of_dma_configure() before calling arch_setup_dma_ops():
>>>>>
>>>>>         dev->coherent_dma_mask = min(dev->coherent_dma_mask,
>>>>>                                      DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>         *dev->dma_mask = min((*dev->dma_mask),
>>>>>                              DMA_BIT_MASK(ilog2(dma_addr + size)));
>>>>>
>>>>> which would be incorrect for size == 0xffffffffffffffffULL when
>>>>> dma_addr != 0.  So the proposed fix really is not papering over
>>>>> the base problem very well.
>>>>>
>>>>
>>>> Ok, but with your fix for of_dma_get_range and the above fix,
>>>> dma_addr will be '0' when size = 0xffffffffffffffffULL,
>>>> but DMA_BIT_MASK(ilog2(dma_addr + size)) would be wrong though,
>>>> making coherent_dma_mask to be smaller 0x7fffffffffffffffULL.
>>>
>>> Yes, that was my point.  Setting size to 0x7fffffffffffffffULL
>>> affects several places.  Another potential location (based only
>>> on the function header comment, not from reading the code) is
>>> iommu_dma_init_domain().  The header comment says:
>>>
>>>     * @base and @size should be exact multiples of IOMMU page granularity to
>>>     * avoid rounding surprises.
>>
>> That is really only referring to the fact that some of the work done
>> therein involves truncation to PFNs, so anyone passing in non-exact
>> values expecting them to round a particular way may get things off by a
>> page one way or the other. It's not going to have much practical
>> significance for real devices (in particular since size is used more as
>> a sanity check than any kind of actual limit there).
>>
>>> I have not read enough context to really understand of_dma_configure(), but
>>> it seems there is yet another issue in how the error return case from
>>> of_dma_get_range() is handled (with the existing code, as well as if
>>> my patch gets accepted).  An error return value can mean _either_
>>> there is no dma-ranges property _or_ "an other problem occurred".  Should
>>> the "an other problem occurred" case be handled by defaulting size to
>>> a value based on dev->coherent_dma_mask (the current case) or should the
>>> attempt to set up the DMA configuration just fail?
>>
>> There is indeed a lot wrong with of_dma_configure() and
>> arch_setup_dma_ops(), but fixing those is beyond the scope of this
>> series. This is just working around a latent bug in the one specific
>> case where a value is *not* derived from DT. Any DT which worked before
>> still works; any DT which made of_dma_configure() go wrong before still
>> makes of_dma_configure() go wrong exactly the same.
>>
>> Whilst it's not ideal, since a DMA mask basically represents the maximum
>> size of address that that particular device can be given, I can't see it
>> making any practical difference for a full 64-bit DMA mask to be trimmed
>> down to 63 bits upon re-probing - no system is likely to have that many
>> physical address bits anyway, and I don't think any IOMMUs support that
>> large an IOVA space either, so as long as it's still big enough to cover
>> "everything", it'll be OK.
>>
>> Of course, whether DMA_BIT_MASK(ilog2(dma_addr + size)) is the right
>> thing to do in the first place is yet another matter, as there are
>> plenty of cases where it results in something which can't reach the
>> given range at all, but again, this isn't the place. Much as I'm keen to
>> get the behaviour of of_dma_configure() sorted out properly, it doesn't
>> seem reasonable that that should suddenly block this
>> almost-entirely-orthogonal series that various other work has been
>> waiting on for some time now. The WIP patch I have for
>> arch_setup_dma_ops() already touches 3 architectures and 4 other
>> subsystems...
> 
> In a reply to my original NACK email, I just now retracted the NACK,
> but with a requested change for readability.
> 
> I buy your analysis and argument here.  The patch will improve things
> a little, but it will be good to revisit of_dma_configure() in the
> future to further clean things up.

Thanks! As mentioned, I have some patches brewing in this area which
should hopefully help somewhat; I'll try to get them posted this week.

Robin.

> 
> -Frank
> 
>>
>> Robin.
>>
>>>
>>>>
>>>> Regards,
>>>>  Sricharan
>>>>
>>>>> I agree that the proper solution involves passing a mask instead
>>>>> of a size to arch_setup_dma_ops().
>>>>>
>>>>
>>>
>>
>>
>