[PATCH V8 00/11] IOMMU probe deferral support

[PATCH V8 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against linux-next. Please note that there is an
conflict in patch#9 while applying against iommu-next because of the
below patch not being there on iommu-next

	"arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
	https://www.spinics.net/lists/arm-kernel/msg556209.html
  
Similarly there was conflict in patch#1 and patch#9 while applying
against master because of the above patch not there and additionally
the below one not being there.

	"iommu: Drop the of_iommu_{set/get}_ops() interface"
	https://lkml.org/lkml/2017/1/3/489

* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

   Build tested on linux-next, iommu-next, master.

Previous post of this series [6]. 

 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] https://www.spinics.net/lists/arm-kernel/msg557110.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          |  58 +++-------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 269 insertions(+), 323 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against linux-next. Please note that there is an
conflict in patch#9 while applying against iommu-next because of the
below patch not being there on iommu-next

	"arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
	https://www.spinics.net/lists/arm-kernel/msg556209.html
  
Similarly there was conflict in patch#1 and patch#9 while applying
against master because of the above patch not there and additionally
the below one not being there.

	"iommu: Drop the of_iommu_{set/get}_ops() interface"
	https://lkml.org/lkml/2017/1/3/489

* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

   Build tested on linux-next, iommu-next, master.

Previous post of this series [6]. 

 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] https://www.spinics.net/lists/arm-kernel/msg557110.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          |  58 +++-------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 269 insertions(+), 323 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 00/11] IOMMU probe deferral support

[Sricharan R]

This series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Rebased the series against linux-next. Please note that there is an
conflict in patch#9 while applying against iommu-next because of the
below patch not being there on iommu-next

	"arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
	https://www.spinics.net/lists/arm-kernel/msg556209.html
  
Similarly there was conflict in patch#1 and patch#9 while applying
against master because of the above patch not there and additionally
the below one not being there.

	"iommu: Drop the of_iommu_{set/get}_ops() interface"
	https://lkml.org/lkml/2017/1/3/489

* Tested with platform and pci devices for probe deferral
  and reprobe on arm64 based platform.

   Build tested on linux-next, iommu-next, master.

Previous post of this series [6]. 

 [V8]
     * Picked up all the acks and tested tags from Marek and
       Hanjun for DT and ACPI patches respectively, since
       no functional changes was done.

     * Addressed Minor comments Sinan and Bjorn.

     * Added Robin's fix for fixing the deferencing NULL for
       of_iommu_table after init in patch #2.

     * Rebased it on top of linux-next

 [V7]
     * Updated the subject and commit log for patch #6 as per
       comments from Lorenzo. No functional changes.

 [V6]
     * Fixed a bug in dma_configure function pointed out by
       Robin.
     * Reordered the patches as per comments from Robin and
       Lorenzo.
     * Added Tags.

 [V5]
     * Reworked the pci configuration code hanging outside and
       pushed it to dma_configure as in PATCH#5,6,7.
       Also added a couple of patches that Lorenzo provided for
       correcting the Probe deferring mechanism in case of
       ACPI devices from here [5].

 [V4]
     * Took the reworked patches [2] from Robin's branch and
       rebased on top of Lorenzo's ACPI IORT ARM support series [3].

     * Added the patches for moving the dma ops configuration of
       acpi based devices to probe time as well.
 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration
       separately based on review comments that both masks and ops are
       required only during the device probe time.

     * Reworked the series based on Generic DT bindings series.

     * Added call to iommu's remove_device in the cleanup path for arm and
       arm64.

     * Removed the notifier trick in arm64 to handle early device
       registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as
       well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in
       last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from
       generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] https://www.spinics.net/lists/arm-kernel/msg557110.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
  ACPI/IORT: Add function to check SMMUs drivers presence
  ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
  iommu/of: Refactor of_iommu_configure() for error handling
  iommu/of: Prepare for deferred IOMMU configuration
  iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
  of/acpi: Configure dma operations at probe time for platform/amba/pci
    bus devices
  drivers: acpi: Handle IOMMU lookup failure with deferred probing or
    error
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          |  58 +++-------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 269 insertions(+), 323 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 [*] Resolved a conflict while rebasing on top linux-next as the patch
     is not there in mainline master.

        "iommu: Drop the of_iommu_{set/get}_ops() interface"
        https://lkml.org/lkml/2017/1/3/489

 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index d7f480a..ee49081 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_get_instance(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return ERR_PTR(err);
 
-	ops = iommu_get_instance(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_get_instance(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 [*] Resolved a conflict while rebasing on top linux-next as the patch
     is not there in mainline master.

        "iommu: Drop the of_iommu_{set/get}_ops() interface"
        https://lkml.org/lkml/2017/1/3/489

 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index d7f480a..ee49081 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_get_instance(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return ERR_PTR(err);
 
-	ops = iommu_get_instance(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_get_instance(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

In preparation for some upcoming cleverness, rework the control flow in
of_iommu_configure() to minimise duplication and improve the propogation
of errors. It's also as good a time as any to switch over from the
now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
IOMMU instance interface directly.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 [*] Resolved a conflict while rebasing on top linux-next as the patch
     is not there in mainline master.

        "iommu: Drop the of_iommu_{set/get}_ops() interface"
        https://lkml.org/lkml/2017/1/3/489

 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index d7f480a..ee49081 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_get_instance(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;
@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as
@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return ERR_PTR(err);
 
-	ops = iommu_get_instance(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_get_instance(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 02/11] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index ee49081..1f92d98 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_get_instance(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 02/11] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index ee49081..1f92d98 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_get_instance(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 02/11] iommu/of: Prepare for deferred IOMMU configuration

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

IOMMU configuration represents unchanging properties of the hardware,
and as such should only need happen once in a device's lifetime, but
the necessary interaction with the IOMMU device and driver complicates
exactly when that point should be.

Since the only reasonable tool available for handling the inter-device
dependency is probe deferral, we need to prepare of_iommu_configure()
to run later than it is currently called (i.e. at driver probe rather
than device creation), to handle being retried, and to tell whether a
not-yet present IOMMU should be waited for or skipped (by virtue of
having declared a built-in driver or not).

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index ee49081..1f92d98 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {
@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_get_instance(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)
@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 03/11] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 03/11] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 03/11] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index b1e6beb..09dedd0 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 04/11] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index b8064bc..57418f7 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index e9afbcc..3cb2288 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -57,6 +57,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -106,6 +107,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 04/11] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index b8064bc..57418f7 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index e9afbcc..3cb2288 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -57,6 +57,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -106,6 +107,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 04/11] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index 09dedd0..c17c19d 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index b8064bc..57418f7 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index e9afbcc..3cb2288 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -57,6 +57,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -106,6 +107,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 05/11] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
Cc: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
Cc: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e0d2e6e..bf0ed09 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 05/11] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e0d2e6e..bf0ed09 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 05/11] ACPI/IORT: Add function to check SMMUs drivers presence

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IOMMU probe deferral implementation requires a mechanism to detect
if drivers for SMMU components are built-in in the kernel to detect
whether IOMMU configuration for a given device should be deferred (ie
SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU
components managed by IORT are built-in in the kernel.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index e0d2e6e..bf0ed09 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 06/11] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Bjorn Helgaas <bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 57418f7..cf35030 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 139466c..c11b3de 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1875,33 +1875,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1915,7 +1888,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index c24721a..5972087 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -715,6 +715,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 06/11] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 57418f7..cf35030 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 139466c..c11b3de 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1875,33 +1875,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1915,7 +1888,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index c24721a..5972087 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -715,6 +715,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 06/11] of/acpi: Configure dma operations at probe time for platform/amba/pci bus devices

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/drivers/acpi/glue.c b/drivers/acpi/glue.c
index fb19e1c..c05f241 100644
--- a/drivers/acpi/glue.c
+++ b/drivers/acpi/glue.c
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {
@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index a1fbf55..4882f06 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index efd71cf..449b948 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 57418f7..cf35030 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 139466c..c11b3de 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1875,33 +1875,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
@@ -1915,7 +1888,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index c24721a..5972087 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -715,6 +715,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
     in of_iommu.c

 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 1f92d98..2d04663 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..ba51ca6 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index 3cb2288..9499861 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
     in of_iommu.c

 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 449b948..82bd45c 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 1f92d98..2d04663 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index c17c19d..ba51ca6 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index 3cb2288..9499861 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index bf0ed09..d01bae8 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_get_instance(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..823b005 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 4242c31..9aa762fe 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -573,7 +573,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 5b36974..8b958b6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -764,8 +764,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index bf0ed09..d01bae8 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_get_instance(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..823b005 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 4242c31..9aa762fe 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -573,7 +573,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 5b36974..8b958b6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -764,8 +764,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

This is an equivalent to the DT's handling of the iommu master's probe
with deferred probing when the corrsponding iommu is not probed yet.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the firmware describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index bf0ed09..d01bae8 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_get_instance(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}
@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 1926918..823b005 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 82bd45c..755a2b5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 4242c31..9aa762fe 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -573,7 +573,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 5b36974..8b958b6 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -764,8 +764,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 09/11] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's IOMMU is probed, the notifier trick required to handle the
early setup of dma_ops before the IOMMU group gets created is not
required. So removing the notifiers here.

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Tested-by: Hanjun Guo <hanjun.guo-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Acked-by: Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---

[*] Rebased the series against linux-next. Please note that there is an
    conflict while applying this patch against iommu-next because of the
    patch [1] "arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
    not there

    [1] https://www.spinics.net/lists/arm-kernel/msg556209.html

 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 6e295e8..3653c4c 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -814,34 +814,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -852,109 +844,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->archdata.dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
 	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
-		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
+		dev_name(dev));
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 09/11] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's IOMMU is probed, the notifier trick required to handle the
early setup of dma_ops before the IOMMU group gets created is not
required. So removing the notifiers here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---

[*] Rebased the series against linux-next. Please note that there is an
    conflict while applying this patch against iommu-next because of the
    patch [1] "arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
    not there

    [1] https://www.spinics.net/lists/arm-kernel/msg556209.html

 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 6e295e8..3653c4c 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -814,34 +814,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -852,109 +844,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->archdata.dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
 	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
-		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
+		dev_name(dev));
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 09/11] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after
the device's IOMMU is probed, the notifier trick required to handle the
early setup of dma_ops before the IOMMU group gets created is not
required. So removing the notifiers here.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[rm: clean up even more]
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---

[*] Rebased the series against linux-next. Please note that there is an
    conflict while applying this patch against iommu-next because of the
    patch [1] "arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA"
    not there

    [1] https://www.spinics.net/lists/arm-kernel/msg556209.html

 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 6e295e8..3653c4c 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -814,34 +814,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 
@@ -852,109 +844,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->archdata.dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
 	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
-		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
+		dev_name(dev));
 }
 
 void arch_teardown_dma_ops(struct device *dev)
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 10/11] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c | 46 ++---------------------------------
 drivers/iommu/arm-smmu.c    | 58 +++++++--------------------------------------
 2 files changed, 10 insertions(+), 94 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 7bb157c..eba4ac8 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2743,51 +2743,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 091d4a6..faea579 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2190,56 +2190,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 10/11] iommu/arm-smmu: Clean up early-probing workarounds

[Sricharan R]

From: Robin Murphy <robin.murphy@arm.com>

Now that the appropriate ordering is enforced via probe-deferral of
masters in core code, rip it all out and bask in the simplicity.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[Sricharan: Rebased on top of ACPI IORT SMMU series]
Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/arm-smmu-v3.c | 46 ++---------------------------------
 drivers/iommu/arm-smmu.c    | 58 +++++++--------------------------------------
 2 files changed, 10 insertions(+), 94 deletions(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 7bb157c..eba4ac8 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -2743,51 +2743,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 091d4a6..faea579 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2190,56 +2190,14 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
-
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", arm_smmu_of_init);
-IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", arm_smmu_of_init);
-IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init arm_smmu_acpi_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu, ACPI_SIG_IORT, arm_smmu_acpi_init);
-#endif
+module_platform_driver(arm_smmu_driver);
+
+IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", NULL);
+IOMMU_OF_DECLARE(arm_smmuv2, "arm,smmu-v2", NULL);
+IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400", NULL);
+IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401", NULL);
+IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500", NULL);
+IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 11/11] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index d01bae8..ac45623 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -992,6 +992,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 4e09b28..93ea70a 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -567,7 +567,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 11/11] ACPI/IORT: Remove linker section for IORT entries probing

[Sricharan R]

From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The IORT linker section introduced by commit 34ceea275f62
("ACPI/IORT: Introduce linker section for IORT entries probing")
was needed to make sure SMMU drivers are registered (and therefore
probed) in the kernel before devices using the SMMU have a chance
to probe in turn.

Through the introduction of deferred IOMMU configuration the linker
section based IORT probing infrastructure is not needed any longer, in
that device/SMMU probe dependencies are managed through the probe
deferral mechanism, making the IORT linker section infrastructure
unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure
from the kernel to complete the ACPI IORT IOMMU configure probe
deferral mechanism implementation.

Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index d01bae8..ac45623 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -992,6 +992,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 4e09b28..93ea70a 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -567,7 +567,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 77e0809..f167e1d04 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

RE: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Lorenzo, Robin,

>-----Original Message-----
>From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>Sent: Friday, February 03, 2017 9:19 PM
>To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-foundation.org;
>linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com; linux-
>pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>Cc: sricharan@codeaurora.org
>Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>
>This is an equivalent to the DT's handling of the iommu master's probe
>with deferred probing when the corrsponding iommu is not probed yet.
>The lack of a registered IOMMU can be caused by the lack of a driver for
>the IOMMU, the IOMMU device probe not having been performed yet, having
>been deferred, or having failed.
>
>The first case occurs when the firmware describes the bus master and
>IOMMU topology correctly but no device driver exists for the IOMMU yet
>or the device driver has not been compiled in. Return NULL, the caller
>will configure the device without an IOMMU.
>
>The second and third cases are handled by deferring the probe of the bus
>master device which will eventually get reprobed after the IOMMU.
>
>The last case is currently handled by deferring the probe of the bus
>master device as well. A mechanism to either configure the bus master
>device without an IOMMU or to fail the bus master device probe depending
>on whether the IOMMU is optional or mandatory would be a good
>enhancement.
>
>Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>---
> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
> drivers/acpi/scan.c        |  7 +++++--
> drivers/base/dma-mapping.c |  2 +-
> include/acpi/acpi_bus.h    |  2 +-
> include/linux/acpi.h       |  7 +++++--
> 5 files changed, 36 insertions(+), 7 deletions(-)
>
>diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>index bf0ed09..d01bae8 100644
>--- a/drivers/acpi/arm64/iort.c
>+++ b/drivers/acpi/arm64/iort.c
>@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
> 			return NULL;
>
> 		ops = iommu_get_instance(iort_fwnode);
>+		/*
>+		 * If the ops look-up fails, this means that either
>+		 * the SMMU drivers have not been probed yet or that
>+		 * the SMMU drivers are not built in the kernel;
>+		 * Depending on whether the SMMU drivers are built-in
>+		 * in the kernel or not, defer the IOMMU configuration
>+		 * or just abort it.
>+		 */
> 		if (!ops)
>-			return NULL;
>+			return iort_iommu_driver_enabled(node->type) ?
>+			       ERR_PTR(-EPROBE_DEFER) : NULL;
>
> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
> 	}
>@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>
> 		while (parent) {
> 			ops = iort_iommu_xlate(dev, parent, streamid);
>+			if (IS_ERR_OR_NULL(ops))
>+				return ops;
>
> 			parent = iort_node_get_id(node, &streamid,
> 						  IORT_IOMMU_TYPE, i++);
> 		}
> 	}
>
>+	/*
>+	 * If we have reason to believe the IOMMU driver missed the initial
>+	 * add_device callback for dev, replay it to get things in order.
>+	 */
>+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>+	    dev->bus && !dev->iommu_group) {
>+		int err = ops->add_device(dev);
>+
>+		if (err)
>+			ops = ERR_PTR(err);
>+	}
>+

On the last post we discussed that the above replay hunk can be made
common. In trying to do that, i ended up with a patch like below. But not
sure if that should be a part of this series though. I tested with OF devices
and would have to be tested with ACPI devices once. Nothing changes
functionally because of this ideally. Should be split in two patches though.

Regards,
 Sricharan

>From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Fri, 3 Feb 2017 15:24:47 +0530
Subject: [PATCH] drivers: iommu: Add iommu_add_device api

The code to call IOMMU driver's add_device is same
for both OF and ACPI cases. So add an api which can
be shared across both the places.

Also, now with probe-deferral the iommu master devices gets
added to the respective iommus during probe time instead
of device creation time. The xlate callbacks of iommu
drivers are also called only at probe time. As a result
the add_iommu_group which gets called when the iommu is
registered to add all devices created before the iommu
becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
also is not needed. So just cleanup those code.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 12 +-------
 drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
 drivers/iommu/of_iommu.c  | 11 +-------
 include/linux/iommu.h     |  8 ++++++
 4 files changed, 27 insertions(+), 74 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index ac45623..ab2a554 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
                }
        }

-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);
-
-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b752c3d..750552d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
        return group->default_domain;
 }

-static int add_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-       int ret;
-
-       if (!ops->add_device)
-               return 0;
-
-       WARN_ON(dev->iommu_group);
-
-       ret = ops->add_device(dev);
-
-       /*
-        * We ignore -ENODEV errors for now, as they just mean that the
-        * device is not translated by an IOMMU. We still care about
-        * other errors and fail to initialize when they happen.
-        */
-       if (ret == -ENODEV)
-               ret = 0;
-
-       return ret;
-}
-
-static int remove_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-
-       if (ops->remove_device && dev->iommu_group)
-               ops->remove_device(dev);
-
-       return 0;
-}
-
 static int iommu_bus_notifier(struct notifier_block *nb,
                              unsigned long action, void *data)
 {
@@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
        unsigned long group_action = 0;

        /*
-        * ADD/DEL call into iommu driver ops if provided, which may
+        * DEL call into iommu driver ops if provided, which may
         * result in ADD/DEL notifiers to group->notifier
         */
-       if (action == BUS_NOTIFY_ADD_DEVICE) {
-               if (ops->add_device)
-                       return ops->add_device(dev);
-       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
+       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
                if (ops->remove_device && dev->iommu_group) {
                        ops->remove_device(dev);
                        return 0;
@@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
 {
        int err;
        struct notifier_block *nb;
-       struct iommu_callback_data cb = {
-               .ops = ops,
-       };

        nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
        if (!nb)
@@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
        if (err)
                goto out_free;

-       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
-       if (err)
-               goto out_err;
-
-
        return 0;

-out_err:
-       /* Clean up */
-       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
-       bus_unregister_notifier(bus, nb);
-
 out_free:
        kfree(nb);

@@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
        return ret;
 }

+const struct iommu_ops *iommu_add_device(struct device *dev,
+                                        const struct iommu_ops *ops)
+{
+       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+           dev->bus && !dev->iommu_group) {
+               int err = ops->add_device(dev);
+
+               if (err)
+                       ops = ERR_PTR(err);
+       }
+
+       return ops;
+}
+EXPORT_SYMBOL_GPL(iommu_add_device);
+
 int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 {
        struct iommu_group *group;
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2d04663..4b49dc2 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
        else
                ops = of_platform_iommu_init(dev, master_np);
-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);

-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index add30c3..1d54a91 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -232,6 +232,8 @@ struct iommu_ops {
 extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
 extern struct iommu_group *iommu_group_get_by_id(int id);
 extern void iommu_domain_free(struct iommu_domain *domain);
+extern const struct iommu_ops *iommu_add_device(struct device *dev,
+                                               const struct iommu_ops *ops);
 extern int iommu_attach_device(struct iommu_domain *domain,
                               struct device *dev);
 extern void iommu_detach_device(struct iommu_domain *domain,
@@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
 {
 }

+static inline const struct iommu_ops *iommu_add_device(struct device *dev,
+                                             const struct iommu_ops *ops)
+{
+       return NULL;
+}
+
 static inline int iommu_attach_device(struct iommu_domain *domain,
                                      struct device *dev)
 {
--
1.8.2.1

RE: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Lorenzo, Robin,

>-----Original Message-----
>From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>Sent: Friday, February 03, 2017 9:19 PM
>To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-foundation.org;
>linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com; linux-
>pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>Cc: sricharan@codeaurora.org
>Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>
>This is an equivalent to the DT's handling of the iommu master's probe
>with deferred probing when the corrsponding iommu is not probed yet.
>The lack of a registered IOMMU can be caused by the lack of a driver for
>the IOMMU, the IOMMU device probe not having been performed yet, having
>been deferred, or having failed.
>
>The first case occurs when the firmware describes the bus master and
>IOMMU topology correctly but no device driver exists for the IOMMU yet
>or the device driver has not been compiled in. Return NULL, the caller
>will configure the device without an IOMMU.
>
>The second and third cases are handled by deferring the probe of the bus
>master device which will eventually get reprobed after the IOMMU.
>
>The last case is currently handled by deferring the probe of the bus
>master device as well. A mechanism to either configure the bus master
>device without an IOMMU or to fail the bus master device probe depending
>on whether the IOMMU is optional or mandatory would be a good
>enhancement.
>
>Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>---
> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
> drivers/acpi/scan.c        |  7 +++++--
> drivers/base/dma-mapping.c |  2 +-
> include/acpi/acpi_bus.h    |  2 +-
> include/linux/acpi.h       |  7 +++++--
> 5 files changed, 36 insertions(+), 7 deletions(-)
>
>diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>index bf0ed09..d01bae8 100644
>--- a/drivers/acpi/arm64/iort.c
>+++ b/drivers/acpi/arm64/iort.c
>@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
> 			return NULL;
>
> 		ops = iommu_get_instance(iort_fwnode);
>+		/*
>+		 * If the ops look-up fails, this means that either
>+		 * the SMMU drivers have not been probed yet or that
>+		 * the SMMU drivers are not built in the kernel;
>+		 * Depending on whether the SMMU drivers are built-in
>+		 * in the kernel or not, defer the IOMMU configuration
>+		 * or just abort it.
>+		 */
> 		if (!ops)
>-			return NULL;
>+			return iort_iommu_driver_enabled(node->type) ?
>+			       ERR_PTR(-EPROBE_DEFER) : NULL;
>
> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
> 	}
>@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>
> 		while (parent) {
> 			ops = iort_iommu_xlate(dev, parent, streamid);
>+			if (IS_ERR_OR_NULL(ops))
>+				return ops;
>
> 			parent = iort_node_get_id(node, &streamid,
> 						  IORT_IOMMU_TYPE, i++);
> 		}
> 	}
>
>+	/*
>+	 * If we have reason to believe the IOMMU driver missed the initial
>+	 * add_device callback for dev, replay it to get things in order.
>+	 */
>+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>+	    dev->bus && !dev->iommu_group) {
>+		int err = ops->add_device(dev);
>+
>+		if (err)
>+			ops = ERR_PTR(err);
>+	}
>+

On the last post we discussed that the above replay hunk can be made
common. In trying to do that, i ended up with a patch like below. But not
sure if that should be a part of this series though. I tested with OF devices
and would have to be tested with ACPI devices once. Nothing changes
functionally because of this ideally. Should be split in two patches though.

Regards,
 Sricharan

>From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Fri, 3 Feb 2017 15:24:47 +0530
Subject: [PATCH] drivers: iommu: Add iommu_add_device api

The code to call IOMMU driver's add_device is same
for both OF and ACPI cases. So add an api which can
be shared across both the places.

Also, now with probe-deferral the iommu master devices gets
added to the respective iommus during probe time instead
of device creation time. The xlate callbacks of iommu
drivers are also called only at probe time. As a result
the add_iommu_group which gets called when the iommu is
registered to add all devices created before the iommu
becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
also is not needed. So just cleanup those code.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 12 +-------
 drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
 drivers/iommu/of_iommu.c  | 11 +-------
 include/linux/iommu.h     |  8 ++++++
 4 files changed, 27 insertions(+), 74 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index ac45623..ab2a554 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
                }
        }

-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);
-
-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b752c3d..750552d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
        return group->default_domain;
 }

-static int add_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-       int ret;
-
-       if (!ops->add_device)
-               return 0;
-
-       WARN_ON(dev->iommu_group);
-
-       ret = ops->add_device(dev);
-
-       /*
-        * We ignore -ENODEV errors for now, as they just mean that the
-        * device is not translated by an IOMMU. We still care about
-        * other errors and fail to initialize when they happen.
-        */
-       if (ret == -ENODEV)
-               ret = 0;
-
-       return ret;
-}
-
-static int remove_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-
-       if (ops->remove_device && dev->iommu_group)
-               ops->remove_device(dev);
-
-       return 0;
-}
-
 static int iommu_bus_notifier(struct notifier_block *nb,
                              unsigned long action, void *data)
 {
@@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
        unsigned long group_action = 0;

        /*
-        * ADD/DEL call into iommu driver ops if provided, which may
+        * DEL call into iommu driver ops if provided, which may
         * result in ADD/DEL notifiers to group->notifier
         */
-       if (action == BUS_NOTIFY_ADD_DEVICE) {
-               if (ops->add_device)
-                       return ops->add_device(dev);
-       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
+       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
                if (ops->remove_device && dev->iommu_group) {
                        ops->remove_device(dev);
                        return 0;
@@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
 {
        int err;
        struct notifier_block *nb;
-       struct iommu_callback_data cb = {
-               .ops = ops,
-       };

        nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
        if (!nb)
@@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
        if (err)
                goto out_free;

-       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
-       if (err)
-               goto out_err;
-
-
        return 0;

-out_err:
-       /* Clean up */
-       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
-       bus_unregister_notifier(bus, nb);
-
 out_free:
        kfree(nb);

@@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
        return ret;
 }

+const struct iommu_ops *iommu_add_device(struct device *dev,
+                                        const struct iommu_ops *ops)
+{
+       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+           dev->bus && !dev->iommu_group) {
+               int err = ops->add_device(dev);
+
+               if (err)
+                       ops = ERR_PTR(err);
+       }
+
+       return ops;
+}
+EXPORT_SYMBOL_GPL(iommu_add_device);
+
 int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 {
        struct iommu_group *group;
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2d04663..4b49dc2 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
        else
                ops = of_platform_iommu_init(dev, master_np);
-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);

-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index add30c3..1d54a91 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -232,6 +232,8 @@ struct iommu_ops {
 extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
 extern struct iommu_group *iommu_group_get_by_id(int id);
 extern void iommu_domain_free(struct iommu_domain *domain);
+extern const struct iommu_ops *iommu_add_device(struct device *dev,
+                                               const struct iommu_ops *ops);
 extern int iommu_attach_device(struct iommu_domain *domain,
                               struct device *dev);
 extern void iommu_detach_device(struct iommu_domain *domain,
@@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
 {
 }

+static inline const struct iommu_ops *iommu_add_device(struct device *dev,
+                                             const struct iommu_ops *ops)
+{
+       return NULL;
+}
+
 static inline int iommu_attach_device(struct iommu_domain *domain,
                                      struct device *dev)
 {
--
1.8.2.1

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Lorenzo, Robin,

>-----Original Message-----
>From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Sricharan R
>Sent: Friday, February 03, 2017 9:19 PM
>To: robin.murphy at arm.com; will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-foundation.org;
>linux-arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com; bhelgaas at google.com; linux-
>pci at vger.kernel.org; linux-acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org; okaya at codeaurora.org
>Cc: sricharan at codeaurora.org
>Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>
>This is an equivalent to the DT's handling of the iommu master's probe
>with deferred probing when the corrsponding iommu is not probed yet.
>The lack of a registered IOMMU can be caused by the lack of a driver for
>the IOMMU, the IOMMU device probe not having been performed yet, having
>been deferred, or having failed.
>
>The first case occurs when the firmware describes the bus master and
>IOMMU topology correctly but no device driver exists for the IOMMU yet
>or the device driver has not been compiled in. Return NULL, the caller
>will configure the device without an IOMMU.
>
>The second and third cases are handled by deferring the probe of the bus
>master device which will eventually get reprobed after the IOMMU.
>
>The last case is currently handled by deferring the probe of the bus
>master device as well. A mechanism to either configure the bus master
>device without an IOMMU or to fail the bus master device probe depending
>on whether the IOMMU is optional or mandatory would be a good
>enhancement.
>
>Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>---
> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
> drivers/acpi/scan.c        |  7 +++++--
> drivers/base/dma-mapping.c |  2 +-
> include/acpi/acpi_bus.h    |  2 +-
> include/linux/acpi.h       |  7 +++++--
> 5 files changed, 36 insertions(+), 7 deletions(-)
>
>diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>index bf0ed09..d01bae8 100644
>--- a/drivers/acpi/arm64/iort.c
>+++ b/drivers/acpi/arm64/iort.c
>@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
> 			return NULL;
>
> 		ops = iommu_get_instance(iort_fwnode);
>+		/*
>+		 * If the ops look-up fails, this means that either
>+		 * the SMMU drivers have not been probed yet or that
>+		 * the SMMU drivers are not built in the kernel;
>+		 * Depending on whether the SMMU drivers are built-in
>+		 * in the kernel or not, defer the IOMMU configuration
>+		 * or just abort it.
>+		 */
> 		if (!ops)
>-			return NULL;
>+			return iort_iommu_driver_enabled(node->type) ?
>+			       ERR_PTR(-EPROBE_DEFER) : NULL;
>
> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
> 	}
>@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>
> 		while (parent) {
> 			ops = iort_iommu_xlate(dev, parent, streamid);
>+			if (IS_ERR_OR_NULL(ops))
>+				return ops;
>
> 			parent = iort_node_get_id(node, &streamid,
> 						  IORT_IOMMU_TYPE, i++);
> 		}
> 	}
>
>+	/*
>+	 * If we have reason to believe the IOMMU driver missed the initial
>+	 * add_device callback for dev, replay it to get things in order.
>+	 */
>+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>+	    dev->bus && !dev->iommu_group) {
>+		int err = ops->add_device(dev);
>+
>+		if (err)
>+			ops = ERR_PTR(err);
>+	}
>+

On the last post we discussed that the above replay hunk can be made
common. In trying to do that, i ended up with a patch like below. But not
sure if that should be a part of this series though. I tested with OF devices
and would have to be tested with ACPI devices once. Nothing changes
functionally because of this ideally. Should be split in two patches though.

Regards,
 Sricharan

>From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Fri, 3 Feb 2017 15:24:47 +0530
Subject: [PATCH] drivers: iommu: Add iommu_add_device api

The code to call IOMMU driver's add_device is same
for both OF and ACPI cases. So add an api which can
be shared across both the places.

Also, now with probe-deferral the iommu master devices gets
added to the respective iommus during probe time instead
of device creation time. The xlate callbacks of iommu
drivers are also called only at probe time. As a result
the add_iommu_group which gets called when the iommu is
registered to add all devices created before the iommu
becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
also is not needed. So just cleanup those code.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/acpi/arm64/iort.c | 12 +-------
 drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
 drivers/iommu/of_iommu.c  | 11 +-------
 include/linux/iommu.h     |  8 ++++++
 4 files changed, 27 insertions(+), 74 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index ac45623..ab2a554 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
                }
        }

-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);
-
-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b752c3d..750552d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
        return group->default_domain;
 }

-static int add_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-       int ret;
-
-       if (!ops->add_device)
-               return 0;
-
-       WARN_ON(dev->iommu_group);
-
-       ret = ops->add_device(dev);
-
-       /*
-        * We ignore -ENODEV errors for now, as they just mean that the
-        * device is not translated by an IOMMU. We still care about
-        * other errors and fail to initialize when they happen.
-        */
-       if (ret == -ENODEV)
-               ret = 0;
-
-       return ret;
-}
-
-static int remove_iommu_group(struct device *dev, void *data)
-{
-       struct iommu_callback_data *cb = data;
-       const struct iommu_ops *ops = cb->ops;
-
-       if (ops->remove_device && dev->iommu_group)
-               ops->remove_device(dev);
-
-       return 0;
-}
-
 static int iommu_bus_notifier(struct notifier_block *nb,
                              unsigned long action, void *data)
 {
@@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
        unsigned long group_action = 0;

        /*
-        * ADD/DEL call into iommu driver ops if provided, which may
+        * DEL call into iommu driver ops if provided, which may
         * result in ADD/DEL notifiers to group->notifier
         */
-       if (action == BUS_NOTIFY_ADD_DEVICE) {
-               if (ops->add_device)
-                       return ops->add_device(dev);
-       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
+       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
                if (ops->remove_device && dev->iommu_group) {
                        ops->remove_device(dev);
                        return 0;
@@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
 {
        int err;
        struct notifier_block *nb;
-       struct iommu_callback_data cb = {
-               .ops = ops,
-       };

        nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
        if (!nb)
@@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
        if (err)
                goto out_free;

-       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
-       if (err)
-               goto out_err;
-
-
        return 0;

-out_err:
-       /* Clean up */
-       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
-       bus_unregister_notifier(bus, nb);
-
 out_free:
        kfree(nb);

@@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
        return ret;
 }

+const struct iommu_ops *iommu_add_device(struct device *dev,
+                                        const struct iommu_ops *ops)
+{
+       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+           dev->bus && !dev->iommu_group) {
+               int err = ops->add_device(dev);
+
+               if (err)
+                       ops = ERR_PTR(err);
+       }
+
+       return ops;
+}
+EXPORT_SYMBOL_GPL(iommu_add_device);
+
 int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
 {
        struct iommu_group *group;
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 2d04663..4b49dc2 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
        else
                ops = of_platform_iommu_init(dev, master_np);
-       /*
-        * If we have reason to believe the IOMMU driver missed the initial
-        * add_device callback for dev, replay it to get things in order.
-        */
-       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
-           dev->bus && !dev->iommu_group) {
-               int err = ops->add_device(dev);

-               if (err)
-                       ops = ERR_PTR(err);
-       }
+       ops = iommu_add_device(dev, ops);

        return ops;
 }
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index add30c3..1d54a91 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -232,6 +232,8 @@ struct iommu_ops {
 extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
 extern struct iommu_group *iommu_group_get_by_id(int id);
 extern void iommu_domain_free(struct iommu_domain *domain);
+extern const struct iommu_ops *iommu_add_device(struct device *dev,
+                                               const struct iommu_ops *ops);
 extern int iommu_attach_device(struct iommu_domain *domain,
                               struct device *dev);
 extern void iommu_detach_device(struct iommu_domain *domain,
@@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
 {
 }

+static inline const struct iommu_ops *iommu_add_device(struct device *dev,
+                                             const struct iommu_ops *ops)
+{
+       return NULL;
+}
+
 static inline int iommu_attach_device(struct iommu_domain *domain,
                                      struct device *dev)
 {
--
1.8.2.1

Re: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 03/02/17 16:15, Sricharan wrote:
> Hi Lorenzo, Robin,
> 
>> -----Original Message-----
>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>> Sent: Friday, February 03, 2017 9:19 PM
>> To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-foundation.org;
>> linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com; linux-
>> pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>> Cc: sricharan@codeaurora.org
>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>
>> This is an equivalent to the DT's handling of the iommu master's probe
>> with deferred probing when the corrsponding iommu is not probed yet.
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the firmware describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>> drivers/acpi/scan.c        |  7 +++++--
>> drivers/base/dma-mapping.c |  2 +-
>> include/acpi/acpi_bus.h    |  2 +-
>> include/linux/acpi.h       |  7 +++++--
>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index bf0ed09..d01bae8 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>> 			return NULL;
>>
>> 		ops = iommu_get_instance(iort_fwnode);
>> +		/*
>> +		 * If the ops look-up fails, this means that either
>> +		 * the SMMU drivers have not been probed yet or that
>> +		 * the SMMU drivers are not built in the kernel;
>> +		 * Depending on whether the SMMU drivers are built-in
>> +		 * in the kernel or not, defer the IOMMU configuration
>> +		 * or just abort it.
>> +		 */
>> 		if (!ops)
>> -			return NULL;
>> +			return iort_iommu_driver_enabled(node->type) ?
>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>
>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>> 	}
>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>
>> 		while (parent) {
>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>> +			if (IS_ERR_OR_NULL(ops))
>> +				return ops;
>>
>> 			parent = iort_node_get_id(node, &streamid,
>> 						  IORT_IOMMU_TYPE, i++);
>> 		}
>> 	}
>>
>> +	/*
>> +	 * If we have reason to believe the IOMMU driver missed the initial
>> +	 * add_device callback for dev, replay it to get things in order.
>> +	 */
>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> +	    dev->bus && !dev->iommu_group) {
>> +		int err = ops->add_device(dev);
>> +
>> +		if (err)
>> +			ops = ERR_PTR(err);
>> +	}
>> +
> 
> On the last post we discussed that the above replay hunk can be made
> common. In trying to do that, i ended up with a patch like below. But not
> sure if that should be a part of this series though. I tested with OF devices
> and would have to be tested with ACPI devices once. Nothing changes
> functionally because of this ideally. Should be split in two patches though.
> 
> Regards,
>  Sricharan
> 
> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Fri, 3 Feb 2017 15:24:47 +0530
> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
> 
> The code to call IOMMU driver's add_device is same
> for both OF and ACPI cases. So add an api which can
> be shared across both the places.
> 
> Also, now with probe-deferral the iommu master devices gets
> added to the respective iommus during probe time instead
> of device creation time. The xlate callbacks of iommu
> drivers are also called only at probe time. As a result
> the add_iommu_group which gets called when the iommu is
> registered to add all devices created before the iommu
> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
> also is not needed. So just cleanup those code.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 12 +-------
>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>  drivers/iommu/of_iommu.c  | 11 +-------
>  include/linux/iommu.h     |  8 ++++++
>  4 files changed, 27 insertions(+), 74 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index ac45623..ab2a554 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>                 }
>         }
> 
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> -
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index b752c3d..750552d 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>         return group->default_domain;
>  }
> 
> -static int add_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -       int ret;
> -
> -       if (!ops->add_device)
> -               return 0;
> -
> -       WARN_ON(dev->iommu_group);
> -
> -       ret = ops->add_device(dev);
> -
> -       /*
> -        * We ignore -ENODEV errors for now, as they just mean that the
> -        * device is not translated by an IOMMU. We still care about
> -        * other errors and fail to initialize when they happen.
> -        */
> -       if (ret == -ENODEV)
> -               ret = 0;
> -
> -       return ret;
> -}
> -
> -static int remove_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -
> -       if (ops->remove_device && dev->iommu_group)
> -               ops->remove_device(dev);
> -
> -       return 0;
> -}
> -
>  static int iommu_bus_notifier(struct notifier_block *nb,
>                               unsigned long action, void *data)
>  {
> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>         unsigned long group_action = 0;
> 
>         /*
> -        * ADD/DEL call into iommu driver ops if provided, which may
> +        * DEL call into iommu driver ops if provided, which may
>          * result in ADD/DEL notifiers to group->notifier
>          */
> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
> -               if (ops->add_device)
> -                       return ops->add_device(dev);

I'm pretty sure this completely breaks x86 and anyone else...

Anyway, I'd be inclined to leave this alone for now - it's essentially
just a workaround to mesh the per-instance probing behaviour with the
per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
code over to a per-instance model will inevitably subsume the underlying
problem, and I think a bit of duplication is probably the lesser of two
evils in the meantime. On which note, I need to have a good look at
Joerg's shiny new series :)

Robin.

> -       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
> +       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
>                 if (ops->remove_device && dev->iommu_group) {
>                         ops->remove_device(dev);
>                         return 0;
> @@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>  {
>         int err;
>         struct notifier_block *nb;
> -       struct iommu_callback_data cb = {
> -               .ops = ops,
> -       };
> 
>         nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
>         if (!nb)
> @@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>         if (err)
>                 goto out_free;
> 
> -       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
> -       if (err)
> -               goto out_err;
> -
> -
>         return 0;
> 
> -out_err:
> -       /* Clean up */
> -       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
> -       bus_unregister_notifier(bus, nb);
> -
>  out_free:
>         kfree(nb);
> 
> @@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
>         return ret;
>  }
> 
> +const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                        const struct iommu_ops *ops)
> +{
> +       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +           dev->bus && !dev->iommu_group) {
> +               int err = ops->add_device(dev);
> +
> +               if (err)
> +                       ops = ERR_PTR(err);
> +       }
> +
> +       return ops;
> +}
> +EXPORT_SYMBOL_GPL(iommu_add_device);
> +
>  int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
>  {
>         struct iommu_group *group;
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 2d04663..4b49dc2 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                 ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
>         else
>                 ops = of_platform_iommu_init(dev, master_np);
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> 
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index add30c3..1d54a91 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -232,6 +232,8 @@ struct iommu_ops {
>  extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
>  extern struct iommu_group *iommu_group_get_by_id(int id);
>  extern void iommu_domain_free(struct iommu_domain *domain);
> +extern const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                               const struct iommu_ops *ops);
>  extern int iommu_attach_device(struct iommu_domain *domain,
>                                struct device *dev);
>  extern void iommu_detach_device(struct iommu_domain *domain,
> @@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
>  {
>  }
> 
> +static inline const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                             const struct iommu_ops *ops)
> +{
> +       return NULL;
> +}
> +
>  static inline int iommu_attach_device(struct iommu_domain *domain,
>                                       struct device *dev)
>  {
> --
> 1.8.2.1
> 

Re: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 03/02/17 16:15, Sricharan wrote:
> Hi Lorenzo, Robin,
> 
>> -----Original Message-----
>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>> Sent: Friday, February 03, 2017 9:19 PM
>> To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-foundation.org;
>> linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com; linux-
>> pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>> Cc: sricharan@codeaurora.org
>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>
>> This is an equivalent to the DT's handling of the iommu master's probe
>> with deferred probing when the corrsponding iommu is not probed yet.
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the firmware describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>> drivers/acpi/scan.c        |  7 +++++--
>> drivers/base/dma-mapping.c |  2 +-
>> include/acpi/acpi_bus.h    |  2 +-
>> include/linux/acpi.h       |  7 +++++--
>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index bf0ed09..d01bae8 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>> 			return NULL;
>>
>> 		ops = iommu_get_instance(iort_fwnode);
>> +		/*
>> +		 * If the ops look-up fails, this means that either
>> +		 * the SMMU drivers have not been probed yet or that
>> +		 * the SMMU drivers are not built in the kernel;
>> +		 * Depending on whether the SMMU drivers are built-in
>> +		 * in the kernel or not, defer the IOMMU configuration
>> +		 * or just abort it.
>> +		 */
>> 		if (!ops)
>> -			return NULL;
>> +			return iort_iommu_driver_enabled(node->type) ?
>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>
>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>> 	}
>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>
>> 		while (parent) {
>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>> +			if (IS_ERR_OR_NULL(ops))
>> +				return ops;
>>
>> 			parent = iort_node_get_id(node, &streamid,
>> 						  IORT_IOMMU_TYPE, i++);
>> 		}
>> 	}
>>
>> +	/*
>> +	 * If we have reason to believe the IOMMU driver missed the initial
>> +	 * add_device callback for dev, replay it to get things in order.
>> +	 */
>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> +	    dev->bus && !dev->iommu_group) {
>> +		int err = ops->add_device(dev);
>> +
>> +		if (err)
>> +			ops = ERR_PTR(err);
>> +	}
>> +
> 
> On the last post we discussed that the above replay hunk can be made
> common. In trying to do that, i ended up with a patch like below. But not
> sure if that should be a part of this series though. I tested with OF devices
> and would have to be tested with ACPI devices once. Nothing changes
> functionally because of this ideally. Should be split in two patches though.
> 
> Regards,
>  Sricharan
> 
> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Fri, 3 Feb 2017 15:24:47 +0530
> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
> 
> The code to call IOMMU driver's add_device is same
> for both OF and ACPI cases. So add an api which can
> be shared across both the places.
> 
> Also, now with probe-deferral the iommu master devices gets
> added to the respective iommus during probe time instead
> of device creation time. The xlate callbacks of iommu
> drivers are also called only at probe time. As a result
> the add_iommu_group which gets called when the iommu is
> registered to add all devices created before the iommu
> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
> also is not needed. So just cleanup those code.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 12 +-------
>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>  drivers/iommu/of_iommu.c  | 11 +-------
>  include/linux/iommu.h     |  8 ++++++
>  4 files changed, 27 insertions(+), 74 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index ac45623..ab2a554 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>                 }
>         }
> 
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> -
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index b752c3d..750552d 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>         return group->default_domain;
>  }
> 
> -static int add_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -       int ret;
> -
> -       if (!ops->add_device)
> -               return 0;
> -
> -       WARN_ON(dev->iommu_group);
> -
> -       ret = ops->add_device(dev);
> -
> -       /*
> -        * We ignore -ENODEV errors for now, as they just mean that the
> -        * device is not translated by an IOMMU. We still care about
> -        * other errors and fail to initialize when they happen.
> -        */
> -       if (ret == -ENODEV)
> -               ret = 0;
> -
> -       return ret;
> -}
> -
> -static int remove_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -
> -       if (ops->remove_device && dev->iommu_group)
> -               ops->remove_device(dev);
> -
> -       return 0;
> -}
> -
>  static int iommu_bus_notifier(struct notifier_block *nb,
>                               unsigned long action, void *data)
>  {
> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>         unsigned long group_action = 0;
> 
>         /*
> -        * ADD/DEL call into iommu driver ops if provided, which may
> +        * DEL call into iommu driver ops if provided, which may
>          * result in ADD/DEL notifiers to group->notifier
>          */
> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
> -               if (ops->add_device)
> -                       return ops->add_device(dev);

I'm pretty sure this completely breaks x86 and anyone else...

Anyway, I'd be inclined to leave this alone for now - it's essentially
just a workaround to mesh the per-instance probing behaviour with the
per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
code over to a per-instance model will inevitably subsume the underlying
problem, and I think a bit of duplication is probably the lesser of two
evils in the meantime. On which note, I need to have a good look at
Joerg's shiny new series :)

Robin.

> -       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
> +       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
>                 if (ops->remove_device && dev->iommu_group) {
>                         ops->remove_device(dev);
>                         return 0;
> @@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>  {
>         int err;
>         struct notifier_block *nb;
> -       struct iommu_callback_data cb = {
> -               .ops = ops,
> -       };
> 
>         nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
>         if (!nb)
> @@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>         if (err)
>                 goto out_free;
> 
> -       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
> -       if (err)
> -               goto out_err;
> -
> -
>         return 0;
> 
> -out_err:
> -       /* Clean up */
> -       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
> -       bus_unregister_notifier(bus, nb);
> -
>  out_free:
>         kfree(nb);
> 
> @@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
>         return ret;
>  }
> 
> +const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                        const struct iommu_ops *ops)
> +{
> +       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +           dev->bus && !dev->iommu_group) {
> +               int err = ops->add_device(dev);
> +
> +               if (err)
> +                       ops = ERR_PTR(err);
> +       }
> +
> +       return ops;
> +}
> +EXPORT_SYMBOL_GPL(iommu_add_device);
> +
>  int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
>  {
>         struct iommu_group *group;
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 2d04663..4b49dc2 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                 ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
>         else
>                 ops = of_platform_iommu_init(dev, master_np);
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> 
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index add30c3..1d54a91 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -232,6 +232,8 @@ struct iommu_ops {
>  extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
>  extern struct iommu_group *iommu_group_get_by_id(int id);
>  extern void iommu_domain_free(struct iommu_domain *domain);
> +extern const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                               const struct iommu_ops *ops);
>  extern int iommu_attach_device(struct iommu_domain *domain,
>                                struct device *dev);
>  extern void iommu_detach_device(struct iommu_domain *domain,
> @@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
>  {
>  }
> 
> +static inline const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                             const struct iommu_ops *ops)
> +{
> +       return NULL;
> +}
> +
>  static inline int iommu_attach_device(struct iommu_domain *domain,
>                                       struct device *dev)
>  {
> --
> 1.8.2.1
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 03/02/17 16:15, Sricharan wrote:
> Hi Lorenzo, Robin,
> 
>> -----Original Message-----
>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Sricharan R
>> Sent: Friday, February 03, 2017 9:19 PM
>> To: robin.murphy at arm.com; will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-foundation.org;
>> linux-arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com; bhelgaas at google.com; linux-
>> pci at vger.kernel.org; linux-acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org; okaya at codeaurora.org
>> Cc: sricharan at codeaurora.org
>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>
>> This is an equivalent to the DT's handling of the iommu master's probe
>> with deferred probing when the corrsponding iommu is not probed yet.
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the firmware describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>> drivers/acpi/scan.c        |  7 +++++--
>> drivers/base/dma-mapping.c |  2 +-
>> include/acpi/acpi_bus.h    |  2 +-
>> include/linux/acpi.h       |  7 +++++--
>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index bf0ed09..d01bae8 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>> 			return NULL;
>>
>> 		ops = iommu_get_instance(iort_fwnode);
>> +		/*
>> +		 * If the ops look-up fails, this means that either
>> +		 * the SMMU drivers have not been probed yet or that
>> +		 * the SMMU drivers are not built in the kernel;
>> +		 * Depending on whether the SMMU drivers are built-in
>> +		 * in the kernel or not, defer the IOMMU configuration
>> +		 * or just abort it.
>> +		 */
>> 		if (!ops)
>> -			return NULL;
>> +			return iort_iommu_driver_enabled(node->type) ?
>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>
>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>> 	}
>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>
>> 		while (parent) {
>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>> +			if (IS_ERR_OR_NULL(ops))
>> +				return ops;
>>
>> 			parent = iort_node_get_id(node, &streamid,
>> 						  IORT_IOMMU_TYPE, i++);
>> 		}
>> 	}
>>
>> +	/*
>> +	 * If we have reason to believe the IOMMU driver missed the initial
>> +	 * add_device callback for dev, replay it to get things in order.
>> +	 */
>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> +	    dev->bus && !dev->iommu_group) {
>> +		int err = ops->add_device(dev);
>> +
>> +		if (err)
>> +			ops = ERR_PTR(err);
>> +	}
>> +
> 
> On the last post we discussed that the above replay hunk can be made
> common. In trying to do that, i ended up with a patch like below. But not
> sure if that should be a part of this series though. I tested with OF devices
> and would have to be tested with ACPI devices once. Nothing changes
> functionally because of this ideally. Should be split in two patches though.
> 
> Regards,
>  Sricharan
> 
> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Fri, 3 Feb 2017 15:24:47 +0530
> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
> 
> The code to call IOMMU driver's add_device is same
> for both OF and ACPI cases. So add an api which can
> be shared across both the places.
> 
> Also, now with probe-deferral the iommu master devices gets
> added to the respective iommus during probe time instead
> of device creation time. The xlate callbacks of iommu
> drivers are also called only at probe time. As a result
> the add_iommu_group which gets called when the iommu is
> registered to add all devices created before the iommu
> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
> also is not needed. So just cleanup those code.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/acpi/arm64/iort.c | 12 +-------
>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>  drivers/iommu/of_iommu.c  | 11 +-------
>  include/linux/iommu.h     |  8 ++++++
>  4 files changed, 27 insertions(+), 74 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index ac45623..ab2a554 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>                 }
>         }
> 
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> -
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index b752c3d..750552d 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>         return group->default_domain;
>  }
> 
> -static int add_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -       int ret;
> -
> -       if (!ops->add_device)
> -               return 0;
> -
> -       WARN_ON(dev->iommu_group);
> -
> -       ret = ops->add_device(dev);
> -
> -       /*
> -        * We ignore -ENODEV errors for now, as they just mean that the
> -        * device is not translated by an IOMMU. We still care about
> -        * other errors and fail to initialize when they happen.
> -        */
> -       if (ret == -ENODEV)
> -               ret = 0;
> -
> -       return ret;
> -}
> -
> -static int remove_iommu_group(struct device *dev, void *data)
> -{
> -       struct iommu_callback_data *cb = data;
> -       const struct iommu_ops *ops = cb->ops;
> -
> -       if (ops->remove_device && dev->iommu_group)
> -               ops->remove_device(dev);
> -
> -       return 0;
> -}
> -
>  static int iommu_bus_notifier(struct notifier_block *nb,
>                               unsigned long action, void *data)
>  {
> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>         unsigned long group_action = 0;
> 
>         /*
> -        * ADD/DEL call into iommu driver ops if provided, which may
> +        * DEL call into iommu driver ops if provided, which may
>          * result in ADD/DEL notifiers to group->notifier
>          */
> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
> -               if (ops->add_device)
> -                       return ops->add_device(dev);

I'm pretty sure this completely breaks x86 and anyone else...

Anyway, I'd be inclined to leave this alone for now - it's essentially
just a workaround to mesh the per-instance probing behaviour with the
per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
code over to a per-instance model will inevitably subsume the underlying
problem, and I think a bit of duplication is probably the lesser of two
evils in the meantime. On which note, I need to have a good look at
Joerg's shiny new series :)

Robin.

> -       } else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
> +       if (action == BUS_NOTIFY_REMOVED_DEVICE) {
>                 if (ops->remove_device && dev->iommu_group) {
>                         ops->remove_device(dev);
>                         return 0;
> @@ -1107,9 +1069,6 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>  {
>         int err;
>         struct notifier_block *nb;
> -       struct iommu_callback_data cb = {
> -               .ops = ops,
> -       };
> 
>         nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
>         if (!nb)
> @@ -1121,18 +1080,8 @@ static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
>         if (err)
>                 goto out_free;
> 
> -       err = bus_for_each_dev(bus, NULL, &cb, add_iommu_group);
> -       if (err)
> -               goto out_err;
> -
> -
>         return 0;
> 
> -out_err:
> -       /* Clean up */
> -       bus_for_each_dev(bus, NULL, &cb, remove_iommu_group);
> -       bus_unregister_notifier(bus, nb);
> -
>  out_free:
>         kfree(nb);
> 
> @@ -1253,6 +1202,21 @@ static int __iommu_attach_device(struct iommu_domain *domain,
>         return ret;
>  }
> 
> +const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                        const struct iommu_ops *ops)
> +{
> +       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> +           dev->bus && !dev->iommu_group) {
> +               int err = ops->add_device(dev);
> +
> +               if (err)
> +                       ops = ERR_PTR(err);
> +       }
> +
> +       return ops;
> +}
> +EXPORT_SYMBOL_GPL(iommu_add_device);
> +
>  int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
>  {
>         struct iommu_group *group;
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 2d04663..4b49dc2 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -224,17 +224,8 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                 ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
>         else
>                 ops = of_platform_iommu_init(dev, master_np);
> -       /*
> -        * If we have reason to believe the IOMMU driver missed the initial
> -        * add_device callback for dev, replay it to get things in order.
> -        */
> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
> -           dev->bus && !dev->iommu_group) {
> -               int err = ops->add_device(dev);
> 
> -               if (err)
> -                       ops = ERR_PTR(err);
> -       }
> +       ops = iommu_add_device(dev, ops);
> 
>         return ops;
>  }
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index add30c3..1d54a91 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -232,6 +232,8 @@ struct iommu_ops {
>  extern struct iommu_domain *iommu_domain_alloc(struct bus_type *bus);
>  extern struct iommu_group *iommu_group_get_by_id(int id);
>  extern void iommu_domain_free(struct iommu_domain *domain);
> +extern const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                               const struct iommu_ops *ops);
>  extern int iommu_attach_device(struct iommu_domain *domain,
>                                struct device *dev);
>  extern void iommu_detach_device(struct iommu_domain *domain,
> @@ -405,6 +407,12 @@ static inline void iommu_domain_free(struct iommu_domain *domain)
>  {
>  }
> 
> +static inline const struct iommu_ops *iommu_add_device(struct device *dev,
> +                                             const struct iommu_ops *ops)
> +{
> +       return NULL;
> +}
> +
>  static inline int iommu_attach_device(struct iommu_domain *domain,
>                                       struct device *dev)
>  {
> --
> 1.8.2.1
> 

RE: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Robin,

>>
>>> -----Original Message-----
>>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>>> Sent: Friday, February 03, 2017 9:19 PM
>>> To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>foundation.org;
>>> linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com;
>linux-
>>> pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>>> Cc: sricharan@codeaurora.org
>>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>>
>>> This is an equivalent to the DT's handling of the iommu master's probe
>>> with deferred probing when the corrsponding iommu is not probed yet.
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the firmware describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>>> drivers/acpi/scan.c        |  7 +++++--
>>> drivers/base/dma-mapping.c |  2 +-
>>> include/acpi/acpi_bus.h    |  2 +-
>>> include/linux/acpi.h       |  7 +++++--
>>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>>> index bf0ed09..d01bae8 100644
>>> --- a/drivers/acpi/arm64/iort.c
>>> +++ b/drivers/acpi/arm64/iort.c
>>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>>> 			return NULL;
>>>
>>> 		ops = iommu_get_instance(iort_fwnode);
>>> +		/*
>>> +		 * If the ops look-up fails, this means that either
>>> +		 * the SMMU drivers have not been probed yet or that
>>> +		 * the SMMU drivers are not built in the kernel;
>>> +		 * Depending on whether the SMMU drivers are built-in
>>> +		 * in the kernel or not, defer the IOMMU configuration
>>> +		 * or just abort it.
>>> +		 */
>>> 		if (!ops)
>>> -			return NULL;
>>> +			return iort_iommu_driver_enabled(node->type) ?
>>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>>
>>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>>> 	}
>>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>>
>>> 		while (parent) {
>>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>>> +			if (IS_ERR_OR_NULL(ops))
>>> +				return ops;
>>>
>>> 			parent = iort_node_get_id(node, &streamid,
>>> 						  IORT_IOMMU_TYPE, i++);
>>> 		}
>>> 	}
>>>
>>> +	/*
>>> +	 * If we have reason to believe the IOMMU driver missed the initial
>>> +	 * add_device callback for dev, replay it to get things in order.
>>> +	 */
>>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>>> +	    dev->bus && !dev->iommu_group) {
>>> +		int err = ops->add_device(dev);
>>> +
>>> +		if (err)
>>> +			ops = ERR_PTR(err);
>>> +	}
>>> +
>>
>> On the last post we discussed that the above replay hunk can be made
>> common. In trying to do that, i ended up with a patch like below. But not
>> sure if that should be a part of this series though. I tested with OF devices
>> and would have to be tested with ACPI devices once. Nothing changes
>> functionally because of this ideally. Should be split in two patches though.
>>
>> Regards,
>>  Sricharan
>>
>> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Fri, 3 Feb 2017 15:24:47 +0530
>> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
>>
>> The code to call IOMMU driver's add_device is same
>> for both OF and ACPI cases. So add an api which can
>> be shared across both the places.
>>
>> Also, now with probe-deferral the iommu master devices gets
>> added to the respective iommus during probe time instead
>> of device creation time. The xlate callbacks of iommu
>> drivers are also called only at probe time. As a result
>> the add_iommu_group which gets called when the iommu is
>> registered to add all devices created before the iommu
>> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
>> also is not needed. So just cleanup those code.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/acpi/arm64/iort.c | 12 +-------
>>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>>  drivers/iommu/of_iommu.c  | 11 +-------
>>  include/linux/iommu.h     |  8 ++++++
>>  4 files changed, 27 insertions(+), 74 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index ac45623..ab2a554 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>                 }
>>         }
>>
>> -       /*
>> -        * If we have reason to believe the IOMMU driver missed the initial
>> -        * add_device callback for dev, replay it to get things in order.
>> -        */
>> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> -           dev->bus && !dev->iommu_group) {
>> -               int err = ops->add_device(dev);
>> -
>> -               if (err)
>> -                       ops = ERR_PTR(err);
>> -       }
>> +       ops = iommu_add_device(dev, ops);
>>
>>         return ops;
>>  }
>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>> index b752c3d..750552d 100644
>> --- a/drivers/iommu/iommu.c
>> +++ b/drivers/iommu/iommu.c
>> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>>         return group->default_domain;
>>  }
>>
>> -static int add_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -       int ret;
>> -
>> -       if (!ops->add_device)
>> -               return 0;
>> -
>> -       WARN_ON(dev->iommu_group);
>> -
>> -       ret = ops->add_device(dev);
>> -
>> -       /*
>> -        * We ignore -ENODEV errors for now, as they just mean that the
>> -        * device is not translated by an IOMMU. We still care about
>> -        * other errors and fail to initialize when they happen.
>> -        */
>> -       if (ret == -ENODEV)
>> -               ret = 0;
>> -
>> -       return ret;
>> -}
>> -
>> -static int remove_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -
>> -       if (ops->remove_device && dev->iommu_group)
>> -               ops->remove_device(dev);
>> -
>> -       return 0;
>> -}
>> -
>>  static int iommu_bus_notifier(struct notifier_block *nb,
>>                               unsigned long action, void *data)
>>  {
>> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>>         unsigned long group_action = 0;
>>
>>         /*
>> -        * ADD/DEL call into iommu driver ops if provided, which may
>> +        * DEL call into iommu driver ops if provided, which may
>>          * result in ADD/DEL notifiers to group->notifier
>>          */
>> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
>> -               if (ops->add_device)
>> -                       return ops->add_device(dev);
>
>I'm pretty sure this completely breaks x86 and anyone else...
>

ha, i just missed thinking about non-arm, for this super cleanup :)

>Anyway, I'd be inclined to leave this alone for now - it's essentially
>just a workaround to mesh the per-instance probing behaviour with the
>per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
>code over to a per-instance model will inevitably subsume the underlying
>problem, and I think a bit of duplication is probably the lesser of two
>evils in the meantime. On which note, I need to have a good look at
>Joerg's shiny new series :)
>

Agree, better way for this cleanup is with the per-instance model and let
the series go otherwise.

Regards,
 Sricharan

RE: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Robin,

>>
>>> -----Original Message-----
>>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Sricharan R
>>> Sent: Friday, February 03, 2017 9:19 PM
>>> To: robin.murphy@arm.com; will.deacon@arm.com; joro@8bytes.org; lorenzo.pieralisi@arm.com; iommu@lists.linux-
>foundation.org;
>>> linux-arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; m.szyprowski@samsung.com; bhelgaas@google.com;
>linux-
>>> pci@vger.kernel.org; linux-acpi@vger.kernel.org; tn@semihalf.com; hanjun.guo@linaro.org; okaya@codeaurora.org
>>> Cc: sricharan@codeaurora.org
>>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>>
>>> This is an equivalent to the DT's handling of the iommu master's probe
>>> with deferred probing when the corrsponding iommu is not probed yet.
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the firmware describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>>> drivers/acpi/scan.c        |  7 +++++--
>>> drivers/base/dma-mapping.c |  2 +-
>>> include/acpi/acpi_bus.h    |  2 +-
>>> include/linux/acpi.h       |  7 +++++--
>>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>>> index bf0ed09..d01bae8 100644
>>> --- a/drivers/acpi/arm64/iort.c
>>> +++ b/drivers/acpi/arm64/iort.c
>>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>>> 			return NULL;
>>>
>>> 		ops = iommu_get_instance(iort_fwnode);
>>> +		/*
>>> +		 * If the ops look-up fails, this means that either
>>> +		 * the SMMU drivers have not been probed yet or that
>>> +		 * the SMMU drivers are not built in the kernel;
>>> +		 * Depending on whether the SMMU drivers are built-in
>>> +		 * in the kernel or not, defer the IOMMU configuration
>>> +		 * or just abort it.
>>> +		 */
>>> 		if (!ops)
>>> -			return NULL;
>>> +			return iort_iommu_driver_enabled(node->type) ?
>>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>>
>>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>>> 	}
>>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>>
>>> 		while (parent) {
>>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>>> +			if (IS_ERR_OR_NULL(ops))
>>> +				return ops;
>>>
>>> 			parent = iort_node_get_id(node, &streamid,
>>> 						  IORT_IOMMU_TYPE, i++);
>>> 		}
>>> 	}
>>>
>>> +	/*
>>> +	 * If we have reason to believe the IOMMU driver missed the initial
>>> +	 * add_device callback for dev, replay it to get things in order.
>>> +	 */
>>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>>> +	    dev->bus && !dev->iommu_group) {
>>> +		int err = ops->add_device(dev);
>>> +
>>> +		if (err)
>>> +			ops = ERR_PTR(err);
>>> +	}
>>> +
>>
>> On the last post we discussed that the above replay hunk can be made
>> common. In trying to do that, i ended up with a patch like below. But not
>> sure if that should be a part of this series though. I tested with OF devices
>> and would have to be tested with ACPI devices once. Nothing changes
>> functionally because of this ideally. Should be split in two patches though.
>>
>> Regards,
>>  Sricharan
>>
>> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Fri, 3 Feb 2017 15:24:47 +0530
>> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
>>
>> The code to call IOMMU driver's add_device is same
>> for both OF and ACPI cases. So add an api which can
>> be shared across both the places.
>>
>> Also, now with probe-deferral the iommu master devices gets
>> added to the respective iommus during probe time instead
>> of device creation time. The xlate callbacks of iommu
>> drivers are also called only at probe time. As a result
>> the add_iommu_group which gets called when the iommu is
>> registered to add all devices created before the iommu
>> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
>> also is not needed. So just cleanup those code.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/acpi/arm64/iort.c | 12 +-------
>>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>>  drivers/iommu/of_iommu.c  | 11 +-------
>>  include/linux/iommu.h     |  8 ++++++
>>  4 files changed, 27 insertions(+), 74 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index ac45623..ab2a554 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>                 }
>>         }
>>
>> -       /*
>> -        * If we have reason to believe the IOMMU driver missed the initial
>> -        * add_device callback for dev, replay it to get things in order.
>> -        */
>> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> -           dev->bus && !dev->iommu_group) {
>> -               int err = ops->add_device(dev);
>> -
>> -               if (err)
>> -                       ops = ERR_PTR(err);
>> -       }
>> +       ops = iommu_add_device(dev, ops);
>>
>>         return ops;
>>  }
>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>> index b752c3d..750552d 100644
>> --- a/drivers/iommu/iommu.c
>> +++ b/drivers/iommu/iommu.c
>> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>>         return group->default_domain;
>>  }
>>
>> -static int add_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -       int ret;
>> -
>> -       if (!ops->add_device)
>> -               return 0;
>> -
>> -       WARN_ON(dev->iommu_group);
>> -
>> -       ret = ops->add_device(dev);
>> -
>> -       /*
>> -        * We ignore -ENODEV errors for now, as they just mean that the
>> -        * device is not translated by an IOMMU. We still care about
>> -        * other errors and fail to initialize when they happen.
>> -        */
>> -       if (ret == -ENODEV)
>> -               ret = 0;
>> -
>> -       return ret;
>> -}
>> -
>> -static int remove_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -
>> -       if (ops->remove_device && dev->iommu_group)
>> -               ops->remove_device(dev);
>> -
>> -       return 0;
>> -}
>> -
>>  static int iommu_bus_notifier(struct notifier_block *nb,
>>                               unsigned long action, void *data)
>>  {
>> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>>         unsigned long group_action = 0;
>>
>>         /*
>> -        * ADD/DEL call into iommu driver ops if provided, which may
>> +        * DEL call into iommu driver ops if provided, which may
>>          * result in ADD/DEL notifiers to group->notifier
>>          */
>> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
>> -               if (ops->add_device)
>> -                       return ops->add_device(dev);
>
>I'm pretty sure this completely breaks x86 and anyone else...
>

ha, i just missed thinking about non-arm, for this super cleanup :)

>Anyway, I'd be inclined to leave this alone for now - it's essentially
>just a workaround to mesh the per-instance probing behaviour with the
>per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
>code over to a per-instance model will inevitably subsume the underlying
>problem, and I think a bit of duplication is probably the lesser of two
>evils in the meantime. On which note, I need to have a good look at
>Joerg's shiny new series :)
>

Agree, better way for this cleanup is with the per-instance model and let
the series go otherwise.

Regards,
 Sricharan


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Robin,

>>
>>> -----Original Message-----
>>> From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Sricharan R
>>> Sent: Friday, February 03, 2017 9:19 PM
>>> To: robin.murphy at arm.com; will.deacon at arm.com; joro at 8bytes.org; lorenzo.pieralisi at arm.com; iommu at lists.linux-
>foundation.org;
>>> linux-arm-kernel at lists.infradead.org; linux-arm-msm at vger.kernel.org; m.szyprowski at samsung.com; bhelgaas at google.com;
>linux-
>>> pci at vger.kernel.org; linux-acpi at vger.kernel.org; tn at semihalf.com; hanjun.guo at linaro.org; okaya at codeaurora.org
>>> Cc: sricharan at codeaurora.org
>>> Subject: [PATCH V8 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error
>>>
>>> This is an equivalent to the DT's handling of the iommu master's probe
>>> with deferred probing when the corrsponding iommu is not probed yet.
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the firmware describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Hanjun Guo <hanjun.guo@linaro.org>
>>> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> ---
>>> drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
>>> drivers/acpi/scan.c        |  7 +++++--
>>> drivers/base/dma-mapping.c |  2 +-
>>> include/acpi/acpi_bus.h    |  2 +-
>>> include/linux/acpi.h       |  7 +++++--
>>> 5 files changed, 36 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>>> index bf0ed09..d01bae8 100644
>>> --- a/drivers/acpi/arm64/iort.c
>>> +++ b/drivers/acpi/arm64/iort.c
>>> @@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
>>> 			return NULL;
>>>
>>> 		ops = iommu_get_instance(iort_fwnode);
>>> +		/*
>>> +		 * If the ops look-up fails, this means that either
>>> +		 * the SMMU drivers have not been probed yet or that
>>> +		 * the SMMU drivers are not built in the kernel;
>>> +		 * Depending on whether the SMMU drivers are built-in
>>> +		 * in the kernel or not, defer the IOMMU configuration
>>> +		 * or just abort it.
>>> +		 */
>>> 		if (!ops)
>>> -			return NULL;
>>> +			return iort_iommu_driver_enabled(node->type) ?
>>> +			       ERR_PTR(-EPROBE_DEFER) : NULL;
>>>
>>> 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
>>> 	}
>>> @@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>>
>>> 		while (parent) {
>>> 			ops = iort_iommu_xlate(dev, parent, streamid);
>>> +			if (IS_ERR_OR_NULL(ops))
>>> +				return ops;
>>>
>>> 			parent = iort_node_get_id(node, &streamid,
>>> 						  IORT_IOMMU_TYPE, i++);
>>> 		}
>>> 	}
>>>
>>> +	/*
>>> +	 * If we have reason to believe the IOMMU driver missed the initial
>>> +	 * add_device callback for dev, replay it to get things in order.
>>> +	 */
>>> +	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>>> +	    dev->bus && !dev->iommu_group) {
>>> +		int err = ops->add_device(dev);
>>> +
>>> +		if (err)
>>> +			ops = ERR_PTR(err);
>>> +	}
>>> +
>>
>> On the last post we discussed that the above replay hunk can be made
>> common. In trying to do that, i ended up with a patch like below. But not
>> sure if that should be a part of this series though. I tested with OF devices
>> and would have to be tested with ACPI devices once. Nothing changes
>> functionally because of this ideally. Should be split in two patches though.
>>
>> Regards,
>>  Sricharan
>>
>> From aafbf2c97375a086327504f2367eaf9197c719b1 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Fri, 3 Feb 2017 15:24:47 +0530
>> Subject: [PATCH] drivers: iommu: Add iommu_add_device api
>>
>> The code to call IOMMU driver's add_device is same
>> for both OF and ACPI cases. So add an api which can
>> be shared across both the places.
>>
>> Also, now with probe-deferral the iommu master devices gets
>> added to the respective iommus during probe time instead
>> of device creation time. The xlate callbacks of iommu
>> drivers are also called only at probe time. As a result
>> the add_iommu_group which gets called when the iommu is
>> registered to add all devices created before the iommu
>> becomes dummy. Similar the BUS_NOTIFY_ADD_DEVICE notification
>> also is not needed. So just cleanup those code.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/acpi/arm64/iort.c | 12 +-------
>>  drivers/iommu/iommu.c     | 70 ++++++++++++-----------------------------------
>>  drivers/iommu/of_iommu.c  | 11 +-------
>>  include/linux/iommu.h     |  8 ++++++
>>  4 files changed, 27 insertions(+), 74 deletions(-)
>>
>> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
>> index ac45623..ab2a554 100644
>> --- a/drivers/acpi/arm64/iort.c
>> +++ b/drivers/acpi/arm64/iort.c
>> @@ -642,17 +642,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>>                 }
>>         }
>>
>> -       /*
>> -        * If we have reason to believe the IOMMU driver missed the initial
>> -        * add_device callback for dev, replay it to get things in order.
>> -        */
>> -       if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
>> -           dev->bus && !dev->iommu_group) {
>> -               int err = ops->add_device(dev);
>> -
>> -               if (err)
>> -                       ops = ERR_PTR(err);
>> -       }
>> +       ops = iommu_add_device(dev, ops);
>>
>>         return ops;
>>  }
>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>> index b752c3d..750552d 100644
>> --- a/drivers/iommu/iommu.c
>> +++ b/drivers/iommu/iommu.c
>> @@ -1015,41 +1015,6 @@ struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
>>         return group->default_domain;
>>  }
>>
>> -static int add_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -       int ret;
>> -
>> -       if (!ops->add_device)
>> -               return 0;
>> -
>> -       WARN_ON(dev->iommu_group);
>> -
>> -       ret = ops->add_device(dev);
>> -
>> -       /*
>> -        * We ignore -ENODEV errors for now, as they just mean that the
>> -        * device is not translated by an IOMMU. We still care about
>> -        * other errors and fail to initialize when they happen.
>> -        */
>> -       if (ret == -ENODEV)
>> -               ret = 0;
>> -
>> -       return ret;
>> -}
>> -
>> -static int remove_iommu_group(struct device *dev, void *data)
>> -{
>> -       struct iommu_callback_data *cb = data;
>> -       const struct iommu_ops *ops = cb->ops;
>> -
>> -       if (ops->remove_device && dev->iommu_group)
>> -               ops->remove_device(dev);
>> -
>> -       return 0;
>> -}
>> -
>>  static int iommu_bus_notifier(struct notifier_block *nb,
>>                               unsigned long action, void *data)
>>  {
>> @@ -1059,13 +1024,10 @@ static int iommu_bus_notifier(struct notifier_block *nb,
>>         unsigned long group_action = 0;
>>
>>         /*
>> -        * ADD/DEL call into iommu driver ops if provided, which may
>> +        * DEL call into iommu driver ops if provided, which may
>>          * result in ADD/DEL notifiers to group->notifier
>>          */
>> -       if (action == BUS_NOTIFY_ADD_DEVICE) {
>> -               if (ops->add_device)
>> -                       return ops->add_device(dev);
>
>I'm pretty sure this completely breaks x86 and anyone else...
>

ha, i just missed thinking about non-arm, for this super cleanup :)

>Anyway, I'd be inclined to leave this alone for now - it's essentially
>just a workaround to mesh the per-instance probing behaviour with the
>per-bus ops model, so it's bound to be a bit ugly. Moving the IOMMU core
>code over to a per-instance model will inevitably subsume the underlying
>problem, and I think a bit of duplication is probably the lesser of two
>evils in the meantime. On which note, I need to have a good look at
>Joerg's shiny new series :)
>

Agree, better way for this cleanup is with the per-instance model and let
the series go otherwise.

Regards,
 Sricharan

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Jean-Philippe Brucker]

Hello,

On 03/02/17 15:48, Sricharan R wrote:
> From: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> 
> In preparation for some upcoming cleverness, rework the control flow in
> of_iommu_configure() to minimise duplication and improve the propogation
> of errors. It's also as good a time as any to switch over from the
> now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
> IOMMU instance interface directly.
> 
> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> ---
>  [*] Resolved a conflict while rebasing on top linux-next as the patch
>      is not there in mainline master.
> 
>         "iommu: Drop the of_iommu_{set/get}_ops() interface"
>         https://lkml.org/lkml/2017/1/3/489
> 
>  drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 53 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index d7f480a..ee49081 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
>  }
>  EXPORT_SYMBOL_GPL(of_get_dma_window);
>  
> +static const struct iommu_ops
> +*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
> +{
> +	const struct iommu_ops *ops;
> +	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
> +	int err;
> +
> +	ops = iommu_get_instance(fwnode);
> +	if (!ops || !ops->of_xlate)
> +		return NULL;
> +
> +	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	err = ops->of_xlate(dev, iommu_spec);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	return ops;
> +}
> +
>  static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  {
>  	struct of_phandle_args *iommu_spec = data;
> @@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  }
>  
>  static const struct iommu_ops
> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>  {
>  	const struct iommu_ops *ops;
>  	struct of_phandle_args iommu_spec;
> +	int err;
>  
>  	/*
>  	 * Start by tracing the RID alias down the PCI topology as
> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  	 * bus into the system beyond, and which IOMMU it ends up at.
>  	 */
>  	iommu_spec.np = NULL;
> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
> -		return NULL;
> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> +			     "iommu-map-mask", &iommu_spec.np,
> +			     iommu_spec.args);
> +	if (err)
> +		return ERR_PTR(err);

This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
an IOMMU:

        map = of_get_property(np, map_name, &map_len);
        if (!map) {
                if (target)
                        return -ENODEV;
                /* Otherwise, no map implies no translation */
                *id_out = rid;
                return 0;
        }

Previously with no iommu-map, we returned -ENODEV but it was discarded by
of_pci_iommu_configure. Now it is propagated and the whole device probing
fails. Instead, maybe of_pci_map_rid should always return 0 if no
iommu-map, and the caller should check if *target is still NULL?

Thanks,
Jean-Philippe

>  
> -	ops = iommu_get_instance(&iommu_spec.np->fwnode);
> -	if (!ops || !ops->of_xlate ||
> -	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
> -	    ops->of_xlate(&pdev->dev, &iommu_spec))
> -		ops = NULL;
> +	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
>  
>  	of_node_put(iommu_spec.np);
>  	return ops;
>  }
>  
> -const struct iommu_ops *of_iommu_configure(struct device *dev,
> -					   struct device_node *master_np)
> +static const struct iommu_ops
> +*of_platform_iommu_init(struct device *dev, struct device_node *np)
>  {
>  	struct of_phandle_args iommu_spec;
> -	struct device_node *np;
>  	const struct iommu_ops *ops = NULL;
>  	int idx = 0;
>  
> -	if (dev_is_pci(dev))
> -		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
> -
>  	/*
>  	 * We don't currently walk up the tree looking for a parent IOMMU.
>  	 * See the `Notes:' section of
>  	 * Documentation/devicetree/bindings/iommu/iommu.txt
>  	 */
> -	while (!of_parse_phandle_with_args(master_np, "iommus",
> -					   "#iommu-cells", idx,
> -					   &iommu_spec)) {
> -		np = iommu_spec.np;
> -		ops = iommu_get_instance(&np->fwnode);
> -
> -		if (!ops || !ops->of_xlate ||
> -		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
> -		    ops->of_xlate(dev, &iommu_spec))
> -			goto err_put_node;
> -
> -		of_node_put(np);
> +	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
> +					   idx, &iommu_spec)) {
> +		ops = of_iommu_xlate(dev, &iommu_spec);
> +		of_node_put(iommu_spec.np);
>  		idx++;
> +		if (IS_ERR_OR_NULL(ops))
> +			break;
>  	}
>  
>  	return ops;
> +}
> +
> +const struct iommu_ops *of_iommu_configure(struct device *dev,
> +					   struct device_node *master_np)
> +{
> +	const struct iommu_ops *ops;
> +
> +	if (!master_np)
> +		return NULL;
> +
> +	if (dev_is_pci(dev))
> +		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
> +	else
> +		ops = of_platform_iommu_init(dev, master_np);
>  
> -err_put_node:
> -	of_node_put(np);
> -	return NULL;
> +	return IS_ERR(ops) ? NULL : ops;
>  }
>  
>  static int __init of_iommu_init(void)
> 

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Jean-Philippe Brucker]

Hello,

On 03/02/17 15:48, Sricharan R wrote:
> From: Robin Murphy <robin.murphy@arm.com>
> 
> In preparation for some upcoming cleverness, rework the control flow in
> of_iommu_configure() to minimise duplication and improve the propogation
> of errors. It's also as good a time as any to switch over from the
> now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
> IOMMU instance interface directly.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---
>  [*] Resolved a conflict while rebasing on top linux-next as the patch
>      is not there in mainline master.
> 
>         "iommu: Drop the of_iommu_{set/get}_ops() interface"
>         https://lkml.org/lkml/2017/1/3/489
> 
>  drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 53 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index d7f480a..ee49081 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
>  }
>  EXPORT_SYMBOL_GPL(of_get_dma_window);
>  
> +static const struct iommu_ops
> +*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
> +{
> +	const struct iommu_ops *ops;
> +	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
> +	int err;
> +
> +	ops = iommu_get_instance(fwnode);
> +	if (!ops || !ops->of_xlate)
> +		return NULL;
> +
> +	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	err = ops->of_xlate(dev, iommu_spec);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	return ops;
> +}
> +
>  static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  {
>  	struct of_phandle_args *iommu_spec = data;
> @@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  }
>  
>  static const struct iommu_ops
> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>  {
>  	const struct iommu_ops *ops;
>  	struct of_phandle_args iommu_spec;
> +	int err;
>  
>  	/*
>  	 * Start by tracing the RID alias down the PCI topology as
> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  	 * bus into the system beyond, and which IOMMU it ends up at.
>  	 */
>  	iommu_spec.np = NULL;
> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
> -		return NULL;
> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> +			     "iommu-map-mask", &iommu_spec.np,
> +			     iommu_spec.args);
> +	if (err)
> +		return ERR_PTR(err);

This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
an IOMMU:

        map = of_get_property(np, map_name, &map_len);
        if (!map) {
                if (target)
                        return -ENODEV;
                /* Otherwise, no map implies no translation */
                *id_out = rid;
                return 0;
        }

Previously with no iommu-map, we returned -ENODEV but it was discarded by
of_pci_iommu_configure. Now it is propagated and the whole device probing
fails. Instead, maybe of_pci_map_rid should always return 0 if no
iommu-map, and the caller should check if *target is still NULL?

Thanks,
Jean-Philippe

>  
> -	ops = iommu_get_instance(&iommu_spec.np->fwnode);
> -	if (!ops || !ops->of_xlate ||
> -	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
> -	    ops->of_xlate(&pdev->dev, &iommu_spec))
> -		ops = NULL;
> +	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
>  
>  	of_node_put(iommu_spec.np);
>  	return ops;
>  }
>  
> -const struct iommu_ops *of_iommu_configure(struct device *dev,
> -					   struct device_node *master_np)
> +static const struct iommu_ops
> +*of_platform_iommu_init(struct device *dev, struct device_node *np)
>  {
>  	struct of_phandle_args iommu_spec;
> -	struct device_node *np;
>  	const struct iommu_ops *ops = NULL;
>  	int idx = 0;
>  
> -	if (dev_is_pci(dev))
> -		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
> -
>  	/*
>  	 * We don't currently walk up the tree looking for a parent IOMMU.
>  	 * See the `Notes:' section of
>  	 * Documentation/devicetree/bindings/iommu/iommu.txt
>  	 */
> -	while (!of_parse_phandle_with_args(master_np, "iommus",
> -					   "#iommu-cells", idx,
> -					   &iommu_spec)) {
> -		np = iommu_spec.np;
> -		ops = iommu_get_instance(&np->fwnode);
> -
> -		if (!ops || !ops->of_xlate ||
> -		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
> -		    ops->of_xlate(dev, &iommu_spec))
> -			goto err_put_node;
> -
> -		of_node_put(np);
> +	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
> +					   idx, &iommu_spec)) {
> +		ops = of_iommu_xlate(dev, &iommu_spec);
> +		of_node_put(iommu_spec.np);
>  		idx++;
> +		if (IS_ERR_OR_NULL(ops))
> +			break;
>  	}
>  
>  	return ops;
> +}
> +
> +const struct iommu_ops *of_iommu_configure(struct device *dev,
> +					   struct device_node *master_np)
> +{
> +	const struct iommu_ops *ops;
> +
> +	if (!master_np)
> +		return NULL;
> +
> +	if (dev_is_pci(dev))
> +		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
> +	else
> +		ops = of_platform_iommu_init(dev, master_np);
>  
> -err_put_node:
> -	of_node_put(np);
> -	return NULL;
> +	return IS_ERR(ops) ? NULL : ops;
>  }
>  
>  static int __init of_iommu_init(void)
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Jean-Philippe Brucker]

Hello,

On 03/02/17 15:48, Sricharan R wrote:
> From: Robin Murphy <robin.murphy@arm.com>
> 
> In preparation for some upcoming cleverness, rework the control flow in
> of_iommu_configure() to minimise duplication and improve the propogation
> of errors. It's also as good a time as any to switch over from the
> now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic
> IOMMU instance interface directly.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---
>  [*] Resolved a conflict while rebasing on top linux-next as the patch
>      is not there in mainline master.
> 
>         "iommu: Drop the of_iommu_{set/get}_ops() interface"
>         https://lkml.org/lkml/2017/1/3/489
> 
>  drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 53 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index d7f480a..ee49081 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
>  }
>  EXPORT_SYMBOL_GPL(of_get_dma_window);
>  
> +static const struct iommu_ops
> +*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
> +{
> +	const struct iommu_ops *ops;
> +	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
> +	int err;
> +
> +	ops = iommu_get_instance(fwnode);
> +	if (!ops || !ops->of_xlate)
> +		return NULL;
> +
> +	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	err = ops->of_xlate(dev, iommu_spec);
> +	if (err)
> +		return ERR_PTR(err);
> +
> +	return ops;
> +}
> +
>  static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  {
>  	struct of_phandle_args *iommu_spec = data;
> @@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  }
>  
>  static const struct iommu_ops
> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>  {
>  	const struct iommu_ops *ops;
>  	struct of_phandle_args iommu_spec;
> +	int err;
>  
>  	/*
>  	 * Start by tracing the RID alias down the PCI topology as
> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>  	 * bus into the system beyond, and which IOMMU it ends up at.
>  	 */
>  	iommu_spec.np = NULL;
> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
> -		return NULL;
> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
> +			     "iommu-map-mask", &iommu_spec.np,
> +			     iommu_spec.args);
> +	if (err)
> +		return ERR_PTR(err);

This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
an IOMMU:

        map = of_get_property(np, map_name, &map_len);
        if (!map) {
                if (target)
                        return -ENODEV;
                /* Otherwise, no map implies no translation */
                *id_out = rid;
                return 0;
        }

Previously with no iommu-map, we returned -ENODEV but it was discarded by
of_pci_iommu_configure. Now it is propagated and the whole device probing
fails. Instead, maybe of_pci_map_rid should always return 0 if no
iommu-map, and the caller should check if *target is still NULL?

Thanks,
Jean-Philippe

>  
> -	ops = iommu_get_instance(&iommu_spec.np->fwnode);
> -	if (!ops || !ops->of_xlate ||
> -	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
> -	    ops->of_xlate(&pdev->dev, &iommu_spec))
> -		ops = NULL;
> +	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
>  
>  	of_node_put(iommu_spec.np);
>  	return ops;
>  }
>  
> -const struct iommu_ops *of_iommu_configure(struct device *dev,
> -					   struct device_node *master_np)
> +static const struct iommu_ops
> +*of_platform_iommu_init(struct device *dev, struct device_node *np)
>  {
>  	struct of_phandle_args iommu_spec;
> -	struct device_node *np;
>  	const struct iommu_ops *ops = NULL;
>  	int idx = 0;
>  
> -	if (dev_is_pci(dev))
> -		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
> -
>  	/*
>  	 * We don't currently walk up the tree looking for a parent IOMMU.
>  	 * See the `Notes:' section of
>  	 * Documentation/devicetree/bindings/iommu/iommu.txt
>  	 */
> -	while (!of_parse_phandle_with_args(master_np, "iommus",
> -					   "#iommu-cells", idx,
> -					   &iommu_spec)) {
> -		np = iommu_spec.np;
> -		ops = iommu_get_instance(&np->fwnode);
> -
> -		if (!ops || !ops->of_xlate ||
> -		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
> -		    ops->of_xlate(dev, &iommu_spec))
> -			goto err_put_node;
> -
> -		of_node_put(np);
> +	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
> +					   idx, &iommu_spec)) {
> +		ops = of_iommu_xlate(dev, &iommu_spec);
> +		of_node_put(iommu_spec.np);
>  		idx++;
> +		if (IS_ERR_OR_NULL(ops))
> +			break;
>  	}
>  
>  	return ops;
> +}
> +
> +const struct iommu_ops *of_iommu_configure(struct device *dev,
> +					   struct device_node *master_np)
> +{
> +	const struct iommu_ops *ops;
> +
> +	if (!master_np)
> +		return NULL;
> +
> +	if (dev_is_pci(dev))
> +		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
> +	else
> +		ops = of_platform_iommu_init(dev, master_np);
>  
> -err_put_node:
> -	of_node_put(np);
> -	return NULL;
> +	return IS_ERR(ops) ? NULL : ops;
>  }
>  
>  static int __init of_iommu_init(void)
> 

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 08/03/17 18:58, Jean-Philippe Brucker wrote:
[...]
>>  static const struct iommu_ops
>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>>  {
>>  	const struct iommu_ops *ops;
>>  	struct of_phandle_args iommu_spec;
>> +	int err;
>>  
>>  	/*
>>  	 * Start by tracing the RID alias down the PCI topology as
>> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>  	 */
>>  	iommu_spec.np = NULL;
>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
>> -		return NULL;
>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> +			     "iommu-map-mask", &iommu_spec.np,
>> +			     iommu_spec.args);
>> +	if (err)
>> +		return ERR_PTR(err);
> 
> This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
> an IOMMU:
> 
>         map = of_get_property(np, map_name, &map_len);
>         if (!map) {
>                 if (target)
>                         return -ENODEV;
>                 /* Otherwise, no map implies no translation */
>                 *id_out = rid;
>                 return 0;
>         }
> 
> Previously with no iommu-map, we returned -ENODEV but it was discarded by
> of_pci_iommu_configure. Now it is propagated and the whole device probing
> fails. Instead, maybe of_pci_map_rid should always return 0 if no
> iommu-map, and the caller should check if *target is still NULL?

Ah yes, Tomasz had found breakages with the "mmu-masters" binding
before, and I'd already pushed out a fixup for this one[1], but I forgot
that that discussion was all off-list (out of diplomatic concern that
the breakage might have been intentional - it wasn't, honest!)

Now that rc1 is out I should re-do that branch with v8 of this series
plus the fixups folded in, unless Sricharan beats me to it.

Thanks for the reminder,
Robin.

[1]:http://www.linux-arm.org/git?p=linux-rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354

> 
> Thanks,
> Jean-Philippe

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 08/03/17 18:58, Jean-Philippe Brucker wrote:
[...]
>>  static const struct iommu_ops
>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>>  {
>>  	const struct iommu_ops *ops;
>>  	struct of_phandle_args iommu_spec;
>> +	int err;
>>  
>>  	/*
>>  	 * Start by tracing the RID alias down the PCI topology as
>> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>  	 */
>>  	iommu_spec.np = NULL;
>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
>> -		return NULL;
>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> +			     "iommu-map-mask", &iommu_spec.np,
>> +			     iommu_spec.args);
>> +	if (err)
>> +		return ERR_PTR(err);
> 
> This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
> an IOMMU:
> 
>         map = of_get_property(np, map_name, &map_len);
>         if (!map) {
>                 if (target)
>                         return -ENODEV;
>                 /* Otherwise, no map implies no translation */
>                 *id_out = rid;
>                 return 0;
>         }
> 
> Previously with no iommu-map, we returned -ENODEV but it was discarded by
> of_pci_iommu_configure. Now it is propagated and the whole device probing
> fails. Instead, maybe of_pci_map_rid should always return 0 if no
> iommu-map, and the caller should check if *target is still NULL?

Ah yes, Tomasz had found breakages with the "mmu-masters" binding
before, and I'd already pushed out a fixup for this one[1], but I forgot
that that discussion was all off-list (out of diplomatic concern that
the breakage might have been intentional - it wasn't, honest!)

Now that rc1 is out I should re-do that branch with v8 of this series
plus the fixups folded in, unless Sricharan beats me to it.

Thanks for the reminder,
Robin.

[1]:http://www.linux-arm.org/git?p=linux-rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354

> 
> Thanks,
> Jean-Philippe


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 08/03/17 18:58, Jean-Philippe Brucker wrote:
[...]
>>  static const struct iommu_ops
>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
>>  {
>>  	const struct iommu_ops *ops;
>>  	struct of_phandle_args iommu_spec;
>> +	int err;
>>  
>>  	/*
>>  	 * Start by tracing the RID alias down the PCI topology as
>> @@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>  	 */
>>  	iommu_spec.np = NULL;
>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> -			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
>> -		return NULL;
>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>> +			     "iommu-map-mask", &iommu_spec.np,
>> +			     iommu_spec.args);
>> +	if (err)
>> +		return ERR_PTR(err);
> 
> This change doesn't work with of_pci_map_rid when the PCI RC isn't behind
> an IOMMU:
> 
>         map = of_get_property(np, map_name, &map_len);
>         if (!map) {
>                 if (target)
>                         return -ENODEV;
>                 /* Otherwise, no map implies no translation */
>                 *id_out = rid;
>                 return 0;
>         }
> 
> Previously with no iommu-map, we returned -ENODEV but it was discarded by
> of_pci_iommu_configure. Now it is propagated and the whole device probing
> fails. Instead, maybe of_pci_map_rid should always return 0 if no
> iommu-map, and the caller should check if *target is still NULL?

Ah yes, Tomasz had found breakages with the "mmu-masters" binding
before, and I'd already pushed out a fixup for this one[1], but I forgot
that that discussion was all off-list (out of diplomatic concern that
the breakage might have been intentional - it wasn't, honest!)

Now that rc1 is out I should re-do that branch with v8 of this series
plus the fixups folded in, unless Sricharan beats me to it.

Thanks for the reminder,
Robin.

[1]:http://www.linux-arm.org/git?p=linux-rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354

> 
> Thanks,
> Jean-Philippe

RE: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[sricharan]

Hi Robin,

>On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>[...]
>>>  static const struct iommu_ops
>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>> *bridge_np)
>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>> +*bridge_np)
>>>  {
>>>  	const struct iommu_ops *ops;
>>>  	struct of_phandle_args iommu_spec;
>>> +	int err;
>>>
>>>  	/*
>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>alias, void *data)
>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>  	 */
>>>  	iommu_spec.np = NULL;
>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> -			   "iommu-map-mask", &iommu_spec.np,
>iommu_spec.args))
>>> -		return NULL;
>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> +			     "iommu-map-mask", &iommu_spec.np,
>>> +			     iommu_spec.args);
>>> +	if (err)
>>> +		return ERR_PTR(err);
>>
>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>> behind an IOMMU:
>>
>>         map = of_get_property(np, map_name, &map_len);
>>         if (!map) {
>>                 if (target)
>>                         return -ENODEV;
>>                 /* Otherwise, no map implies no translation */
>>                 *id_out = rid;
>>                 return 0;
>>         }
>>
>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>> by of_pci_iommu_configure. Now it is propagated and the whole device
>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>> no iommu-map, and the caller should check if *target is still NULL?
>
>Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>before, and I'd already pushed out a fixup for this one[1], but I forgot
that
>that discussion was all off-list (out of diplomatic concern that the
breakage
>might have been intentional - it wasn't, honest!)
>
>Now that rc1 is out I should re-do that branch with v8 of this series plus
the
>fixups folded in, unless Sricharan beats me to it.
>

Right, I had this one [1] as V9 which had all your fixes that we discussed
offline as
well.  I was about to post a V9 today on top of -rc1 today as well.

[1] https://github.com/sricharanaz/iommu/tree/pd_v9

Regards,
 Sricharan


>Thanks for the reminder,
>Robin.
>
>[1]:http://www.linux-arm.org/git?p=linux-
>rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>
>>
>> Thanks,
>> Jean-Philippe
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info
>at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[sricharan]

Hi Robin,

>On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>[...]
>>>  static const struct iommu_ops
>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>> *bridge_np)
>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>> +*bridge_np)
>>>  {
>>>  	const struct iommu_ops *ops;
>>>  	struct of_phandle_args iommu_spec;
>>> +	int err;
>>>
>>>  	/*
>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>alias, void *data)
>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>  	 */
>>>  	iommu_spec.np = NULL;
>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> -			   "iommu-map-mask", &iommu_spec.np,
>iommu_spec.args))
>>> -		return NULL;
>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> +			     "iommu-map-mask", &iommu_spec.np,
>>> +			     iommu_spec.args);
>>> +	if (err)
>>> +		return ERR_PTR(err);
>>
>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>> behind an IOMMU:
>>
>>         map = of_get_property(np, map_name, &map_len);
>>         if (!map) {
>>                 if (target)
>>                         return -ENODEV;
>>                 /* Otherwise, no map implies no translation */
>>                 *id_out = rid;
>>                 return 0;
>>         }
>>
>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>> by of_pci_iommu_configure. Now it is propagated and the whole device
>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>> no iommu-map, and the caller should check if *target is still NULL?
>
>Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>before, and I'd already pushed out a fixup for this one[1], but I forgot
that
>that discussion was all off-list (out of diplomatic concern that the
breakage
>might have been intentional - it wasn't, honest!)
>
>Now that rc1 is out I should re-do that branch with v8 of this series plus
the
>fixups folded in, unless Sricharan beats me to it.
>

Right, I had this one [1] as V9 which had all your fixes that we discussed
offline as
well.  I was about to post a V9 today on top of -rc1 today as well.

[1] https://github.com/sricharanaz/iommu/tree/pd_v9

Regards,
 Sricharan


>Thanks for the reminder,
>Robin.
>
>[1]:http://www.linux-arm.org/git?p=linux-
>rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>
>>
>> Thanks,
>> Jean-Philippe
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>the body of a message to majordomo@vger.kernel.org More majordomo info
>at  http://vger.kernel.org/majordomo-info.html


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[sricharan]

Hi Robin,

>On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>[...]
>>>  static const struct iommu_ops
>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>> *bridge_np)
>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>> +*bridge_np)
>>>  {
>>>  	const struct iommu_ops *ops;
>>>  	struct of_phandle_args iommu_spec;
>>> +	int err;
>>>
>>>  	/*
>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>alias, void *data)
>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>  	 */
>>>  	iommu_spec.np = NULL;
>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> -			   "iommu-map-mask", &iommu_spec.np,
>iommu_spec.args))
>>> -		return NULL;
>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>> +			     "iommu-map-mask", &iommu_spec.np,
>>> +			     iommu_spec.args);
>>> +	if (err)
>>> +		return ERR_PTR(err);
>>
>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>> behind an IOMMU:
>>
>>         map = of_get_property(np, map_name, &map_len);
>>         if (!map) {
>>                 if (target)
>>                         return -ENODEV;
>>                 /* Otherwise, no map implies no translation */
>>                 *id_out = rid;
>>                 return 0;
>>         }
>>
>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>> by of_pci_iommu_configure. Now it is propagated and the whole device
>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>> no iommu-map, and the caller should check if *target is still NULL?
>
>Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>before, and I'd already pushed out a fixup for this one[1], but I forgot
that
>that discussion was all off-list (out of diplomatic concern that the
breakage
>might have been intentional - it wasn't, honest!)
>
>Now that rc1 is out I should re-do that branch with v8 of this series plus
the
>fixups folded in, unless Sricharan beats me to it.
>

Right, I had this one [1] as V9 which had all your fixes that we discussed
offline as
well.  I was about to post a V9 today on top of -rc1 today as well.

[1] https://github.com/sricharanaz/iommu/tree/pd_v9

Regards,
 Sricharan


>Thanks for the reminder,
>Robin.
>
>[1]:http://www.linux-arm.org/git?p=linux-
>rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>
>>
>> Thanks,
>> Jean-Philippe
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>the body of a message to majordomo at vger.kernel.org More majordomo info
>at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 09/03/17 09:52, sricharan wrote:
> Hi Robin,
> 
>> On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>> [...]
>>>>  static const struct iommu_ops
>>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>>> *bridge_np)
>>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>>> +*bridge_np)
>>>>  {
>>>>  	const struct iommu_ops *ops;
>>>>  	struct of_phandle_args iommu_spec;
>>>> +	int err;
>>>>
>>>>  	/*
>>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>> alias, void *data)
>>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>>  	 */
>>>>  	iommu_spec.np = NULL;
>>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> -			   "iommu-map-mask", &iommu_spec.np,
>> iommu_spec.args))
>>>> -		return NULL;
>>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> +			     "iommu-map-mask", &iommu_spec.np,
>>>> +			     iommu_spec.args);
>>>> +	if (err)
>>>> +		return ERR_PTR(err);
>>>
>>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>>> behind an IOMMU:
>>>
>>>         map = of_get_property(np, map_name, &map_len);
>>>         if (!map) {
>>>                 if (target)
>>>                         return -ENODEV;
>>>                 /* Otherwise, no map implies no translation */
>>>                 *id_out = rid;
>>>                 return 0;
>>>         }
>>>
>>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>>> by of_pci_iommu_configure. Now it is propagated and the whole device
>>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>>> no iommu-map, and the caller should check if *target is still NULL?
>>
>> Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>> before, and I'd already pushed out a fixup for this one[1], but I forgot
> that
>> that discussion was all off-list (out of diplomatic concern that the
> breakage
>> might have been intentional - it wasn't, honest!)
>>
>> Now that rc1 is out I should re-do that branch with v8 of this series plus
> the
>> fixups folded in, unless Sricharan beats me to it.
>>
> 
> Right, I had this one [1] as V9 which had all your fixes that we discussed
> offline as
> well.  I was about to post a V9 today on top of -rc1 today as well.
> 
> [1] https://github.com/sricharanaz/iommu/tree/pd_v9

Awesome, I'll try to get on with testing that ASAP. Thanks!

Robin.

> 
> Regards,
>  Sricharan
> 
> 
>> Thanks for the reminder,
>> Robin.
>>
>> [1]:http://www.linux-arm.org/git?p=linux-
>> rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>>
>>>
>>> Thanks,
>>> Jean-Philippe
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info
>> at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 09/03/17 09:52, sricharan wrote:
> Hi Robin,
> 
>> On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>> [...]
>>>>  static const struct iommu_ops
>>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>>> *bridge_np)
>>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>>> +*bridge_np)
>>>>  {
>>>>  	const struct iommu_ops *ops;
>>>>  	struct of_phandle_args iommu_spec;
>>>> +	int err;
>>>>
>>>>  	/*
>>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>> alias, void *data)
>>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>>  	 */
>>>>  	iommu_spec.np = NULL;
>>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> -			   "iommu-map-mask", &iommu_spec.np,
>> iommu_spec.args))
>>>> -		return NULL;
>>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> +			     "iommu-map-mask", &iommu_spec.np,
>>>> +			     iommu_spec.args);
>>>> +	if (err)
>>>> +		return ERR_PTR(err);
>>>
>>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>>> behind an IOMMU:
>>>
>>>         map = of_get_property(np, map_name, &map_len);
>>>         if (!map) {
>>>                 if (target)
>>>                         return -ENODEV;
>>>                 /* Otherwise, no map implies no translation */
>>>                 *id_out = rid;
>>>                 return 0;
>>>         }
>>>
>>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>>> by of_pci_iommu_configure. Now it is propagated and the whole device
>>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>>> no iommu-map, and the caller should check if *target is still NULL?
>>
>> Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>> before, and I'd already pushed out a fixup for this one[1], but I forgot
> that
>> that discussion was all off-list (out of diplomatic concern that the
> breakage
>> might have been intentional - it wasn't, honest!)
>>
>> Now that rc1 is out I should re-do that branch with v8 of this series plus
> the
>> fixups folded in, unless Sricharan beats me to it.
>>
> 
> Right, I had this one [1] as V9 which had all your fixes that we discussed
> offline as
> well.  I was about to post a V9 today on top of -rc1 today as well.
> 
> [1] https://github.com/sricharanaz/iommu/tree/pd_v9

Awesome, I'll try to get on with testing that ASAP. Thanks!

Robin.

> 
> Regards,
>  Sricharan
> 
> 
>> Thanks for the reminder,
>> Robin.
>>
>> [1]:http://www.linux-arm.org/git?p=linux-
>> rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>>
>>>
>>> Thanks,
>>> Jean-Philippe
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>> the body of a message to majordomo@vger.kernel.org More majordomo info
>> at  http://vger.kernel.org/majordomo-info.html
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 01/11] iommu/of: Refactor of_iommu_configure() for error handling

[Robin Murphy]

On 09/03/17 09:52, sricharan wrote:
> Hi Robin,
> 
>> On 08/03/17 18:58, Jean-Philippe Brucker wrote:
>> [...]
>>>>  static const struct iommu_ops
>>>> -*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node
>>>> *bridge_np)
>>>> +*of_pci_iommu_init(struct pci_dev *pdev, struct device_node
>>>> +*bridge_np)
>>>>  {
>>>>  	const struct iommu_ops *ops;
>>>>  	struct of_phandle_args iommu_spec;
>>>> +	int err;
>>>>
>>>>  	/*
>>>>  	 * Start by tracing the RID alias down the PCI topology as @@
>>>> -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16
>> alias, void *data)
>>>>  	 * bus into the system beyond, and which IOMMU it ends up at.
>>>>  	 */
>>>>  	iommu_spec.np = NULL;
>>>> -	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> -			   "iommu-map-mask", &iommu_spec.np,
>> iommu_spec.args))
>>>> -		return NULL;
>>>> +	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
>>>> +			     "iommu-map-mask", &iommu_spec.np,
>>>> +			     iommu_spec.args);
>>>> +	if (err)
>>>> +		return ERR_PTR(err);
>>>
>>> This change doesn't work with of_pci_map_rid when the PCI RC isn't
>>> behind an IOMMU:
>>>
>>>         map = of_get_property(np, map_name, &map_len);
>>>         if (!map) {
>>>                 if (target)
>>>                         return -ENODEV;
>>>                 /* Otherwise, no map implies no translation */
>>>                 *id_out = rid;
>>>                 return 0;
>>>         }
>>>
>>> Previously with no iommu-map, we returned -ENODEV but it was discarded
>>> by of_pci_iommu_configure. Now it is propagated and the whole device
>>> probing fails. Instead, maybe of_pci_map_rid should always return 0 if
>>> no iommu-map, and the caller should check if *target is still NULL?
>>
>> Ah yes, Tomasz had found breakages with the "mmu-masters" binding
>> before, and I'd already pushed out a fixup for this one[1], but I forgot
> that
>> that discussion was all off-list (out of diplomatic concern that the
> breakage
>> might have been intentional - it wasn't, honest!)
>>
>> Now that rc1 is out I should re-do that branch with v8 of this series plus
> the
>> fixups folded in, unless Sricharan beats me to it.
>>
> 
> Right, I had this one [1] as V9 which had all your fixes that we discussed
> offline as
> well.  I was about to post a V9 today on top of -rc1 today as well.
> 
> [1] https://github.com/sricharanaz/iommu/tree/pd_v9

Awesome, I'll try to get on with testing that ASAP. Thanks!

Robin.

> 
> Regards,
>  Sricharan
> 
> 
>> Thanks for the reminder,
>> Robin.
>>
>> [1]:http://www.linux-arm.org/git?p=linux-
>> rm.git;a=commitdiff;h=0049a34e523506813995c05766f5e2c16d616354
>>
>>>
>>> Thanks,
>>> Jean-Philippe
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
>> the body of a message to majordomo at vger.kernel.org More majordomo info
>> at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan,

On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
>
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
>
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
>
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
>
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>

This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
As the IOMMU nodes in DT are not yet enabled, all devices having iommus
properties in DT now fail to probe.

This can be fixed by either:
  - Disabling CONFIG_IPMMU_VMSA, or
  - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").

Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
upstreamed yet, so bisection pointed to a merge commit.

> ---
>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>      in of_iommu.c
>
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 449b948..82bd45c 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>  {
>         struct device *bridge = NULL, *dma_dev = dev;
>         enum dev_dma_attr attr;
> +       int ret = 0;
>
>         if (dev_is_pci(dev)) {
>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>         }
>
>         if (dma_dev->of_node) {
> -               of_dma_configure(dev, dma_dev->of_node);
> +               ret = of_dma_configure(dev, dma_dev->of_node);
>         } else if (has_acpi_companion(dma_dev)) {
>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>                 if (attr != DEV_DMA_NOT_SUPPORTED)
> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>         if (bridge)
>                 pci_put_host_bridge_device(bridge);
>
> -       return 0;
> +       return ret;
>  }
>
>  void dma_deconfigure(struct device *dev)
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 1f92d98..2d04663 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                         ops = ERR_PTR(err);
>         }
>
> -       return IS_ERR(ops) ? NULL : ops;
> +       return ops;
>  }
>
>  static int __init of_iommu_init(void)
> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>         for_each_matching_node_and_match(np, matches, &match) {
>                 const of_iommu_init_fn init_fn = match->data;
>
> -               if (init_fn(np))
> +               if (init_fn && init_fn(np))
>                         pr_err("Failed to initialise IOMMU %s\n",
>                                 of_node_full_name(np));
>         }
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..ba51ca6 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>         u64 dma_addr, paddr, size;
>         int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>                 coherent ? " " : " not ");
>
>         iommu = of_iommu_configure(dev, np);
> +       if (IS_ERR(iommu))
> +               return PTR_ERR(iommu);
> +
>         dev_dbg(dev, "device is%sbehind an iommu\n",
>                 iommu ? " " : " not ");
>
>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +       return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index 3cb2288..9499861 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>         return of_node_get(cpu_dev->of_node);
>  }
>
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>
> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  {
>         return NULL;
>  }
> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +
> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
> +{
> +       return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan,

On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
>
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
>
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
>
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
>
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>

This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
As the IOMMU nodes in DT are not yet enabled, all devices having iommus
properties in DT now fail to probe.

This can be fixed by either:
  - Disabling CONFIG_IPMMU_VMSA, or
  - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").

Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
upstreamed yet, so bisection pointed to a merge commit.

> ---
>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>      in of_iommu.c
>
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 449b948..82bd45c 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>  {
>         struct device *bridge = NULL, *dma_dev = dev;
>         enum dev_dma_attr attr;
> +       int ret = 0;
>
>         if (dev_is_pci(dev)) {
>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>         }
>
>         if (dma_dev->of_node) {
> -               of_dma_configure(dev, dma_dev->of_node);
> +               ret = of_dma_configure(dev, dma_dev->of_node);
>         } else if (has_acpi_companion(dma_dev)) {
>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>                 if (attr != DEV_DMA_NOT_SUPPORTED)
> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>         if (bridge)
>                 pci_put_host_bridge_device(bridge);
>
> -       return 0;
> +       return ret;
>  }
>
>  void dma_deconfigure(struct device *dev)
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 1f92d98..2d04663 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                         ops = ERR_PTR(err);
>         }
>
> -       return IS_ERR(ops) ? NULL : ops;
> +       return ops;
>  }
>
>  static int __init of_iommu_init(void)
> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>         for_each_matching_node_and_match(np, matches, &match) {
>                 const of_iommu_init_fn init_fn = match->data;
>
> -               if (init_fn(np))
> +               if (init_fn && init_fn(np))
>                         pr_err("Failed to initialise IOMMU %s\n",
>                                 of_node_full_name(np));
>         }
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..ba51ca6 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>         u64 dma_addr, paddr, size;
>         int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>                 coherent ? " " : " not ");
>
>         iommu = of_iommu_configure(dev, np);
> +       if (IS_ERR(iommu))
> +               return PTR_ERR(iommu);
> +
>         dev_dbg(dev, "device is%sbehind an iommu\n",
>                 iommu ? " " : " not ");
>
>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +       return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index 3cb2288..9499861 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>         return of_node_get(cpu_dev->of_node);
>  }
>
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>
> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  {
>         return NULL;
>  }
> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +
> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
> +{
> +       return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan,

On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
>
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
>
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
>
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
>
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>

This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
As the IOMMU nodes in DT are not yet enabled, all devices having iommus
properties in DT now fail to probe.

This can be fixed by either:
  - Disabling CONFIG_IPMMU_VMSA, or
  - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").

Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
upstreamed yet, so bisection pointed to a merge commit.

> ---
>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>      in of_iommu.c
>
>  drivers/base/dma-mapping.c | 5 +++--
>  drivers/iommu/of_iommu.c   | 4 ++--
>  drivers/of/device.c        | 7 ++++++-
>  include/linux/of_device.h  | 9 ++++++---
>  4 files changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index 449b948..82bd45c 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>  {
>         struct device *bridge = NULL, *dma_dev = dev;
>         enum dev_dma_attr attr;
> +       int ret = 0;
>
>         if (dev_is_pci(dev)) {
>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>         }
>
>         if (dma_dev->of_node) {
> -               of_dma_configure(dev, dma_dev->of_node);
> +               ret = of_dma_configure(dev, dma_dev->of_node);
>         } else if (has_acpi_companion(dma_dev)) {
>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>                 if (attr != DEV_DMA_NOT_SUPPORTED)
> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>         if (bridge)
>                 pci_put_host_bridge_device(bridge);
>
> -       return 0;
> +       return ret;
>  }
>
>  void dma_deconfigure(struct device *dev)
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 1f92d98..2d04663 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>                         ops = ERR_PTR(err);
>         }
>
> -       return IS_ERR(ops) ? NULL : ops;
> +       return ops;
>  }
>
>  static int __init of_iommu_init(void)
> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>         for_each_matching_node_and_match(np, matches, &match) {
>                 const of_iommu_init_fn init_fn = match->data;
>
> -               if (init_fn(np))
> +               if (init_fn && init_fn(np))
>                         pr_err("Failed to initialise IOMMU %s\n",
>                                 of_node_full_name(np));
>         }
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index c17c19d..ba51ca6 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>         u64 dma_addr, paddr, size;
>         int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>                 coherent ? " " : " not ");
>
>         iommu = of_iommu_configure(dev, np);
> +       if (IS_ERR(iommu))
> +               return PTR_ERR(iommu);
> +
>         dev_dbg(dev, "device is%sbehind an iommu\n",
>                 iommu ? " " : " not ");
>
>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +       return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index 3cb2288..9499861 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>         return of_node_get(cpu_dev->of_node);
>  }
>
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>
> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  {
>         return NULL;
>  }
> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +
> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
> +{
> +       return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Geert,

On 02/05/17 19:35, Geert Uytterhoeven wrote:
> Hi Sricharan,
> 
> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> 
> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> properties in DT now fail to probe.

How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
registered then they should merely defer until we reach the point of
giving up and ignoring the IOMMU. Is it just that you have no other
late-probing drivers or post-init module loads to kick the deferred
queue after that point? I did try to find a way to explicitly kick it
from a suitably late initcall, but there didn't seem to be any obvious
public interface - anyone have any suggestions?

I think that's more of a general problem with the probe deferral
mechanism itself (I've seen the same thing happen with some of the
CoreSight stuff on Juno due to the number of inter-component
dependencies) rather than any specific fault of this series.

Robin.

> This can be fixed by either:
>   - Disabling CONFIG_IPMMU_VMSA, or
>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> 
> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
> upstreamed yet, so bisection pointed to a merge commit.
> 
>> ---
>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>      in of_iommu.c
>>
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index 449b948..82bd45c 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>  {
>>         struct device *bridge = NULL, *dma_dev = dev;
>>         enum dev_dma_attr attr;
>> +       int ret = 0;
>>
>>         if (dev_is_pci(dev)) {
>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>         }
>>
>>         if (dma_dev->of_node) {
>> -               of_dma_configure(dev, dma_dev->of_node);
>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>         } else if (has_acpi_companion(dma_dev)) {
>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>         if (bridge)
>>                 pci_put_host_bridge_device(bridge);
>>
>> -       return 0;
>> +       return ret;
>>  }
>>
>>  void dma_deconfigure(struct device *dev)
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 1f92d98..2d04663 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>                         ops = ERR_PTR(err);
>>         }
>>
>> -       return IS_ERR(ops) ? NULL : ops;
>> +       return ops;
>>  }
>>
>>  static int __init of_iommu_init(void)
>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>         for_each_matching_node_and_match(np, matches, &match) {
>>                 const of_iommu_init_fn init_fn = match->data;
>>
>> -               if (init_fn(np))
>> +               if (init_fn && init_fn(np))
>>                         pr_err("Failed to initialise IOMMU %s\n",
>>                                 of_node_full_name(np));
>>         }
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..ba51ca6 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>   * to fix up DMA configuration.
>>   */
>> -void of_dma_configure(struct device *dev, struct device_node *np)
>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>  {
>>         u64 dma_addr, paddr, size;
>>         int ret;
>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>                 coherent ? " " : " not ");
>>
>>         iommu = of_iommu_configure(dev, np);
>> +       if (IS_ERR(iommu))
>> +               return PTR_ERR(iommu);
>> +
>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>                 iommu ? " " : " not ");
>>
>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>> +
>> +       return 0;
>>  }
>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>
>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>> index 3cb2288..9499861 100644
>> --- a/include/linux/of_device.h
>> +++ b/include/linux/of_device.h
>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>         return of_node_get(cpu_dev->of_node);
>>  }
>>
>> -void of_dma_configure(struct device *dev, struct device_node *np);
>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>  void of_dma_deconfigure(struct device *dev);
>>  #else /* CONFIG_OF */
>>
>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>  {
>>         return NULL;
>>  }
>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>> -{}
>> +
>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>> +{
>> +       return 0;
>> +}
>>  static inline void of_dma_deconfigure(struct device *dev)
>>  {}
>>  #endif /* CONFIG_OF */
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
> 

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Geert,

On 02/05/17 19:35, Geert Uytterhoeven wrote:
> Hi Sricharan,
> 
> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> 
> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> properties in DT now fail to probe.

How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
registered then they should merely defer until we reach the point of
giving up and ignoring the IOMMU. Is it just that you have no other
late-probing drivers or post-init module loads to kick the deferred
queue after that point? I did try to find a way to explicitly kick it
from a suitably late initcall, but there didn't seem to be any obvious
public interface - anyone have any suggestions?

I think that's more of a general problem with the probe deferral
mechanism itself (I've seen the same thing happen with some of the
CoreSight stuff on Juno due to the number of inter-component
dependencies) rather than any specific fault of this series.

Robin.

> This can be fixed by either:
>   - Disabling CONFIG_IPMMU_VMSA, or
>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> 
> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
> upstreamed yet, so bisection pointed to a merge commit.
> 
>> ---
>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>      in of_iommu.c
>>
>>  drivers/base/dma-mapping.c | 5 +++--
>>  drivers/iommu/of_iommu.c   | 4 ++--
>>  drivers/of/device.c        | 7 ++++++-
>>  include/linux/of_device.h  | 9 ++++++---
>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index 449b948..82bd45c 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>  {
>>         struct device *bridge = NULL, *dma_dev = dev;
>>         enum dev_dma_attr attr;
>> +       int ret = 0;
>>
>>         if (dev_is_pci(dev)) {
>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>         }
>>
>>         if (dma_dev->of_node) {
>> -               of_dma_configure(dev, dma_dev->of_node);
>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>         } else if (has_acpi_companion(dma_dev)) {
>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>         if (bridge)
>>                 pci_put_host_bridge_device(bridge);
>>
>> -       return 0;
>> +       return ret;
>>  }
>>
>>  void dma_deconfigure(struct device *dev)
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 1f92d98..2d04663 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>                         ops = ERR_PTR(err);
>>         }
>>
>> -       return IS_ERR(ops) ? NULL : ops;
>> +       return ops;
>>  }
>>
>>  static int __init of_iommu_init(void)
>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>         for_each_matching_node_and_match(np, matches, &match) {
>>                 const of_iommu_init_fn init_fn = match->data;
>>
>> -               if (init_fn(np))
>> +               if (init_fn && init_fn(np))
>>                         pr_err("Failed to initialise IOMMU %s\n",
>>                                 of_node_full_name(np));
>>         }
>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>> index c17c19d..ba51ca6 100644
>> --- a/drivers/of/device.c
>> +++ b/drivers/of/device.c
>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>   * to fix up DMA configuration.
>>   */
>> -void of_dma_configure(struct device *dev, struct device_node *np)
>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>  {
>>         u64 dma_addr, paddr, size;
>>         int ret;
>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>                 coherent ? " " : " not ");
>>
>>         iommu = of_iommu_configure(dev, np);
>> +       if (IS_ERR(iommu))
>> +               return PTR_ERR(iommu);
>> +
>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>                 iommu ? " " : " not ");
>>
>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>> +
>> +       return 0;
>>  }
>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>
>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>> index 3cb2288..9499861 100644
>> --- a/include/linux/of_device.h
>> +++ b/include/linux/of_device.h
>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>         return of_node_get(cpu_dev->of_node);
>>  }
>>
>> -void of_dma_configure(struct device *dev, struct device_node *np);
>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>  void of_dma_deconfigure(struct device *dev);
>>  #else /* CONFIG_OF */
>>
>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>  {
>>         return NULL;
>>  }
>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>> -{}
>> +
>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>> +{
>> +       return 0;
>> +}
>>  static inline void of_dma_deconfigure(struct device *dev)
>>  {}
>>  #endif /* CONFIG_OF */
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
> 

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Robin,

On 5/3/2017 3:24 PM, Robin Murphy wrote:
> Hi Geert,
> 
> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> Hi Sricharan,
>>
>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
>>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>
>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>> be handled separately from the .of_xlate() failures to support deferred
>>> probing.
>>>
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the device tree describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>
>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> properties in DT now fail to probe.
> 
> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> registered then they should merely defer until we reach the point of
> giving up and ignoring the IOMMU. Is it just that you have no other
> late-probing drivers or post-init module loads to kick the deferred
> queue after that point? I did try to find a way to explicitly kick it
> from a suitably late initcall, but there didn't seem to be any obvious
> public interface - anyone have any suggestions?
> 
> I think that's more of a general problem with the probe deferral
> mechanism itself (I've seen the same thing happen with some of the
> CoreSight stuff on Juno due to the number of inter-component
> dependencies) rather than any specific fault of this series.
> 

I was thinking of an additional check like below to avoid the
situation ?

>From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
Date: Wed, 3 May 2017 13:16:59 +0530
Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER

While returning EPROBE_DEFER for iommu masters
take in to account of iommu nodes that could be
marked in DT as 'status=disabled', in which case
simply return NULL and let the master's probe
continue rather than deferring.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 9f44ee8..e6e9bec 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)

        ops = iommu_ops_from_fwnode(fwnode);
        if ((ops && !ops->of_xlate) ||
+           !of_device_is_available(iommu_spec->np) ||
            (!ops && !of_iommu_driver_present(iommu_spec->np)))
                return NULL;

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


Regards,
 Sricharan


> Robin.
> 
>> This can be fixed by either:
>>   - Disabling CONFIG_IPMMU_VMSA, or
>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>
>> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
>> upstreamed yet, so bisection pointed to a merge commit.
>>
>>> ---
>>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>>      in of_iommu.c
>>>
>>>  drivers/base/dma-mapping.c | 5 +++--
>>>  drivers/iommu/of_iommu.c   | 4 ++--
>>>  drivers/of/device.c        | 7 ++++++-
>>>  include/linux/of_device.h  | 9 ++++++---
>>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>>> index 449b948..82bd45c 100644
>>> --- a/drivers/base/dma-mapping.c
>>> +++ b/drivers/base/dma-mapping.c
>>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>>  {
>>>         struct device *bridge = NULL, *dma_dev = dev;
>>>         enum dev_dma_attr attr;
>>> +       int ret = 0;
>>>
>>>         if (dev_is_pci(dev)) {
>>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>>         }
>>>
>>>         if (dma_dev->of_node) {
>>> -               of_dma_configure(dev, dma_dev->of_node);
>>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>>         } else if (has_acpi_companion(dma_dev)) {
>>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>>         if (bridge)
>>>                 pci_put_host_bridge_device(bridge);
>>>
>>> -       return 0;
>>> +       return ret;
>>>  }
>>>
>>>  void dma_deconfigure(struct device *dev)
>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>> index 1f92d98..2d04663 100644
>>> --- a/drivers/iommu/of_iommu.c
>>> +++ b/drivers/iommu/of_iommu.c
>>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>>                         ops = ERR_PTR(err);
>>>         }
>>>
>>> -       return IS_ERR(ops) ? NULL : ops;
>>> +       return ops;
>>>  }
>>>
>>>  static int __init of_iommu_init(void)
>>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>>         for_each_matching_node_and_match(np, matches, &match) {
>>>                 const of_iommu_init_fn init_fn = match->data;
>>>
>>> -               if (init_fn(np))
>>> +               if (init_fn && init_fn(np))
>>>                         pr_err("Failed to initialise IOMMU %s\n",
>>>                                 of_node_full_name(np));
>>>         }
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..ba51ca6 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>>   * to fix up DMA configuration.
>>>   */
>>> -void of_dma_configure(struct device *dev, struct device_node *np)
>>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>>  {
>>>         u64 dma_addr, paddr, size;
>>>         int ret;
>>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>                 coherent ? " " : " not ");
>>>
>>>         iommu = of_iommu_configure(dev, np);
>>> +       if (IS_ERR(iommu))
>>> +               return PTR_ERR(iommu);
>>> +
>>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>>                 iommu ? " " : " not ");
>>>
>>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>>> +
>>> +       return 0;
>>>  }
>>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>>
>>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>>> index 3cb2288..9499861 100644
>>> --- a/include/linux/of_device.h
>>> +++ b/include/linux/of_device.h
>>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>         return of_node_get(cpu_dev->of_node);
>>>  }
>>>
>>> -void of_dma_configure(struct device *dev, struct device_node *np);
>>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>>  void of_dma_deconfigure(struct device *dev);
>>>  #else /* CONFIG_OF */
>>>
>>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>  {
>>>         return NULL;
>>>  }
>>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>>> -{}
>>> +
>>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>>> +{
>>> +       return 0;
>>> +}
>>>  static inline void of_dma_deconfigure(struct device *dev)
>>>  {}
>>>  #endif /* CONFIG_OF */
>>
>> Gr{oetje,eeting}s,
>>
>>                         Geert
>>
>> --
>> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org
>>
>> In personal conversations with technical people, I call myself a hacker. But
>> when I'm talking to journalists I just say "programmer" or something like that.
>>                                 -- Linus Torvalds
>>
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Robin,

On 5/3/2017 3:24 PM, Robin Murphy wrote:
> Hi Geert,
> 
> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> Hi Sricharan,
>>
>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>
>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>> be handled separately from the .of_xlate() failures to support deferred
>>> probing.
>>>
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the device tree describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>
>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> properties in DT now fail to probe.
> 
> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> registered then they should merely defer until we reach the point of
> giving up and ignoring the IOMMU. Is it just that you have no other
> late-probing drivers or post-init module loads to kick the deferred
> queue after that point? I did try to find a way to explicitly kick it
> from a suitably late initcall, but there didn't seem to be any obvious
> public interface - anyone have any suggestions?
> 
> I think that's more of a general problem with the probe deferral
> mechanism itself (I've seen the same thing happen with some of the
> CoreSight stuff on Juno due to the number of inter-component
> dependencies) rather than any specific fault of this series.
> 

I was thinking of an additional check like below to avoid the
situation ?

>From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Wed, 3 May 2017 13:16:59 +0530
Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER

While returning EPROBE_DEFER for iommu masters
take in to account of iommu nodes that could be
marked in DT as 'status=disabled', in which case
simply return NULL and let the master's probe
continue rather than deferring.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 9f44ee8..e6e9bec 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)

        ops = iommu_ops_from_fwnode(fwnode);
        if ((ops && !ops->of_xlate) ||
+           !of_device_is_available(iommu_spec->np) ||
            (!ops && !of_iommu_driver_present(iommu_spec->np)))
                return NULL;

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


Regards,
 Sricharan


> Robin.
> 
>> This can be fixed by either:
>>   - Disabling CONFIG_IPMMU_VMSA, or
>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>
>> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
>> upstreamed yet, so bisection pointed to a merge commit.
>>
>>> ---
>>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>>      in of_iommu.c
>>>
>>>  drivers/base/dma-mapping.c | 5 +++--
>>>  drivers/iommu/of_iommu.c   | 4 ++--
>>>  drivers/of/device.c        | 7 ++++++-
>>>  include/linux/of_device.h  | 9 ++++++---
>>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>>> index 449b948..82bd45c 100644
>>> --- a/drivers/base/dma-mapping.c
>>> +++ b/drivers/base/dma-mapping.c
>>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>>  {
>>>         struct device *bridge = NULL, *dma_dev = dev;
>>>         enum dev_dma_attr attr;
>>> +       int ret = 0;
>>>
>>>         if (dev_is_pci(dev)) {
>>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>>         }
>>>
>>>         if (dma_dev->of_node) {
>>> -               of_dma_configure(dev, dma_dev->of_node);
>>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>>         } else if (has_acpi_companion(dma_dev)) {
>>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>>         if (bridge)
>>>                 pci_put_host_bridge_device(bridge);
>>>
>>> -       return 0;
>>> +       return ret;
>>>  }
>>>
>>>  void dma_deconfigure(struct device *dev)
>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>> index 1f92d98..2d04663 100644
>>> --- a/drivers/iommu/of_iommu.c
>>> +++ b/drivers/iommu/of_iommu.c
>>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>>                         ops = ERR_PTR(err);
>>>         }
>>>
>>> -       return IS_ERR(ops) ? NULL : ops;
>>> +       return ops;
>>>  }
>>>
>>>  static int __init of_iommu_init(void)
>>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>>         for_each_matching_node_and_match(np, matches, &match) {
>>>                 const of_iommu_init_fn init_fn = match->data;
>>>
>>> -               if (init_fn(np))
>>> +               if (init_fn && init_fn(np))
>>>                         pr_err("Failed to initialise IOMMU %s\n",
>>>                                 of_node_full_name(np));
>>>         }
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..ba51ca6 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>>   * to fix up DMA configuration.
>>>   */
>>> -void of_dma_configure(struct device *dev, struct device_node *np)
>>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>>  {
>>>         u64 dma_addr, paddr, size;
>>>         int ret;
>>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>                 coherent ? " " : " not ");
>>>
>>>         iommu = of_iommu_configure(dev, np);
>>> +       if (IS_ERR(iommu))
>>> +               return PTR_ERR(iommu);
>>> +
>>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>>                 iommu ? " " : " not ");
>>>
>>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>>> +
>>> +       return 0;
>>>  }
>>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>>
>>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>>> index 3cb2288..9499861 100644
>>> --- a/include/linux/of_device.h
>>> +++ b/include/linux/of_device.h
>>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>         return of_node_get(cpu_dev->of_node);
>>>  }
>>>
>>> -void of_dma_configure(struct device *dev, struct device_node *np);
>>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>>  void of_dma_deconfigure(struct device *dev);
>>>  #else /* CONFIG_OF */
>>>
>>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>  {
>>>         return NULL;
>>>  }
>>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>>> -{}
>>> +
>>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>>> +{
>>> +       return 0;
>>> +}
>>>  static inline void of_dma_deconfigure(struct device *dev)
>>>  {}
>>>  #endif /* CONFIG_OF */
>>
>> Gr{oetje,eeting}s,
>>
>>                         Geert
>>
>> --
>> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>>
>> In personal conversations with technical people, I call myself a hacker. But
>> when I'm talking to journalists I just say "programmer" or something like that.
>>                                 -- Linus Torvalds
>>
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Robin,

On 5/3/2017 3:24 PM, Robin Murphy wrote:
> Hi Geert,
> 
> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> Hi Sricharan,
>>
>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>
>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>> be handled separately from the .of_xlate() failures to support deferred
>>> probing.
>>>
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the device tree describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>
>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> properties in DT now fail to probe.
> 
> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> registered then they should merely defer until we reach the point of
> giving up and ignoring the IOMMU. Is it just that you have no other
> late-probing drivers or post-init module loads to kick the deferred
> queue after that point? I did try to find a way to explicitly kick it
> from a suitably late initcall, but there didn't seem to be any obvious
> public interface - anyone have any suggestions?
> 
> I think that's more of a general problem with the probe deferral
> mechanism itself (I've seen the same thing happen with some of the
> CoreSight stuff on Juno due to the number of inter-component
> dependencies) rather than any specific fault of this series.
> 

I was thinking of an additional check like below to avoid the
situation ?

>From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Wed, 3 May 2017 13:16:59 +0530
Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER

While returning EPROBE_DEFER for iommu masters
take in to account of iommu nodes that could be
marked in DT as 'status=disabled', in which case
simply return NULL and let the master's probe
continue rather than deferring.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 9f44ee8..e6e9bec 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)

        ops = iommu_ops_from_fwnode(fwnode);
        if ((ops && !ops->of_xlate) ||
+           !of_device_is_available(iommu_spec->np) ||
            (!ops && !of_iommu_driver_present(iommu_spec->np)))
                return NULL;

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


Regards,
 Sricharan


> Robin.
> 
>> This can be fixed by either:
>>   - Disabling CONFIG_IPMMU_VMSA, or
>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>
>> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
>> upstreamed yet, so bisection pointed to a merge commit.
>>
>>> ---
>>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>>      in of_iommu.c
>>>
>>>  drivers/base/dma-mapping.c | 5 +++--
>>>  drivers/iommu/of_iommu.c   | 4 ++--
>>>  drivers/of/device.c        | 7 ++++++-
>>>  include/linux/of_device.h  | 9 ++++++---
>>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>>> index 449b948..82bd45c 100644
>>> --- a/drivers/base/dma-mapping.c
>>> +++ b/drivers/base/dma-mapping.c
>>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>>  {
>>>         struct device *bridge = NULL, *dma_dev = dev;
>>>         enum dev_dma_attr attr;
>>> +       int ret = 0;
>>>
>>>         if (dev_is_pci(dev)) {
>>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>>         }
>>>
>>>         if (dma_dev->of_node) {
>>> -               of_dma_configure(dev, dma_dev->of_node);
>>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>>         } else if (has_acpi_companion(dma_dev)) {
>>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>>         if (bridge)
>>>                 pci_put_host_bridge_device(bridge);
>>>
>>> -       return 0;
>>> +       return ret;
>>>  }
>>>
>>>  void dma_deconfigure(struct device *dev)
>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>> index 1f92d98..2d04663 100644
>>> --- a/drivers/iommu/of_iommu.c
>>> +++ b/drivers/iommu/of_iommu.c
>>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>>                         ops = ERR_PTR(err);
>>>         }
>>>
>>> -       return IS_ERR(ops) ? NULL : ops;
>>> +       return ops;
>>>  }
>>>
>>>  static int __init of_iommu_init(void)
>>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>>         for_each_matching_node_and_match(np, matches, &match) {
>>>                 const of_iommu_init_fn init_fn = match->data;
>>>
>>> -               if (init_fn(np))
>>> +               if (init_fn && init_fn(np))
>>>                         pr_err("Failed to initialise IOMMU %s\n",
>>>                                 of_node_full_name(np));
>>>         }
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..ba51ca6 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>>   * to fix up DMA configuration.
>>>   */
>>> -void of_dma_configure(struct device *dev, struct device_node *np)
>>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>>  {
>>>         u64 dma_addr, paddr, size;
>>>         int ret;
>>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>                 coherent ? " " : " not ");
>>>
>>>         iommu = of_iommu_configure(dev, np);
>>> +       if (IS_ERR(iommu))
>>> +               return PTR_ERR(iommu);
>>> +
>>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>>                 iommu ? " " : " not ");
>>>
>>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>>> +
>>> +       return 0;
>>>  }
>>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>>
>>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>>> index 3cb2288..9499861 100644
>>> --- a/include/linux/of_device.h
>>> +++ b/include/linux/of_device.h
>>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>         return of_node_get(cpu_dev->of_node);
>>>  }
>>>
>>> -void of_dma_configure(struct device *dev, struct device_node *np);
>>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>>  void of_dma_deconfigure(struct device *dev);
>>>  #else /* CONFIG_OF */
>>>
>>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>  {
>>>         return NULL;
>>>  }
>>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>>> -{}
>>> +
>>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>>> +{
>>> +       return 0;
>>> +}
>>>  static inline void of_dma_deconfigure(struct device *dev)
>>>  {}
>>>  #endif /* CONFIG_OF */
>>
>> Gr{oetje,eeting}s,
>>
>>                         Geert
>>
>> --
>> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>>
>> In personal conversations with technical people, I call myself a hacker. But
>> when I'm talking to journalists I just say "programmer" or something like that.
>>                                 -- Linus Torvalds
>>
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Robin,

On 5/3/2017 3:24 PM, Robin Murphy wrote:
> Hi Geert,
> 
> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> Hi Sricharan,
>>
>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>
>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>> be handled separately from the .of_xlate() failures to support deferred
>>> probing.
>>>
>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>> been deferred, or having failed.
>>>
>>> The first case occurs when the device tree describes the bus master and
>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>> or the device driver has not been compiled in. Return NULL, the caller
>>> will configure the device without an IOMMU.
>>>
>>> The second and third cases are handled by deferring the probe of the bus
>>> master device which will eventually get reprobed after the IOMMU.
>>>
>>> The last case is currently handled by deferring the probe of the bus
>>> master device as well. A mechanism to either configure the bus master
>>> device without an IOMMU or to fail the bus master device probe depending
>>> on whether the IOMMU is optional or mandatory would be a good
>>> enhancement.
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>
>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> properties in DT now fail to probe.
> 
> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> registered then they should merely defer until we reach the point of
> giving up and ignoring the IOMMU. Is it just that you have no other
> late-probing drivers or post-init module loads to kick the deferred
> queue after that point? I did try to find a way to explicitly kick it
> from a suitably late initcall, but there didn't seem to be any obvious
> public interface - anyone have any suggestions?
> 
> I think that's more of a general problem with the probe deferral
> mechanism itself (I've seen the same thing happen with some of the
> CoreSight stuff on Juno due to the number of inter-component
> dependencies) rather than any specific fault of this series.
> 

I was thinking of an additional check like below to avoid the
situation ?

>From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Wed, 3 May 2017 13:16:59 +0530
Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER

While returning EPROBE_DEFER for iommu masters
take in to account of iommu nodes that could be
marked in DT as 'status=disabled', in which case
simply return NULL and let the master's probe
continue rather than deferring.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 9f44ee8..e6e9bec 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)

        ops = iommu_ops_from_fwnode(fwnode);
        if ((ops && !ops->of_xlate) ||
+           !of_device_is_available(iommu_spec->np) ||
            (!ops && !of_iommu_driver_present(iommu_spec->np)))
                return NULL;

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


Regards,
 Sricharan


> Robin.
> 
>> This can be fixed by either:
>>   - Disabling CONFIG_IPMMU_VMSA, or
>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>
>> Note that this was a bit hard to investigate, as R-Car Gen3 support wasn't
>> upstreamed yet, so bisection pointed to a merge commit.
>>
>>> ---
>>>  [*] Fixed minor comment from Bjorn for removing the pci.h header inclusion
>>>      in of_iommu.c
>>>
>>>  drivers/base/dma-mapping.c | 5 +++--
>>>  drivers/iommu/of_iommu.c   | 4 ++--
>>>  drivers/of/device.c        | 7 ++++++-
>>>  include/linux/of_device.h  | 9 ++++++---
>>>  4 files changed, 17 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>>> index 449b948..82bd45c 100644
>>> --- a/drivers/base/dma-mapping.c
>>> +++ b/drivers/base/dma-mapping.c
>>> @@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
>>>  {
>>>         struct device *bridge = NULL, *dma_dev = dev;
>>>         enum dev_dma_attr attr;
>>> +       int ret = 0;
>>>
>>>         if (dev_is_pci(dev)) {
>>>                 bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>> @@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
>>>         }
>>>
>>>         if (dma_dev->of_node) {
>>> -               of_dma_configure(dev, dma_dev->of_node);
>>> +               ret = of_dma_configure(dev, dma_dev->of_node);
>>>         } else if (has_acpi_companion(dma_dev)) {
>>>                 attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
>>>                 if (attr != DEV_DMA_NOT_SUPPORTED)
>>> @@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
>>>         if (bridge)
>>>                 pci_put_host_bridge_device(bridge);
>>>
>>> -       return 0;
>>> +       return ret;
>>>  }
>>>
>>>  void dma_deconfigure(struct device *dev)
>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>> index 1f92d98..2d04663 100644
>>> --- a/drivers/iommu/of_iommu.c
>>> +++ b/drivers/iommu/of_iommu.c
>>> @@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>>                         ops = ERR_PTR(err);
>>>         }
>>>
>>> -       return IS_ERR(ops) ? NULL : ops;
>>> +       return ops;
>>>  }
>>>
>>>  static int __init of_iommu_init(void)
>>> @@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
>>>         for_each_matching_node_and_match(np, matches, &match) {
>>>                 const of_iommu_init_fn init_fn = match->data;
>>>
>>> -               if (init_fn(np))
>>> +               if (init_fn && init_fn(np))
>>>                         pr_err("Failed to initialise IOMMU %s\n",
>>>                                 of_node_full_name(np));
>>>         }
>>> diff --git a/drivers/of/device.c b/drivers/of/device.c
>>> index c17c19d..ba51ca6 100644
>>> --- a/drivers/of/device.c
>>> +++ b/drivers/of/device.c
>>> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>>>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>>>   * to fix up DMA configuration.
>>>   */
>>> -void of_dma_configure(struct device *dev, struct device_node *np)
>>> +int of_dma_configure(struct device *dev, struct device_node *np)
>>>  {
>>>         u64 dma_addr, paddr, size;
>>>         int ret;
>>> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>>>                 coherent ? " " : " not ");
>>>
>>>         iommu = of_iommu_configure(dev, np);
>>> +       if (IS_ERR(iommu))
>>> +               return PTR_ERR(iommu);
>>> +
>>>         dev_dbg(dev, "device is%sbehind an iommu\n",
>>>                 iommu ? " " : " not ");
>>>
>>>         arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>>> +
>>> +       return 0;
>>>  }
>>>  EXPORT_SYMBOL_GPL(of_dma_configure);
>>>
>>> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
>>> index 3cb2288..9499861 100644
>>> --- a/include/linux/of_device.h
>>> +++ b/include/linux/of_device.h
>>> @@ -56,7 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>         return of_node_get(cpu_dev->of_node);
>>>  }
>>>
>>> -void of_dma_configure(struct device *dev, struct device_node *np);
>>> +int of_dma_configure(struct device *dev, struct device_node *np);
>>>  void of_dma_deconfigure(struct device *dev);
>>>  #else /* CONFIG_OF */
>>>
>>> @@ -105,8 +105,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>>>  {
>>>         return NULL;
>>>  }
>>> -static inline void of_dma_configure(struct device *dev, struct device_node *np)
>>> -{}
>>> +
>>> +static inline int of_dma_configure(struct device *dev, struct device_node *np)
>>> +{
>>> +       return 0;
>>> +}
>>>  static inline void of_dma_deconfigure(struct device *dev)
>>>  {}
>>>  #endif /* CONFIG_OF */
>>
>> Gr{oetje,eeting}s,
>>
>>                         Geert
>>
>> --
>> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org
>>
>> In personal conversations with technical people, I call myself a hacker. But
>> when I'm talking to journalists I just say "programmer" or something like that.
>>                                 -- Linus Torvalds
>>
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 5/3/2017 3:54 PM, Sricharan R wrote:
> Hi Robin,
> 
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> Hi Geert,
>>
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> Hi Sricharan,
>>>
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.
>>
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;
> 

While same as the other 'status=disabled' patch [1], better not to
defer the probe itself in the case ?

[1] https://patchwork.kernel.org/patch/9681211/

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 5/3/2017 3:54 PM, Sricharan R wrote:
> Hi Robin,
> 
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> Hi Geert,
>>
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> Hi Sricharan,
>>>
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.
>>
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;
> 

While same as the other 'status=disabled' patch [1], better not to
defer the probe itself in the case ?

[1] https://patchwork.kernel.org/patch/9681211/

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 5/3/2017 3:54 PM, Sricharan R wrote:
> Hi Robin,
> 
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> Hi Geert,
>>
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> Hi Sricharan,
>>>
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.
>>
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;
> 

While same as the other 'status=disabled' patch [1], better not to
defer the probe itself in the case ?

[1] https://patchwork.kernel.org/patch/9681211/

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi,

On 5/3/2017 3:54 PM, Sricharan R wrote:
> Hi Robin,
> 
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> Hi Geert,
>>
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> Hi Sricharan,
>>>
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.
>>
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;
> 

While same as the other 'status=disabled' patch [1], better not to
defer the probe itself in the case ?

[1] https://patchwork.kernel.org/patch/9681211/

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan, Robin,

On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.

I had a deeper look into the issue.

What changed, is that of_dma_configure() now returns an error code,
and dma_configure() looks at it.

Actually there are two failure modes:
  1. Devices with an iommus property pointing to a disabled IOMMU node.
     These return -EPROBE_DEFER, and are now retried forever.
  2. Devices that are blacklisted in the IPMMU driver, as we don't want to
     use them with an IOMMU yet.
     These return -ENODEV, due to ipmmu_of_xlate_dma().

> I was thinking of an additional check like below to avoid the
> situation ?
>
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
>
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Thanks, this fixes the first class of failures.

The second class can be worked around using:

--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -196,6 +196,11 @@ static const struct iommu_ops
                ops = of_iommu_xlate(dev, &iommu_spec);
                of_node_put(iommu_spec.np);
                idx++;
+               if (PTR_ERR(ops) == -ENODEV) {
+                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
+                                __func__);
+                       return NULL;
+               }
                if (IS_ERR_OR_NULL(ops))
                        break;
        }

but obviously that's too hackish to apply...
Magnus, do you have a suggestion?

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan, Robin,

On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.

I had a deeper look into the issue.

What changed, is that of_dma_configure() now returns an error code,
and dma_configure() looks at it.

Actually there are two failure modes:
  1. Devices with an iommus property pointing to a disabled IOMMU node.
     These return -EPROBE_DEFER, and are now retried forever.
  2. Devices that are blacklisted in the IPMMU driver, as we don't want to
     use them with an IOMMU yet.
     These return -ENODEV, due to ipmmu_of_xlate_dma().

> I was thinking of an additional check like below to avoid the
> situation ?
>
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Thanks, this fixes the first class of failures.

The second class can be worked around using:

--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -196,6 +196,11 @@ static const struct iommu_ops
                ops = of_iommu_xlate(dev, &iommu_spec);
                of_node_put(iommu_spec.np);
                idx++;
+               if (PTR_ERR(ops) == -ENODEV) {
+                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
+                                __func__);
+                       return NULL;
+               }
                if (IS_ERR_OR_NULL(ops))
                        break;
        }

but obviously that's too hackish to apply...
Magnus, do you have a suggestion?

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan, Robin,

On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.

I had a deeper look into the issue.

What changed, is that of_dma_configure() now returns an error code,
and dma_configure() looks at it.

Actually there are two failure modes:
  1. Devices with an iommus property pointing to a disabled IOMMU node.
     These return -EPROBE_DEFER, and are now retried forever.
  2. Devices that are blacklisted in the IPMMU driver, as we don't want to
     use them with an IOMMU yet.
     These return -ENODEV, due to ipmmu_of_xlate_dma().

> I was thinking of an additional check like below to avoid the
> situation ?
>
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Thanks, this fixes the first class of failures.

The second class can be worked around using:

--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -196,6 +196,11 @@ static const struct iommu_ops
                ops = of_iommu_xlate(dev, &iommu_spec);
                of_node_put(iommu_spec.np);
                idx++;
+               if (PTR_ERR(ops) == -ENODEV) {
+                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
+                                __func__);
+                       return NULL;
+               }
                if (IS_ERR_OR_NULL(ops))
                        break;
        }

but obviously that's too hackish to apply...
Magnus, do you have a suggestion?

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Geert Uytterhoeven]

Hi Sricharan, Robin,

On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>
>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>> be handled separately from the .of_xlate() failures to support deferred
>>>> probing.
>>>>
>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>> been deferred, or having failed.
>>>>
>>>> The first case occurs when the device tree describes the bus master and
>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>> will configure the device without an IOMMU.
>>>>
>>>> The second and third cases are handled by deferring the probe of the bus
>>>> master device which will eventually get reprobed after the IOMMU.
>>>>
>>>> The last case is currently handled by deferring the probe of the bus
>>>> master device as well. A mechanism to either configure the bus master
>>>> device without an IOMMU or to fail the bus master device probe depending
>>>> on whether the IOMMU is optional or mandatory would be a good
>>>> enhancement.
>>>>
>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>
>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>> properties in DT now fail to probe.
>>
>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> registered then they should merely defer until we reach the point of
>> giving up and ignoring the IOMMU. Is it just that you have no other
>> late-probing drivers or post-init module loads to kick the deferred
>> queue after that point? I did try to find a way to explicitly kick it
>> from a suitably late initcall, but there didn't seem to be any obvious
>> public interface - anyone have any suggestions?
>>
>> I think that's more of a general problem with the probe deferral
>> mechanism itself (I've seen the same thing happen with some of the
>> CoreSight stuff on Juno due to the number of inter-component
>> dependencies) rather than any specific fault of this series.

I had a deeper look into the issue.

What changed, is that of_dma_configure() now returns an error code,
and dma_configure() looks at it.

Actually there are two failure modes:
  1. Devices with an iommus property pointing to a disabled IOMMU node.
     These return -EPROBE_DEFER, and are now retried forever.
  2. Devices that are blacklisted in the IPMMU driver, as we don't want to
     use them with an IOMMU yet.
     These return -ENODEV, due to ipmmu_of_xlate_dma().

> I was thinking of an additional check like below to avoid the
> situation ?
>
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Thanks, this fixes the first class of failures.

The second class can be worked around using:

--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -196,6 +196,11 @@ static const struct iommu_ops
                ops = of_iommu_xlate(dev, &iommu_spec);
                of_node_put(iommu_spec.np);
                idx++;
+               if (PTR_ERR(ops) == -ENODEV) {
+                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
+                                __func__);
+                       return NULL;
+               }
                if (IS_ERR_OR_NULL(ops))
                        break;
        }

but obviously that's too hackish to apply...
Magnus, do you have a suggestion?

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Will Deacon]

Hi Sricharan,

On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>>
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>>
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>>
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>>
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>>
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>>
> >>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >>
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> > 
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the root
filesystem. The problem is that the USB controller is behind an SMMU which
is marked as 'status = "disabled"' in the devicetree. Whilst there was a
separate thread with Ard about exactly what this means in terms of the DMA
ops used by upstream devices, your patch above fixes the regression and
I think should go in regardless. The DMA ops issue will likely require
an additional DT binding anyway, to advertise the behaviour of the
IOMMU when it is disabled.

Tested-by: Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>
Acked-by: Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org>

Could you resend it as a proper patch, please?

Will

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Will Deacon]

Hi Sricharan,

On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>>
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>>
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>>
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>>
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>>
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> > 
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the root
filesystem. The problem is that the USB controller is behind an SMMU which
is marked as 'status = "disabled"' in the devicetree. Whilst there was a
separate thread with Ard about exactly what this means in terms of the DMA
ops used by upstream devices, your patch above fixes the regression and
I think should go in regardless. The DMA ops issue will likely require
an additional DT binding anyway, to advertise the behaviour of the
IOMMU when it is disabled.

Tested-by: Will Deacon <will.deacon@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>

Could you resend it as a proper patch, please?

Will

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Will Deacon]

Hi Sricharan,

On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>>
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>>
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>>
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>>
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>>
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> > 
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the root
filesystem. The problem is that the USB controller is behind an SMMU which
is marked as 'status = "disabled"' in the devicetree. Whilst there was a
separate thread with Ard about exactly what this means in terms of the DMA
ops used by upstream devices, your patch above fixes the regression and
I think should go in regardless. The DMA ops issue will likely require
an additional DT binding anyway, to advertise the behaviour of the
IOMMU when it is disabled.

Tested-by: Will Deacon <will.deacon@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>

Could you resend it as a proper patch, please?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Will Deacon]

Hi Sricharan,

On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>>
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>>
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>>
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>>
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>>
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> > 
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the root
filesystem. The problem is that the USB controller is behind an SMMU which
is marked as 'status = "disabled"' in the devicetree. Whilst there was a
separate thread with Ard about exactly what this means in terms of the DMA
ops used by upstream devices, your patch above fixes the regression and
I think should go in regardless. The DMA ops issue will likely require
an additional DT binding anyway, to advertise the behaviour of the
IOMMU when it is disabled.

Tested-by: Will Deacon <will.deacon@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>

Could you resend it as a proper patch, please?

Will

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>> 
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>> 
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>> 
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>> 
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>> 
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>> 
> >>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> >>> Signed-off-by: Laurent Pichart
> >>> <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >> 
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

This looks good to me, but won't be enough. The ipmmu-vmsa driver in v4.12-rc1 
doesn't call iommu_device_register() and thus won't be found by 
iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and thus 
will always be considered as absent.

I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been 
nice to check existing IOMMU drivers before merging this patch series...

> >> This can be fixed by either:
> >>   - Disabling CONFIG_IPMMU_VMSA, or
> >>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >> 
> >> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>> 
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>> 
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>> 
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>> 
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>> 
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>> 
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart
> >>> <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> 
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

This looks good to me, but won't be enough. The ipmmu-vmsa driver in v4.12-rc1 
doesn't call iommu_device_register() and thus won't be found by 
iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and thus 
will always be considered as absent.

I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been 
nice to check existing IOMMU drivers before merging this patch series...

> >> This can be fixed by either:
> >>   - Disabling CONFIG_IPMMU_VMSA, or
> >>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >> 
> >> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>> 
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>> 
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>> 
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>> 
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>> 
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>> 
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart
> >>> <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> 
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

This looks good to me, but won't be enough. The ipmmu-vmsa driver in v4.12-rc1 
doesn't call iommu_device_register() and thus won't be found by 
iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and thus 
will always be considered as absent.

I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been 
nice to check existing IOMMU drivers before merging this patch series...

> >> This can be fixed by either:
> >>   - Disabling CONFIG_IPMMU_VMSA, or
> >>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >> 
> >> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>> 
> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
> >>> be handled separately from the .of_xlate() failures to support deferred
> >>> probing.
> >>> 
> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
> >>> been deferred, or having failed.
> >>> 
> >>> The first case occurs when the device tree describes the bus master and
> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
> >>> or the device driver has not been compiled in. Return NULL, the caller
> >>> will configure the device without an IOMMU.
> >>> 
> >>> The second and third cases are handled by deferring the probe of the bus
> >>> master device which will eventually get reprobed after the IOMMU.
> >>> 
> >>> The last case is currently handled by deferring the probe of the bus
> >>> master device as well. A mechanism to either configure the bus master
> >>> device without an IOMMU or to fail the bus master device probe depending
> >>> on whether the IOMMU is optional or mandatory would be a good
> >>> enhancement.
> >>> 
> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>> Signed-off-by: Laurent Pichart
> >>> <laurent.pinchart+renesas@ideasonboard.com>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> 
> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >> properties in DT now fail to probe.
> > 
> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> > registered then they should merely defer until we reach the point of
> > giving up and ignoring the IOMMU. Is it just that you have no other
> > late-probing drivers or post-init module loads to kick the deferred
> > queue after that point? I did try to find a way to explicitly kick it
> > from a suitably late initcall, but there didn't seem to be any obvious
> > public interface - anyone have any suggestions?
> > 
> > I think that's more of a general problem with the probe deferral
> > mechanism itself (I've seen the same thing happen with some of the
> > CoreSight stuff on Juno due to the number of inter-component
> > dependencies) rather than any specific fault of this series.
> 
> I was thinking of an additional check like below to avoid the
> situation ?
> 
> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 13:16:59 +0530
> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> 
> While returning EPROBE_DEFER for iommu masters
> take in to account of iommu nodes that could be
> marked in DT as 'status=disabled', in which case
> simply return NULL and let the master's probe
> continue rather than deferring.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 9f44ee8..e6e9bec 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> *np)
> 
>         ops = iommu_ops_from_fwnode(fwnode);
>         if ((ops && !ops->of_xlate) ||
> +           !of_device_is_available(iommu_spec->np) ||
>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>                 return NULL;

This looks good to me, but won't be enough. The ipmmu-vmsa driver in v4.12-rc1 
doesn't call iommu_device_register() and thus won't be found by 
iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and thus 
will always be considered as absent.

I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been 
nice to check existing IOMMU drivers before merging this patch series...

> >> This can be fixed by either:
> >>   - Disabling CONFIG_IPMMU_VMSA, or
> >>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >> 
> >> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>>> 
> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
> >>>> to be handled separately from the .of_xlate() failures to support
> >>>> deferred probing.
> >>>> 
> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>> having been deferred, or having failed.
> >>>> 
> >>>> The first case occurs when the device tree describes the bus master
> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
> >>>> yet or the device driver has not been compiled in. Return NULL, the
> >>>> caller will configure the device without an IOMMU.
> >>>> 
> >>>> The second and third cases are handled by deferring the probe of the
> >>>> bus master device which will eventually get reprobed after the IOMMU.
> >>>> 
> >>>> The last case is currently handled by deferring the probe of the bus
> >>>> master device as well. A mechanism to either configure the bus master
> >>>> device without an IOMMU or to fail the bus master device probe
> >>>> depending on whether the IOMMU is optional or mandatory would be a good
> >>>> enhancement.
> >>>> 
> >>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
> >>>> Signed-off-by: Laurent Pichart
> >>>> <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> >>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >>> 
> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >>> properties in DT now fail to probe.
> >> 
> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> >> registered then they should merely defer until we reach the point of
> >> giving up and ignoring the IOMMU. Is it just that you have no other
> >> late-probing drivers or post-init module loads to kick the deferred
> >> queue after that point? I did try to find a way to explicitly kick it
> >> from a suitably late initcall, but there didn't seem to be any obvious
> >> public interface - anyone have any suggestions?
> >> 
> >> I think that's more of a general problem with the probe deferral
> >> mechanism itself (I've seen the same thing happen with some of the
> >> CoreSight stuff on Juno due to the number of inter-component
> >> dependencies) rather than any specific fault of this series.
> > 
> > I was thinking of an additional check like below to avoid the
> > situation ?
> > 
> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> > Date: Wed, 3 May 2017 13:16:59 +0530
> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > 
> > While returning EPROBE_DEFER for iommu masters
> > take in to account of iommu nodes that could be
> > marked in DT as 'status=disabled', in which case
> > simply return NULL and let the master's probe
> > continue rather than deferring.
> > 
> > Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> > ---
> > 
> >  drivers/iommu/of_iommu.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > index 9f44ee8..e6e9bec 100644
> > --- a/drivers/iommu/of_iommu.c
> > +++ b/drivers/iommu/of_iommu.c
> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> > *np)
> > 
> >         ops = iommu_ops_from_fwnode(fwnode);
> >         if ((ops && !ops->of_xlate) ||
> > +           !of_device_is_available(iommu_spec->np) ||
> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >                 return NULL;
> 
> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found by
> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and
> thus will always be considered as absent.
> 
> I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been
> nice to check existing IOMMU drivers before merging this patch series...

Please pardon the question, but has this patch series been tested on ARM32 ?

When the device is probed the arch_setup_dma_ops() function is called. It sets 
the device's dma_ops and the mapping (in __arm_iommu_attach_device()). If 
probe is deferred, arch_teardown_dma_ops() is called which in turn calls 
arch_teardown_dma_ops(). This removes the mapping but doesn't touch the 
dma_ops. The next time the device is probed, arch_setup_dma_ops() bails out 
immediately as the dma_ops are already set, leaving us with a device bound to 
IOMMU operations but with no mapping. This oopses later as soon as the kernel 
tries to map memory for the device through the IOMMU.

I might be missing something obvious, but I don't see how this can work.

> >>> This can be fixed by either:
> >>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>> 
> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>> 
> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
> >>>> to be handled separately from the .of_xlate() failures to support
> >>>> deferred probing.
> >>>> 
> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>> having been deferred, or having failed.
> >>>> 
> >>>> The first case occurs when the device tree describes the bus master
> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
> >>>> yet or the device driver has not been compiled in. Return NULL, the
> >>>> caller will configure the device without an IOMMU.
> >>>> 
> >>>> The second and third cases are handled by deferring the probe of the
> >>>> bus master device which will eventually get reprobed after the IOMMU.
> >>>> 
> >>>> The last case is currently handled by deferring the probe of the bus
> >>>> master device as well. A mechanism to either configure the bus master
> >>>> device without an IOMMU or to fail the bus master device probe
> >>>> depending on whether the IOMMU is optional or mandatory would be a good
> >>>> enhancement.
> >>>> 
> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>> Signed-off-by: Laurent Pichart
> >>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> 
> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >>> properties in DT now fail to probe.
> >> 
> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> >> registered then they should merely defer until we reach the point of
> >> giving up and ignoring the IOMMU. Is it just that you have no other
> >> late-probing drivers or post-init module loads to kick the deferred
> >> queue after that point? I did try to find a way to explicitly kick it
> >> from a suitably late initcall, but there didn't seem to be any obvious
> >> public interface - anyone have any suggestions?
> >> 
> >> I think that's more of a general problem with the probe deferral
> >> mechanism itself (I've seen the same thing happen with some of the
> >> CoreSight stuff on Juno due to the number of inter-component
> >> dependencies) rather than any specific fault of this series.
> > 
> > I was thinking of an additional check like below to avoid the
> > situation ?
> > 
> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > From: Sricharan R <sricharan@codeaurora.org>
> > Date: Wed, 3 May 2017 13:16:59 +0530
> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > 
> > While returning EPROBE_DEFER for iommu masters
> > take in to account of iommu nodes that could be
> > marked in DT as 'status=disabled', in which case
> > simply return NULL and let the master's probe
> > continue rather than deferring.
> > 
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > ---
> > 
> >  drivers/iommu/of_iommu.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > index 9f44ee8..e6e9bec 100644
> > --- a/drivers/iommu/of_iommu.c
> > +++ b/drivers/iommu/of_iommu.c
> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> > *np)
> > 
> >         ops = iommu_ops_from_fwnode(fwnode);
> >         if ((ops && !ops->of_xlate) ||
> > +           !of_device_is_available(iommu_spec->np) ||
> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >                 return NULL;
> 
> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found by
> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and
> thus will always be considered as absent.
> 
> I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been
> nice to check existing IOMMU drivers before merging this patch series...

Please pardon the question, but has this patch series been tested on ARM32 ?

When the device is probed the arch_setup_dma_ops() function is called. It sets 
the device's dma_ops and the mapping (in __arm_iommu_attach_device()). If 
probe is deferred, arch_teardown_dma_ops() is called which in turn calls 
arch_teardown_dma_ops(). This removes the mapping but doesn't touch the 
dma_ops. The next time the device is probed, arch_setup_dma_ops() bails out 
immediately as the dma_ops are already set, leaving us with a device bound to 
IOMMU operations but with no mapping. This oopses later as soon as the kernel 
tries to map memory for the device through the IOMMU.

I might be missing something obvious, but I don't see how this can work.

> >>> This can be fixed by either:
> >>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>> 
> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>> 
> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
> >>>> to be handled separately from the .of_xlate() failures to support
> >>>> deferred probing.
> >>>> 
> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>> having been deferred, or having failed.
> >>>> 
> >>>> The first case occurs when the device tree describes the bus master
> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
> >>>> yet or the device driver has not been compiled in. Return NULL, the
> >>>> caller will configure the device without an IOMMU.
> >>>> 
> >>>> The second and third cases are handled by deferring the probe of the
> >>>> bus master device which will eventually get reprobed after the IOMMU.
> >>>> 
> >>>> The last case is currently handled by deferring the probe of the bus
> >>>> master device as well. A mechanism to either configure the bus master
> >>>> device without an IOMMU or to fail the bus master device probe
> >>>> depending on whether the IOMMU is optional or mandatory would be a good
> >>>> enhancement.
> >>>> 
> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>> Signed-off-by: Laurent Pichart
> >>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> 
> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >>> properties in DT now fail to probe.
> >> 
> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> >> registered then they should merely defer until we reach the point of
> >> giving up and ignoring the IOMMU. Is it just that you have no other
> >> late-probing drivers or post-init module loads to kick the deferred
> >> queue after that point? I did try to find a way to explicitly kick it
> >> from a suitably late initcall, but there didn't seem to be any obvious
> >> public interface - anyone have any suggestions?
> >> 
> >> I think that's more of a general problem with the probe deferral
> >> mechanism itself (I've seen the same thing happen with some of the
> >> CoreSight stuff on Juno due to the number of inter-component
> >> dependencies) rather than any specific fault of this series.
> > 
> > I was thinking of an additional check like below to avoid the
> > situation ?
> > 
> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > From: Sricharan R <sricharan@codeaurora.org>
> > Date: Wed, 3 May 2017 13:16:59 +0530
> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > 
> > While returning EPROBE_DEFER for iommu masters
> > take in to account of iommu nodes that could be
> > marked in DT as 'status=disabled', in which case
> > simply return NULL and let the master's probe
> > continue rather than deferring.
> > 
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > ---
> > 
> >  drivers/iommu/of_iommu.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > index 9f44ee8..e6e9bec 100644
> > --- a/drivers/iommu/of_iommu.c
> > +++ b/drivers/iommu/of_iommu.c
> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> > *np)
> > 
> >         ops = iommu_ops_from_fwnode(fwnode);
> >         if ((ops && !ops->of_xlate) ||
> > +           !of_device_is_available(iommu_spec->np) ||
> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >                 return NULL;
> 
> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found by
> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and
> thus will always be considered as absent.
> 
> I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been
> nice to check existing IOMMU drivers before merging this patch series...

Please pardon the question, but has this patch series been tested on ARM32 ?

When the device is probed the arch_setup_dma_ops() function is called. It sets 
the device's dma_ops and the mapping (in __arm_iommu_attach_device()). If 
probe is deferred, arch_teardown_dma_ops() is called which in turn calls 
arch_teardown_dma_ops(). This removes the mapping but doesn't touch the 
dma_ops. The next time the device is probed, arch_setup_dma_ops() bails out 
immediately as the dma_ops are already set, leaving us with a device bound to 
IOMMU operations but with no mapping. This oopses later as soon as the kernel 
tries to map memory for the device through the IOMMU.

I might be missing something obvious, but I don't see how this can work.

> >>> This can be fixed by either:
> >>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>> 
> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>> 
> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
> >>>> to be handled separately from the .of_xlate() failures to support
> >>>> deferred probing.
> >>>> 
> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>> having been deferred, or having failed.
> >>>> 
> >>>> The first case occurs when the device tree describes the bus master
> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
> >>>> yet or the device driver has not been compiled in. Return NULL, the
> >>>> caller will configure the device without an IOMMU.
> >>>> 
> >>>> The second and third cases are handled by deferring the probe of the
> >>>> bus master device which will eventually get reprobed after the IOMMU.
> >>>> 
> >>>> The last case is currently handled by deferring the probe of the bus
> >>>> master device as well. A mechanism to either configure the bus master
> >>>> device without an IOMMU or to fail the bus master device probe
> >>>> depending on whether the IOMMU is optional or mandatory would be a good
> >>>> enhancement.
> >>>> 
> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>> Signed-off-by: Laurent Pichart
> >>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> 
> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
> >>> properties in DT now fail to probe.
> >> 
> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
> >> registered then they should merely defer until we reach the point of
> >> giving up and ignoring the IOMMU. Is it just that you have no other
> >> late-probing drivers or post-init module loads to kick the deferred
> >> queue after that point? I did try to find a way to explicitly kick it
> >> from a suitably late initcall, but there didn't seem to be any obvious
> >> public interface - anyone have any suggestions?
> >> 
> >> I think that's more of a general problem with the probe deferral
> >> mechanism itself (I've seen the same thing happen with some of the
> >> CoreSight stuff on Juno due to the number of inter-component
> >> dependencies) rather than any specific fault of this series.
> > 
> > I was thinking of an additional check like below to avoid the
> > situation ?
> > 
> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > From: Sricharan R <sricharan@codeaurora.org>
> > Date: Wed, 3 May 2017 13:16:59 +0530
> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > 
> > While returning EPROBE_DEFER for iommu masters
> > take in to account of iommu nodes that could be
> > marked in DT as 'status=disabled', in which case
> > simply return NULL and let the master's probe
> > continue rather than deferring.
> > 
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > ---
> > 
> >  drivers/iommu/of_iommu.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > index 9f44ee8..e6e9bec 100644
> > --- a/drivers/iommu/of_iommu.c
> > +++ b/drivers/iommu/of_iommu.c
> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
> > *np)
> > 
> >         ops = iommu_ops_from_fwnode(fwnode);
> >         if ((ops && !ops->of_xlate) ||
> > +           !of_device_is_available(iommu_spec->np) ||
> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >                 return NULL;
> 
> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found by
> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), and
> thus will always be considered as absent.
> 
> I agree that the ipmmu-vmsa driver needs to be fixed, but it would have been
> nice to check existing IOMMU drivers before merging this patch series...

Please pardon the question, but has this patch series been tested on ARM32 ?

When the device is probed the arch_setup_dma_ops() function is called. It sets 
the device's dma_ops and the mapping (in __arm_iommu_attach_device()). If 
probe is deferred, arch_teardown_dma_ops() is called which in turn calls 
arch_teardown_dma_ops(). This removes the mapping but doesn't touch the 
dma_ops. The next time the device is probed, arch_setup_dma_ops() bails out 
immediately as the dma_ops are already set, leaving us with a device bound to 
IOMMU operations but with no mapping. This oopses later as soon as the kernel 
tries to map memory for the device through the IOMMU.

I might be missing something obvious, but I don't see how this can work.

> >>> This can be fixed by either:
> >>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>> 
> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Laurent,

On 2017-05-16 03:04, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>
>> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
>> >>>> to be handled separately from the .of_xlate() failures to support
>> >>>> deferred probing.
>> >>>>
>> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>> having been deferred, or having failed.
>> >>>>
>> >>>> The first case occurs when the device tree describes the bus master
>> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
>> >>>> yet or the device driver has not been compiled in. Return NULL, the
>> >>>> caller will configure the device without an IOMMU.
>> >>>>
>> >>>> The second and third cases are handled by deferring the probe of the
>> >>>> bus master device which will eventually get reprobed after the IOMMU.
>> >>>>
>> >>>> The last case is currently handled by deferring the probe of the bus
>> >>>> master device as well. A mechanism to either configure the bus master
>> >>>> device without an IOMMU or to fail the bus master device probe
>> >>>> depending on whether the IOMMU is optional or mandatory would be a good
>> >>>> enhancement.
>> >>>>
>> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>> Signed-off-by: Laurent Pichart
>> >>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>
>> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >>> properties in DT now fail to probe.
>> >>
>> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> >> registered then they should merely defer until we reach the point of
>> >> giving up and ignoring the IOMMU. Is it just that you have no other
>> >> late-probing drivers or post-init module loads to kick the deferred
>> >> queue after that point? I did try to find a way to explicitly kick it
>> >> from a suitably late initcall, but there didn't seem to be any obvious
>> >> public interface - anyone have any suggestions?
>> >>
>> >> I think that's more of a general problem with the probe deferral
>> >> mechanism itself (I've seen the same thing happen with some of the
>> >> CoreSight stuff on Juno due to the number of inter-component
>> >> dependencies) rather than any specific fault of this series.
>> >
>> > I was thinking of an additional check like below to avoid the
>> > situation ?
>> >
>> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> > From: Sricharan R <sricharan@codeaurora.org>
>> > Date: Wed, 3 May 2017 13:16:59 +0530
>> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >
>> > While returning EPROBE_DEFER for iommu masters
>> > take in to account of iommu nodes that could be
>> > marked in DT as 'status=disabled', in which case
>> > simply return NULL and let the master's probe
>> > continue rather than deferring.
>> >
>> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> > ---
>> >
>> >  drivers/iommu/of_iommu.c | 1 +
>> >  1 file changed, 1 insertion(+)
>> >
>> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> > index 9f44ee8..e6e9bec 100644
>> > --- a/drivers/iommu/of_iommu.c
>> > +++ b/drivers/iommu/of_iommu.c
>> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
>> > *np)
>> >
>> >         ops = iommu_ops_from_fwnode(fwnode);
>> >         if ((ops && !ops->of_xlate) ||
>> > +           !of_device_is_available(iommu_spec->np) ||
>> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >                 return NULL;
>> 
>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found 
>> by
>> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), 
>> and
>> thus will always be considered as absent.
>> 
>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would 
>> have been
>> nice to check existing IOMMU drivers before merging this patch 
>> series...
> 
> Please pardon the question, but has this patch series been tested on 
> ARM32 ?
> 
> When the device is probed the arch_setup_dma_ops() function is called. 
> It sets
> the device's dma_ops and the mapping (in __arm_iommu_attach_device()). 
> If
> probe is deferred, arch_teardown_dma_ops() is called which in turn 
> calls
> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails 
> out
> immediately as the dma_ops are already set, leaving us with a device 
> bound to
> IOMMU operations but with no mapping. This oopses later as soon as the 
> kernel
> tries to map memory for the device through the IOMMU.

Resetting the dma_ops for arm32 was added in this patch [1], which I 
missed
to send in the original series, but now have added to Russell's patch 
tracking
system.

[1] https://patchwork.kernel.org/patch/9434105/

Regards,
  Sricharan

> 
> I might be missing something obvious, but I don't see how this can 
> work.
> 
>> >>> This can be fixed by either:
>> >>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>
>> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Laurent,

On 2017-05-16 03:04, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>
>> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
>> >>>> to be handled separately from the .of_xlate() failures to support
>> >>>> deferred probing.
>> >>>>
>> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>> having been deferred, or having failed.
>> >>>>
>> >>>> The first case occurs when the device tree describes the bus master
>> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
>> >>>> yet or the device driver has not been compiled in. Return NULL, the
>> >>>> caller will configure the device without an IOMMU.
>> >>>>
>> >>>> The second and third cases are handled by deferring the probe of the
>> >>>> bus master device which will eventually get reprobed after the IOMMU.
>> >>>>
>> >>>> The last case is currently handled by deferring the probe of the bus
>> >>>> master device as well. A mechanism to either configure the bus master
>> >>>> device without an IOMMU or to fail the bus master device probe
>> >>>> depending on whether the IOMMU is optional or mandatory would be a good
>> >>>> enhancement.
>> >>>>
>> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>> Signed-off-by: Laurent Pichart
>> >>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>
>> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >>> properties in DT now fail to probe.
>> >>
>> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> >> registered then they should merely defer until we reach the point of
>> >> giving up and ignoring the IOMMU. Is it just that you have no other
>> >> late-probing drivers or post-init module loads to kick the deferred
>> >> queue after that point? I did try to find a way to explicitly kick it
>> >> from a suitably late initcall, but there didn't seem to be any obvious
>> >> public interface - anyone have any suggestions?
>> >>
>> >> I think that's more of a general problem with the probe deferral
>> >> mechanism itself (I've seen the same thing happen with some of the
>> >> CoreSight stuff on Juno due to the number of inter-component
>> >> dependencies) rather than any specific fault of this series.
>> >
>> > I was thinking of an additional check like below to avoid the
>> > situation ?
>> >
>> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> > From: Sricharan R <sricharan@codeaurora.org>
>> > Date: Wed, 3 May 2017 13:16:59 +0530
>> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >
>> > While returning EPROBE_DEFER for iommu masters
>> > take in to account of iommu nodes that could be
>> > marked in DT as 'status=disabled', in which case
>> > simply return NULL and let the master's probe
>> > continue rather than deferring.
>> >
>> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> > ---
>> >
>> >  drivers/iommu/of_iommu.c | 1 +
>> >  1 file changed, 1 insertion(+)
>> >
>> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> > index 9f44ee8..e6e9bec 100644
>> > --- a/drivers/iommu/of_iommu.c
>> > +++ b/drivers/iommu/of_iommu.c
>> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
>> > *np)
>> >
>> >         ops = iommu_ops_from_fwnode(fwnode);
>> >         if ((ops && !ops->of_xlate) ||
>> > +           !of_device_is_available(iommu_spec->np) ||
>> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >                 return NULL;
>> 
>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found 
>> by
>> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), 
>> and
>> thus will always be considered as absent.
>> 
>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would 
>> have been
>> nice to check existing IOMMU drivers before merging this patch 
>> series...
> 
> Please pardon the question, but has this patch series been tested on 
> ARM32 ?
> 
> When the device is probed the arch_setup_dma_ops() function is called. 
> It sets
> the device's dma_ops and the mapping (in __arm_iommu_attach_device()). 
> If
> probe is deferred, arch_teardown_dma_ops() is called which in turn 
> calls
> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails 
> out
> immediately as the dma_ops are already set, leaving us with a device 
> bound to
> IOMMU operations but with no mapping. This oopses later as soon as the 
> kernel
> tries to map memory for the device through the IOMMU.

Resetting the dma_ops for arm32 was added in this patch [1], which I 
missed
to send in the original series, but now have added to Russell's patch 
tracking
system.

[1] https://patchwork.kernel.org/patch/9434105/

Regards,
  Sricharan

> 
> I might be missing something obvious, but I don't see how this can 
> work.
> 
>> >>> This can be fixed by either:
>> >>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>
>> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan at codeaurora.org]

Hi Laurent,

On 2017-05-16 03:04, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> > On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>
>> >>>> Failures to look up an IOMMU when parsing the DT iommus property need
>> >>>> to be handled separately from the .of_xlate() failures to support
>> >>>> deferred probing.
>> >>>>
>> >>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>> having been deferred, or having failed.
>> >>>>
>> >>>> The first case occurs when the device tree describes the bus master
>> >>>> and IOMMU topology correctly but no device driver exists for the IOMMU
>> >>>> yet or the device driver has not been compiled in. Return NULL, the
>> >>>> caller will configure the device without an IOMMU.
>> >>>>
>> >>>> The second and third cases are handled by deferring the probe of the
>> >>>> bus master device which will eventually get reprobed after the IOMMU.
>> >>>>
>> >>>> The last case is currently handled by deferring the probe of the bus
>> >>>> master device as well. A mechanism to either configure the bus master
>> >>>> device without an IOMMU or to fail the bus master device probe
>> >>>> depending on whether the IOMMU is optional or mandatory would be a good
>> >>>> enhancement.
>> >>>>
>> >>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>> Signed-off-by: Laurent Pichart
>> >>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>
>> >>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >>> properties in DT now fail to probe.
>> >>
>> >> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> >> registered then they should merely defer until we reach the point of
>> >> giving up and ignoring the IOMMU. Is it just that you have no other
>> >> late-probing drivers or post-init module loads to kick the deferred
>> >> queue after that point? I did try to find a way to explicitly kick it
>> >> from a suitably late initcall, but there didn't seem to be any obvious
>> >> public interface - anyone have any suggestions?
>> >>
>> >> I think that's more of a general problem with the probe deferral
>> >> mechanism itself (I've seen the same thing happen with some of the
>> >> CoreSight stuff on Juno due to the number of inter-component
>> >> dependencies) rather than any specific fault of this series.
>> >
>> > I was thinking of an additional check like below to avoid the
>> > situation ?
>> >
>> > From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> > From: Sricharan R <sricharan@codeaurora.org>
>> > Date: Wed, 3 May 2017 13:16:59 +0530
>> > Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >
>> > While returning EPROBE_DEFER for iommu masters
>> > take in to account of iommu nodes that could be
>> > marked in DT as 'status=disabled', in which case
>> > simply return NULL and let the master's probe
>> > continue rather than deferring.
>> >
>> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> > ---
>> >
>> >  drivers/iommu/of_iommu.c | 1 +
>> >  1 file changed, 1 insertion(+)
>> >
>> > diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> > index 9f44ee8..e6e9bec 100644
>> > --- a/drivers/iommu/of_iommu.c
>> > +++ b/drivers/iommu/of_iommu.c
>> > @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node
>> > *np)
>> >
>> >         ops = iommu_ops_from_fwnode(fwnode);
>> >         if ((ops && !ops->of_xlate) ||
>> > +           !of_device_is_available(iommu_spec->np) ||
>> >             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >                 return NULL;
>> 
>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found 
>> by
>> iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(), 
>> and
>> thus will always be considered as absent.
>> 
>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would 
>> have been
>> nice to check existing IOMMU drivers before merging this patch 
>> series...
> 
> Please pardon the question, but has this patch series been tested on 
> ARM32 ?
> 
> When the device is probed the arch_setup_dma_ops() function is called. 
> It sets
> the device's dma_ops and the mapping (in __arm_iommu_attach_device()). 
> If
> probe is deferred, arch_teardown_dma_ops() is called which in turn 
> calls
> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails 
> out
> immediately as the dma_ops are already set, leaving us with a device 
> bound to
> IOMMU operations but with no mapping. This oopses later as soon as the 
> kernel
> tries to map memory for the device through the IOMMU.

Resetting the dma_ops for arm32 was added in this patch [1], which I 
missed
to send in the original series, but now have added to Russell's patch 
tracking
system.

[1] https://patchwork.kernel.org/patch/9434105/

Regards,
  Sricharan

> 
> I might be missing something obvious, but I don't see how this can 
> work.
> 
>> >>> This can be fixed by either:
>> >>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>
>> >>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>> wasn't upstreamed yet, so bisection pointed to a merge commit.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Will,

On 2017-05-15 19:52, Will Deacon wrote:
> Hi Sricharan,
> 
> On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>
>> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> >>> be handled separately from the .of_xlate() failures to support deferred
>> >>> probing.
>> >>>
>> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> >>> been deferred, or having failed.
>> >>>
>> >>> The first case occurs when the device tree describes the bus master and
>> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> >>> or the device driver has not been compiled in. Return NULL, the caller
>> >>> will configure the device without an IOMMU.
>> >>>
>> >>> The second and third cases are handled by deferring the probe of the bus
>> >>> master device which will eventually get reprobed after the IOMMU.
>> >>>
>> >>> The last case is currently handled by deferring the probe of the bus
>> >>> master device as well. A mechanism to either configure the bus master
>> >>> device without an IOMMU or to fail the bus master device probe depending
>> >>> on whether the IOMMU is optional or mandatory would be a good
>> >>> enhancement.
>> >>>
>> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>
>> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >> properties in DT now fail to probe.
>> >
>> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> > registered then they should merely defer until we reach the point of
>> > giving up and ignoring the IOMMU. Is it just that you have no other
>> > late-probing drivers or post-init module loads to kick the deferred
>> > queue after that point? I did try to find a way to explicitly kick it
>> > from a suitably late initcall, but there didn't seem to be any obvious
>> > public interface - anyone have any suggestions?
>> >
>> > I think that's more of a general problem with the probe deferral
>> > mechanism itself (I've seen the same thing happen with some of the
>> > CoreSight stuff on Juno due to the number of inter-component
>> > dependencies) rather than any specific fault of this series.
>> >
>> 
>> I was thinking of an additional check like below to avoid the
>> situation ?
>> 
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> 
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct 
>> device_node *np)
>> 
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
> 
> Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the 
> root
> filesystem. The problem is that the USB controller is behind an SMMU 
> which
> is marked as 'status = "disabled"' in the devicetree. Whilst there was 
> a
> separate thread with Ard about exactly what this means in terms of the 
> DMA
> ops used by upstream devices, your patch above fixes the regression and
> I think should go in regardless. The DMA ops issue will likely require
> an additional DT binding anyway, to advertise the behaviour of the
> IOMMU when it is disabled.
> 
> Tested-by: Will Deacon <will.deacon@arm.com>
> Acked-by: Will Deacon <will.deacon@arm.com>
> 
> Could you resend it as a proper patch, please?

Sure, will send this as a separate patch.

Regards,
  Sricharan

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Will,

On 2017-05-15 19:52, Will Deacon wrote:
> Hi Sricharan,
> 
> On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>
>> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> >>> be handled separately from the .of_xlate() failures to support deferred
>> >>> probing.
>> >>>
>> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> >>> been deferred, or having failed.
>> >>>
>> >>> The first case occurs when the device tree describes the bus master and
>> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> >>> or the device driver has not been compiled in. Return NULL, the caller
>> >>> will configure the device without an IOMMU.
>> >>>
>> >>> The second and third cases are handled by deferring the probe of the bus
>> >>> master device which will eventually get reprobed after the IOMMU.
>> >>>
>> >>> The last case is currently handled by deferring the probe of the bus
>> >>> master device as well. A mechanism to either configure the bus master
>> >>> device without an IOMMU or to fail the bus master device probe depending
>> >>> on whether the IOMMU is optional or mandatory would be a good
>> >>> enhancement.
>> >>>
>> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>
>> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >> properties in DT now fail to probe.
>> >
>> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> > registered then they should merely defer until we reach the point of
>> > giving up and ignoring the IOMMU. Is it just that you have no other
>> > late-probing drivers or post-init module loads to kick the deferred
>> > queue after that point? I did try to find a way to explicitly kick it
>> > from a suitably late initcall, but there didn't seem to be any obvious
>> > public interface - anyone have any suggestions?
>> >
>> > I think that's more of a general problem with the probe deferral
>> > mechanism itself (I've seen the same thing happen with some of the
>> > CoreSight stuff on Juno due to the number of inter-component
>> > dependencies) rather than any specific fault of this series.
>> >
>> 
>> I was thinking of an additional check like below to avoid the
>> situation ?
>> 
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> 
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct 
>> device_node *np)
>> 
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
> 
> Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the 
> root
> filesystem. The problem is that the USB controller is behind an SMMU 
> which
> is marked as 'status = "disabled"' in the devicetree. Whilst there was 
> a
> separate thread with Ard about exactly what this means in terms of the 
> DMA
> ops used by upstream devices, your patch above fixes the regression and
> I think should go in regardless. The DMA ops issue will likely require
> an additional DT binding anyway, to advertise the behaviour of the
> IOMMU when it is disabled.
> 
> Tested-by: Will Deacon <will.deacon@arm.com>
> Acked-by: Will Deacon <will.deacon@arm.com>
> 
> Could you resend it as a proper patch, please?

Sure, will send this as a separate patch.

Regards,
  Sricharan

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan at codeaurora.org]

Hi Will,

On 2017-05-15 19:52, Will Deacon wrote:
> Hi Sricharan,
> 
> On Wed, May 03, 2017 at 03:54:59PM +0530, Sricharan R wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> > On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> >>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>
>> >>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> >>> be handled separately from the .of_xlate() failures to support deferred
>> >>> probing.
>> >>>
>> >>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> >>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> >>> been deferred, or having failed.
>> >>>
>> >>> The first case occurs when the device tree describes the bus master and
>> >>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> >>> or the device driver has not been compiled in. Return NULL, the caller
>> >>> will configure the device without an IOMMU.
>> >>>
>> >>> The second and third cases are handled by deferring the probe of the bus
>> >>> master device which will eventually get reprobed after the IOMMU.
>> >>>
>> >>> The last case is currently handled by deferring the probe of the bus
>> >>> master device as well. A mechanism to either configure the bus master
>> >>> device without an IOMMU or to fail the bus master device probe depending
>> >>> on whether the IOMMU is optional or mandatory would be a good
>> >>> enhancement.
>> >>>
>> >>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>
>> >> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>> >> properties in DT now fail to probe.
>> >
>> > How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>> > registered then they should merely defer until we reach the point of
>> > giving up and ignoring the IOMMU. Is it just that you have no other
>> > late-probing drivers or post-init module loads to kick the deferred
>> > queue after that point? I did try to find a way to explicitly kick it
>> > from a suitably late initcall, but there didn't seem to be any obvious
>> > public interface - anyone have any suggestions?
>> >
>> > I think that's more of a general problem with the probe deferral
>> > mechanism itself (I've seen the same thing happen with some of the
>> > CoreSight stuff on Juno due to the number of inter-component
>> > dependencies) rather than any specific fault of this series.
>> >
>> 
>> I was thinking of an additional check like below to avoid the
>> situation ?
>> 
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> 
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct 
>> device_node *np)
>> 
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
> 
> Without this patch, v4.12-rc1 hangs on my Juno waiting to mount the 
> root
> filesystem. The problem is that the USB controller is behind an SMMU 
> which
> is marked as 'status = "disabled"' in the devicetree. Whilst there was 
> a
> separate thread with Ard about exactly what this means in terms of the 
> DMA
> ops used by upstream devices, your patch above fixes the regression and
> I think should go in regardless. The DMA ops issue will likely require
> an additional DT binding anyway, to advertise the behaviour of the
> IOMMU when it is disabled.
> 
> Tested-by: Will Deacon <will.deacon@arm.com>
> Acked-by: Will Deacon <will.deacon@arm.com>
> 
> Could you resend it as a proper patch, please?

Sure, will send this as a separate patch.

Regards,
  Sricharan

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> On 2017-05-16 03:04, Laurent Pinchart wrote:
> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> 
> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>> support deferred probing.
> >>>>>> 
> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>>>> having been deferred, or having failed.
> >>>>>> 
> >>>>>> The first case occurs when the device tree describes the bus master
> >>>>>> and IOMMU topology correctly but no device driver exists for the
> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> >>>>>> the caller will configure the device without an IOMMU.
> >>>>>> 
> >>>>>> The second and third cases are handled by deferring the probe of the
> >>>>>> bus master device which will eventually get reprobed after the
> >>>>>> IOMMU.
> >>>>>> 
> >>>>>> The last case is currently handled by deferring the probe of the bus
> >>>>>> master device as well. A mechanism to either configure the bus
> >>>>>> master device without an IOMMU or to fail the bus master device probe
> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> >>>>>> good enhancement.
> >>>>>> 
> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>> Signed-off-by: Laurent Pichart
> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> 
> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>> iommus properties in DT now fail to probe.
> >>>> 
> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>> ops registered then they should merely defer until we reach the point
> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> >>>> late-probing drivers or post-init module loads to kick the deferred
> >>>> queue after that point? I did try to find a way to explicitly kick it
> >>>> from a suitably late initcall, but there didn't seem to be any obvious
> >>>> public interface - anyone have any suggestions?
> >>>> 
> >>>> I think that's more of a general problem with the probe deferral
> >>>> mechanism itself (I've seen the same thing happen with some of the
> >>>> CoreSight stuff on Juno due to the number of inter-component
> >>>> dependencies) rather than any specific fault of this series.
> >>> 
> >>> I was thinking of an additional check like below to avoid the
> >>> situation ?
> >>> 
> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> >>> From: Sricharan R <sricharan@codeaurora.org>
> >>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>> 
> >>> While returning EPROBE_DEFER for iommu masters
> >>> take in to account of iommu nodes that could be
> >>> marked in DT as 'status=disabled', in which case
> >>> simply return NULL and let the master's probe
> >>> continue rather than deferring.
> >>> 
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> ---
> >>> 
> >>>  drivers/iommu/of_iommu.c | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>> 
> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>> index 9f44ee8..e6e9bec 100644
> >>> --- a/drivers/iommu/of_iommu.c
> >>> +++ b/drivers/iommu/of_iommu.c
> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>> device_node *np)
> >>> 
> >>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>         if ((ops && !ops->of_xlate) ||
> >>> +           !of_device_is_available(iommu_spec->np) ||
> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>                 return NULL;
> >> 
> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> >> and thus will always be considered as absent.
> >> 
> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >> have been nice to check existing IOMMU drivers before merging this patch
> >> series...
> > 
> > Please pardon the question, but has this patch series been tested on
> > ARM32 ?
> > 
> > When the device is probed the arch_setup_dma_ops() function is called.
> > It sets the device's dma_ops and the mapping (in
> > __arm_iommu_attach_device()). If probe is deferred,
> > arch_teardown_dma_ops() is called which in turn calls
> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > out immediately as the dma_ops are already set, leaving us with a device
> > bound to IOMMU operations but with no mapping. This oopses later as soon
> > as the kernel tries to map memory for the device through the IOMMU.
> 
> Resetting the dma_ops for arm32 was added in this patch [1], which I
> missed to send in the original series, but now have added to Russell's patch
> tracking system.

Thank you. I fear that won't be enough though.

> [1] https://patchwork.kernel.org/patch/9434105/

Quoting the patch:

> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> ,dma_ops should be cleared in the teardown path. Otherwise
> this causes problem when the probe of device is retried after
> being deferred. The device's iommu structures are cleared
> after EPROBEDEFER error, but on the next try dma_ops will still
> be set to old value, which is not right.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> ---
>   arch/arm/mm/dma-mapping.c | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index ab4f745..a40f03e 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
*dev)
>   	__arm_iommu_detach_device(dev);
>   	arm_iommu_release_mapping(mapping);
> +	set_dma_ops(dev, NULL);
>   }
>   #else

The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().

However, the situation is perhaps more complex. Note the check at the 
beginning of arch_setup_dma_ops():

	/*
	 * Don't override the dma_ops if they have already been set. Ideally
	 * this should be the only location where dma_ops are set, remove this
	 * check when all other callers of set_dma_ops will have disappeared.
	 */
	if (dev->dma_ops)
		return;

If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
them. To be safe you should only set them to NULL if they have been set by 
arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
to add a field to struct dev_archdata to store that information. To avoid 
growing the structure, which is embedded in every struct device, you could 
possibly turn the dma_coherent bool into a bitfield.

@@ -19,7 +19,8 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
+	bool dma_coherent:1;
+	bool dma_ops_setup:1;
 };
 
 struct omap_device;

I haven't checked, however, whether the dma_coherent field would need to be 
accessed atomically, so this might be a bad idea.

Last but not least, a fix must be merged in v4.12, and the sooner the better.

> > I might be missing something obvious, but I don't see how this can
> > work.
> > 
> >>>>> This can be fixed by either:
> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>> 
> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> On 2017-05-16 03:04, Laurent Pinchart wrote:
> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> 
> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>> support deferred probing.
> >>>>>> 
> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>>>> having been deferred, or having failed.
> >>>>>> 
> >>>>>> The first case occurs when the device tree describes the bus master
> >>>>>> and IOMMU topology correctly but no device driver exists for the
> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> >>>>>> the caller will configure the device without an IOMMU.
> >>>>>> 
> >>>>>> The second and third cases are handled by deferring the probe of the
> >>>>>> bus master device which will eventually get reprobed after the
> >>>>>> IOMMU.
> >>>>>> 
> >>>>>> The last case is currently handled by deferring the probe of the bus
> >>>>>> master device as well. A mechanism to either configure the bus
> >>>>>> master device without an IOMMU or to fail the bus master device probe
> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> >>>>>> good enhancement.
> >>>>>> 
> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>> Signed-off-by: Laurent Pichart
> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> 
> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>> iommus properties in DT now fail to probe.
> >>>> 
> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>> ops registered then they should merely defer until we reach the point
> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> >>>> late-probing drivers or post-init module loads to kick the deferred
> >>>> queue after that point? I did try to find a way to explicitly kick it
> >>>> from a suitably late initcall, but there didn't seem to be any obvious
> >>>> public interface - anyone have any suggestions?
> >>>> 
> >>>> I think that's more of a general problem with the probe deferral
> >>>> mechanism itself (I've seen the same thing happen with some of the
> >>>> CoreSight stuff on Juno due to the number of inter-component
> >>>> dependencies) rather than any specific fault of this series.
> >>> 
> >>> I was thinking of an additional check like below to avoid the
> >>> situation ?
> >>> 
> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> >>> From: Sricharan R <sricharan@codeaurora.org>
> >>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>> 
> >>> While returning EPROBE_DEFER for iommu masters
> >>> take in to account of iommu nodes that could be
> >>> marked in DT as 'status=disabled', in which case
> >>> simply return NULL and let the master's probe
> >>> continue rather than deferring.
> >>> 
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> ---
> >>> 
> >>>  drivers/iommu/of_iommu.c | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>> 
> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>> index 9f44ee8..e6e9bec 100644
> >>> --- a/drivers/iommu/of_iommu.c
> >>> +++ b/drivers/iommu/of_iommu.c
> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>> device_node *np)
> >>> 
> >>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>         if ((ops && !ops->of_xlate) ||
> >>> +           !of_device_is_available(iommu_spec->np) ||
> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>                 return NULL;
> >> 
> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> >> and thus will always be considered as absent.
> >> 
> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >> have been nice to check existing IOMMU drivers before merging this patch
> >> series...
> > 
> > Please pardon the question, but has this patch series been tested on
> > ARM32 ?
> > 
> > When the device is probed the arch_setup_dma_ops() function is called.
> > It sets the device's dma_ops and the mapping (in
> > __arm_iommu_attach_device()). If probe is deferred,
> > arch_teardown_dma_ops() is called which in turn calls
> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > out immediately as the dma_ops are already set, leaving us with a device
> > bound to IOMMU operations but with no mapping. This oopses later as soon
> > as the kernel tries to map memory for the device through the IOMMU.
> 
> Resetting the dma_ops for arm32 was added in this patch [1], which I
> missed to send in the original series, but now have added to Russell's patch
> tracking system.

Thank you. I fear that won't be enough though.

> [1] https://patchwork.kernel.org/patch/9434105/

Quoting the patch:

> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> ,dma_ops should be cleared in the teardown path. Otherwise
> this causes problem when the probe of device is retried after
> being deferred. The device's iommu structures are cleared
> after EPROBEDEFER error, but on the next try dma_ops will still
> be set to old value, which is not right.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> ---
>   arch/arm/mm/dma-mapping.c | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index ab4f745..a40f03e 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
*dev)
>   	__arm_iommu_detach_device(dev);
>   	arm_iommu_release_mapping(mapping);
> +	set_dma_ops(dev, NULL);
>   }
>   #else

The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().

However, the situation is perhaps more complex. Note the check at the 
beginning of arch_setup_dma_ops():

	/*
	 * Don't override the dma_ops if they have already been set. Ideally
	 * this should be the only location where dma_ops are set, remove this
	 * check when all other callers of set_dma_ops will have disappeared.
	 */
	if (dev->dma_ops)
		return;

If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
them. To be safe you should only set them to NULL if they have been set by 
arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
to add a field to struct dev_archdata to store that information. To avoid 
growing the structure, which is embedded in every struct device, you could 
possibly turn the dma_coherent bool into a bitfield.

@@ -19,7 +19,8 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
+	bool dma_coherent:1;
+	bool dma_ops_setup:1;
 };
 
 struct omap_device;

I haven't checked, however, whether the dma_coherent field would need to be 
accessed atomically, so this might be a bad idea.

Last but not least, a fix must be merged in v4.12, and the sooner the better.

> > I might be missing something obvious, but I don't see how this can
> > work.
> > 
> >>>>> This can be fixed by either:
> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>> 
> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> On 2017-05-16 03:04, Laurent Pinchart wrote:
> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> 
> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>> support deferred probing.
> >>>>>> 
> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> >>>>>> having been deferred, or having failed.
> >>>>>> 
> >>>>>> The first case occurs when the device tree describes the bus master
> >>>>>> and IOMMU topology correctly but no device driver exists for the
> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> >>>>>> the caller will configure the device without an IOMMU.
> >>>>>> 
> >>>>>> The second and third cases are handled by deferring the probe of the
> >>>>>> bus master device which will eventually get reprobed after the
> >>>>>> IOMMU.
> >>>>>> 
> >>>>>> The last case is currently handled by deferring the probe of the bus
> >>>>>> master device as well. A mechanism to either configure the bus
> >>>>>> master device without an IOMMU or to fail the bus master device probe
> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> >>>>>> good enhancement.
> >>>>>> 
> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>> Signed-off-by: Laurent Pichart
> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> 
> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>> iommus properties in DT now fail to probe.
> >>>> 
> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>> ops registered then they should merely defer until we reach the point
> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> >>>> late-probing drivers or post-init module loads to kick the deferred
> >>>> queue after that point? I did try to find a way to explicitly kick it
> >>>> from a suitably late initcall, but there didn't seem to be any obvious
> >>>> public interface - anyone have any suggestions?
> >>>> 
> >>>> I think that's more of a general problem with the probe deferral
> >>>> mechanism itself (I've seen the same thing happen with some of the
> >>>> CoreSight stuff on Juno due to the number of inter-component
> >>>> dependencies) rather than any specific fault of this series.
> >>> 
> >>> I was thinking of an additional check like below to avoid the
> >>> situation ?
> >>> 
> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> >>> From: Sricharan R <sricharan@codeaurora.org>
> >>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>> 
> >>> While returning EPROBE_DEFER for iommu masters
> >>> take in to account of iommu nodes that could be
> >>> marked in DT as 'status=disabled', in which case
> >>> simply return NULL and let the master's probe
> >>> continue rather than deferring.
> >>> 
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> ---
> >>> 
> >>>  drivers/iommu/of_iommu.c | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>> 
> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>> index 9f44ee8..e6e9bec 100644
> >>> --- a/drivers/iommu/of_iommu.c
> >>> +++ b/drivers/iommu/of_iommu.c
> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>> device_node *np)
> >>> 
> >>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>         if ((ops && !ops->of_xlate) ||
> >>> +           !of_device_is_available(iommu_spec->np) ||
> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>                 return NULL;
> >> 
> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> >> and thus will always be considered as absent.
> >> 
> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >> have been nice to check existing IOMMU drivers before merging this patch
> >> series...
> > 
> > Please pardon the question, but has this patch series been tested on
> > ARM32 ?
> > 
> > When the device is probed the arch_setup_dma_ops() function is called.
> > It sets the device's dma_ops and the mapping (in
> > __arm_iommu_attach_device()). If probe is deferred,
> > arch_teardown_dma_ops() is called which in turn calls
> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > out immediately as the dma_ops are already set, leaving us with a device
> > bound to IOMMU operations but with no mapping. This oopses later as soon
> > as the kernel tries to map memory for the device through the IOMMU.
> 
> Resetting the dma_ops for arm32 was added in this patch [1], which I
> missed to send in the original series, but now have added to Russell's patch
> tracking system.

Thank you. I fear that won't be enough though.

> [1] https://patchwork.kernel.org/patch/9434105/

Quoting the patch:

> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> ,dma_ops should be cleared in the teardown path. Otherwise
> this causes problem when the probe of device is retried after
> being deferred. The device's iommu structures are cleared
> after EPROBEDEFER error, but on the next try dma_ops will still
> be set to old value, which is not right.
>
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> ---
>   arch/arm/mm/dma-mapping.c | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index ab4f745..a40f03e 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
*dev)
>   	__arm_iommu_detach_device(dev);
>   	arm_iommu_release_mapping(mapping);
> +	set_dma_ops(dev, NULL);
>   }
>   #else

The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().

However, the situation is perhaps more complex. Note the check at the 
beginning of arch_setup_dma_ops():

	/*
	 * Don't override the dma_ops if they have already been set. Ideally
	 * this should be the only location where dma_ops are set, remove this
	 * check when all other callers of set_dma_ops will have disappeared.
	 */
	if (dev->dma_ops)
		return;

If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
them. To be safe you should only set them to NULL if they have been set by 
arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
to add a field to struct dev_archdata to store that information. To avoid 
growing the structure, which is embedded in every struct device, you could 
possibly turn the dma_coherent bool into a bitfield.

@@ -19,7 +19,8 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
+	bool dma_coherent:1;
+	bool dma_ops_setup:1;
 };
 
 struct omap_device;

I haven't checked, however, whether the dma_coherent field would need to be 
accessed atomically, so this might be a bad idea.

Last but not least, a fix must be merged in v4.12, and the sooner the better.

> > I might be missing something obvious, but I don't see how this can
> > work.
> > 
> >>>>> This can be fixed by either:
> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>> 
> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sakari Ailus]

Hi Laurent,

On Tue, May 16, 2017 at 10:17:08AM +0300, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> > On 2017-05-16 03:04, Laurent Pinchart wrote:
> > > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> > >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> > >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> > >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> 
> > >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> > >>>>>> need to be handled separately from the .of_xlate() failures to
> > >>>>>> support deferred probing.
> > >>>>>> 
> > >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> > >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> > >>>>>> having been deferred, or having failed.
> > >>>>>> 
> > >>>>>> The first case occurs when the device tree describes the bus master
> > >>>>>> and IOMMU topology correctly but no device driver exists for the
> > >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> > >>>>>> the caller will configure the device without an IOMMU.
> > >>>>>> 
> > >>>>>> The second and third cases are handled by deferring the probe of the
> > >>>>>> bus master device which will eventually get reprobed after the
> > >>>>>> IOMMU.
> > >>>>>> 
> > >>>>>> The last case is currently handled by deferring the probe of the bus
> > >>>>>> master device as well. A mechanism to either configure the bus
> > >>>>>> master device without an IOMMU or to fail the bus master device probe
> > >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> > >>>>>> good enhancement.
> > >>>>>> 
> > >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> > >>>>>> Signed-off-by: Laurent Pichart
> > >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>>>> 
> > >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> > >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> > >>>>> iommus properties in DT now fail to probe.
> > >>>> 
> > >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> > >>>> ops registered then they should merely defer until we reach the point
> > >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> > >>>> late-probing drivers or post-init module loads to kick the deferred
> > >>>> queue after that point? I did try to find a way to explicitly kick it
> > >>>> from a suitably late initcall, but there didn't seem to be any obvious
> > >>>> public interface - anyone have any suggestions?
> > >>>> 
> > >>>> I think that's more of a general problem with the probe deferral
> > >>>> mechanism itself (I've seen the same thing happen with some of the
> > >>>> CoreSight stuff on Juno due to the number of inter-component
> > >>>> dependencies) rather than any specific fault of this series.
> > >>> 
> > >>> I was thinking of an additional check like below to avoid the
> > >>> situation ?
> > >>> 
> > >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > >>> From: Sricharan R <sricharan@codeaurora.org>
> > >>> Date: Wed, 3 May 2017 13:16:59 +0530
> > >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > >>> 
> > >>> While returning EPROBE_DEFER for iommu masters
> > >>> take in to account of iommu nodes that could be
> > >>> marked in DT as 'status=disabled', in which case
> > >>> simply return NULL and let the master's probe
> > >>> continue rather than deferring.
> > >>> 
> > >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>> ---
> > >>> 
> > >>>  drivers/iommu/of_iommu.c | 1 +
> > >>>  1 file changed, 1 insertion(+)
> > >>> 
> > >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > >>> index 9f44ee8..e6e9bec 100644
> > >>> --- a/drivers/iommu/of_iommu.c
> > >>> +++ b/drivers/iommu/of_iommu.c
> > >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> > >>> device_node *np)
> > >>> 
> > >>>         ops = iommu_ops_from_fwnode(fwnode);
> > >>>         if ((ops && !ops->of_xlate) ||
> > >>> +           !of_device_is_available(iommu_spec->np) ||
> > >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> > >>>                 return NULL;
> > >> 
> > >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> > >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> > >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> > >> and thus will always be considered as absent.
> > >> 
> > >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> > >> have been nice to check existing IOMMU drivers before merging this patch
> > >> series...
> > > 
> > > Please pardon the question, but has this patch series been tested on
> > > ARM32 ?
> > > 
> > > When the device is probed the arch_setup_dma_ops() function is called.
> > > It sets the device's dma_ops and the mapping (in
> > > __arm_iommu_attach_device()). If probe is deferred,
> > > arch_teardown_dma_ops() is called which in turn calls
> > > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > > out immediately as the dma_ops are already set, leaving us with a device
> > > bound to IOMMU operations but with no mapping. This oopses later as soon
> > > as the kernel tries to map memory for the device through the IOMMU.
> > 
> > Resetting the dma_ops for arm32 was added in this patch [1], which I
> > missed to send in the original series, but now have added to Russell's patch
> > tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
> > [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
> > arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> > ,dma_ops should be cleared in the teardown path. Otherwise
> > this causes problem when the probe of device is retried after
> > being deferred. The device's iommu structures are cleared
> > after EPROBEDEFER error, but on the next try dma_ops will still
> > be set to old value, which is not right.
> >
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> > ---
> >   arch/arm/mm/dma-mapping.c | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index ab4f745..a40f03e 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
> >   	__arm_iommu_detach_device(dev);
> >   	arm_iommu_release_mapping(mapping);
> > +	set_dma_ops(dev, NULL);
> >   }
> >   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.

A bool bit field? :-)

I think I'd just use bool for both. I wouldn't expect dma_coherent change
once it has been set before device driver probe though.

If you like a bit field, then I'd propose making it unsigned int.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sakari Ailus]

Hi Laurent,

On Tue, May 16, 2017 at 10:17:08AM +0300, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> > On 2017-05-16 03:04, Laurent Pinchart wrote:
> > > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> > >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> > >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> > >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> 
> > >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> > >>>>>> need to be handled separately from the .of_xlate() failures to
> > >>>>>> support deferred probing.
> > >>>>>> 
> > >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> > >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> > >>>>>> having been deferred, or having failed.
> > >>>>>> 
> > >>>>>> The first case occurs when the device tree describes the bus master
> > >>>>>> and IOMMU topology correctly but no device driver exists for the
> > >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> > >>>>>> the caller will configure the device without an IOMMU.
> > >>>>>> 
> > >>>>>> The second and third cases are handled by deferring the probe of the
> > >>>>>> bus master device which will eventually get reprobed after the
> > >>>>>> IOMMU.
> > >>>>>> 
> > >>>>>> The last case is currently handled by deferring the probe of the bus
> > >>>>>> master device as well. A mechanism to either configure the bus
> > >>>>>> master device without an IOMMU or to fail the bus master device probe
> > >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> > >>>>>> good enhancement.
> > >>>>>> 
> > >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> > >>>>>> Signed-off-by: Laurent Pichart
> > >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>>>> 
> > >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> > >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> > >>>>> iommus properties in DT now fail to probe.
> > >>>> 
> > >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> > >>>> ops registered then they should merely defer until we reach the point
> > >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> > >>>> late-probing drivers or post-init module loads to kick the deferred
> > >>>> queue after that point? I did try to find a way to explicitly kick it
> > >>>> from a suitably late initcall, but there didn't seem to be any obvious
> > >>>> public interface - anyone have any suggestions?
> > >>>> 
> > >>>> I think that's more of a general problem with the probe deferral
> > >>>> mechanism itself (I've seen the same thing happen with some of the
> > >>>> CoreSight stuff on Juno due to the number of inter-component
> > >>>> dependencies) rather than any specific fault of this series.
> > >>> 
> > >>> I was thinking of an additional check like below to avoid the
> > >>> situation ?
> > >>> 
> > >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > >>> From: Sricharan R <sricharan@codeaurora.org>
> > >>> Date: Wed, 3 May 2017 13:16:59 +0530
> > >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > >>> 
> > >>> While returning EPROBE_DEFER for iommu masters
> > >>> take in to account of iommu nodes that could be
> > >>> marked in DT as 'status=disabled', in which case
> > >>> simply return NULL and let the master's probe
> > >>> continue rather than deferring.
> > >>> 
> > >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>> ---
> > >>> 
> > >>>  drivers/iommu/of_iommu.c | 1 +
> > >>>  1 file changed, 1 insertion(+)
> > >>> 
> > >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > >>> index 9f44ee8..e6e9bec 100644
> > >>> --- a/drivers/iommu/of_iommu.c
> > >>> +++ b/drivers/iommu/of_iommu.c
> > >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> > >>> device_node *np)
> > >>> 
> > >>>         ops = iommu_ops_from_fwnode(fwnode);
> > >>>         if ((ops && !ops->of_xlate) ||
> > >>> +           !of_device_is_available(iommu_spec->np) ||
> > >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> > >>>                 return NULL;
> > >> 
> > >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> > >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> > >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> > >> and thus will always be considered as absent.
> > >> 
> > >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> > >> have been nice to check existing IOMMU drivers before merging this patch
> > >> series...
> > > 
> > > Please pardon the question, but has this patch series been tested on
> > > ARM32 ?
> > > 
> > > When the device is probed the arch_setup_dma_ops() function is called.
> > > It sets the device's dma_ops and the mapping (in
> > > __arm_iommu_attach_device()). If probe is deferred,
> > > arch_teardown_dma_ops() is called which in turn calls
> > > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > > out immediately as the dma_ops are already set, leaving us with a device
> > > bound to IOMMU operations but with no mapping. This oopses later as soon
> > > as the kernel tries to map memory for the device through the IOMMU.
> > 
> > Resetting the dma_ops for arm32 was added in this patch [1], which I
> > missed to send in the original series, but now have added to Russell's patch
> > tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
> > [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
> > arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> > ,dma_ops should be cleared in the teardown path. Otherwise
> > this causes problem when the probe of device is retried after
> > being deferred. The device's iommu structures are cleared
> > after EPROBEDEFER error, but on the next try dma_ops will still
> > be set to old value, which is not right.
> >
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> > ---
> >   arch/arm/mm/dma-mapping.c | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index ab4f745..a40f03e 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
> >   	__arm_iommu_detach_device(dev);
> >   	arm_iommu_release_mapping(mapping);
> > +	set_dma_ops(dev, NULL);
> >   }
> >   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.

A bool bit field? :-)

I think I'd just use bool for both. I wouldn't expect dma_coherent change
once it has been set before device driver probe though.

If you like a bit field, then I'd propose making it unsigned int.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sakari Ailus]

Hi Laurent,

On Tue, May 16, 2017 at 10:17:08AM +0300, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> > On 2017-05-16 03:04, Laurent Pinchart wrote:
> > > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> > >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> > >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> > >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> > >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> > >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> 
> > >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> > >>>>>> need to be handled separately from the .of_xlate() failures to
> > >>>>>> support deferred probing.
> > >>>>>> 
> > >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
> > >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
> > >>>>>> having been deferred, or having failed.
> > >>>>>> 
> > >>>>>> The first case occurs when the device tree describes the bus master
> > >>>>>> and IOMMU topology correctly but no device driver exists for the
> > >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
> > >>>>>> the caller will configure the device without an IOMMU.
> > >>>>>> 
> > >>>>>> The second and third cases are handled by deferring the probe of the
> > >>>>>> bus master device which will eventually get reprobed after the
> > >>>>>> IOMMU.
> > >>>>>> 
> > >>>>>> The last case is currently handled by deferring the probe of the bus
> > >>>>>> master device as well. A mechanism to either configure the bus
> > >>>>>> master device without an IOMMU or to fail the bus master device probe
> > >>>>>> depending on whether the IOMMU is optional or mandatory would be a
> > >>>>>> good enhancement.
> > >>>>>> 
> > >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> > >>>>>> Signed-off-by: Laurent Pichart
> > >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> > >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>>>> 
> > >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> > >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> > >>>>> iommus properties in DT now fail to probe.
> > >>>> 
> > >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> > >>>> ops registered then they should merely defer until we reach the point
> > >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
> > >>>> late-probing drivers or post-init module loads to kick the deferred
> > >>>> queue after that point? I did try to find a way to explicitly kick it
> > >>>> from a suitably late initcall, but there didn't seem to be any obvious
> > >>>> public interface - anyone have any suggestions?
> > >>>> 
> > >>>> I think that's more of a general problem with the probe deferral
> > >>>> mechanism itself (I've seen the same thing happen with some of the
> > >>>> CoreSight stuff on Juno due to the number of inter-component
> > >>>> dependencies) rather than any specific fault of this series.
> > >>> 
> > >>> I was thinking of an additional check like below to avoid the
> > >>> situation ?
> > >>> 
> > >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
> > >>> From: Sricharan R <sricharan@codeaurora.org>
> > >>> Date: Wed, 3 May 2017 13:16:59 +0530
> > >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> > >>> 
> > >>> While returning EPROBE_DEFER for iommu masters
> > >>> take in to account of iommu nodes that could be
> > >>> marked in DT as 'status=disabled', in which case
> > >>> simply return NULL and let the master's probe
> > >>> continue rather than deferring.
> > >>> 
> > >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > >>> ---
> > >>> 
> > >>>  drivers/iommu/of_iommu.c | 1 +
> > >>>  1 file changed, 1 insertion(+)
> > >>> 
> > >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> > >>> index 9f44ee8..e6e9bec 100644
> > >>> --- a/drivers/iommu/of_iommu.c
> > >>> +++ b/drivers/iommu/of_iommu.c
> > >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> > >>> device_node *np)
> > >>> 
> > >>>         ops = iommu_ops_from_fwnode(fwnode);
> > >>>         if ((ops && !ops->of_xlate) ||
> > >>> +           !of_device_is_available(iommu_spec->np) ||
> > >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> > >>>                 return NULL;
> > >> 
> > >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> > >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> > >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
> > >> and thus will always be considered as absent.
> > >> 
> > >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> > >> have been nice to check existing IOMMU drivers before merging this patch
> > >> series...
> > > 
> > > Please pardon the question, but has this patch series been tested on
> > > ARM32 ?
> > > 
> > > When the device is probed the arch_setup_dma_ops() function is called.
> > > It sets the device's dma_ops and the mapping (in
> > > __arm_iommu_attach_device()). If probe is deferred,
> > > arch_teardown_dma_ops() is called which in turn calls
> > > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> > > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> > > out immediately as the dma_ops are already set, leaving us with a device
> > > bound to IOMMU operations but with no mapping. This oopses later as soon
> > > as the kernel tries to map memory for the device through the IOMMU.
> > 
> > Resetting the dma_ops for arm32 was added in this patch [1], which I
> > missed to send in the original series, but now have added to Russell's patch
> > tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
> > [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
> > arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> > ,dma_ops should be cleared in the teardown path. Otherwise
> > this causes problem when the probe of device is retried after
> > being deferred. The device's iommu structures are cleared
> > after EPROBEDEFER error, but on the next try dma_ops will still
> > be set to old value, which is not right.
> >
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> > ---
> >   arch/arm/mm/dma-mapping.c | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index ab4f745..a40f03e 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
> >   	__arm_iommu_detach_device(dev);
> >   	arm_iommu_release_mapping(mapping);
> > +	set_dma_ops(dev, NULL);
> >   }
> >   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.

A bool bit field? :-)

I think I'd just use bool for both. I wouldn't expect dma_coherent change
once it has been set before device driver probe though.

If you like a bit field, then I'd propose making it unsigned int.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus at iki.fi	XMPP: sailus at retiisi.org.uk

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Laurent,

On 2017-05-16 12:47, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>>
>> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>> >>>>>> need to be handled separately from the .of_xlate() failures to
>> >>>>>> support deferred probing.
>> >>>>>>
>> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>>>> having been deferred, or having failed.
>> >>>>>>
>> >>>>>> The first case occurs when the device tree describes the bus master
>> >>>>>> and IOMMU topology correctly but no device driver exists for the
>> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>> >>>>>> the caller will configure the device without an IOMMU.
>> >>>>>>
>> >>>>>> The second and third cases are handled by deferring the probe of the
>> >>>>>> bus master device which will eventually get reprobed after the
>> >>>>>> IOMMU.
>> >>>>>>
>> >>>>>> The last case is currently handled by deferring the probe of the bus
>> >>>>>> master device as well. A mechanism to either configure the bus
>> >>>>>> master device without an IOMMU or to fail the bus master device probe
>> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
>> >>>>>> good enhancement.
>> >>>>>>
>> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>>>> Signed-off-by: Laurent Pichart
>> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>>>
>> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>> >>>>> iommus properties in DT now fail to probe.
>> >>>>
>> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>> >>>> ops registered then they should merely defer until we reach the point
>> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>> >>>> late-probing drivers or post-init module loads to kick the deferred
>> >>>> queue after that point? I did try to find a way to explicitly kick it
>> >>>> from a suitably late initcall, but there didn't seem to be any obvious
>> >>>> public interface - anyone have any suggestions?
>> >>>>
>> >>>> I think that's more of a general problem with the probe deferral
>> >>>> mechanism itself (I've seen the same thing happen with some of the
>> >>>> CoreSight stuff on Juno due to the number of inter-component
>> >>>> dependencies) rather than any specific fault of this series.
>> >>>
>> >>> I was thinking of an additional check like below to avoid the
>> >>> situation ?
>> >>>
>> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> >>> From: Sricharan R <sricharan@codeaurora.org>
>> >>> Date: Wed, 3 May 2017 13:16:59 +0530
>> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >>>
>> >>> While returning EPROBE_DEFER for iommu masters
>> >>> take in to account of iommu nodes that could be
>> >>> marked in DT as 'status=disabled', in which case
>> >>> simply return NULL and let the master's probe
>> >>> continue rather than deferring.
>> >>>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>> ---
>> >>>
>> >>>  drivers/iommu/of_iommu.c | 1 +
>> >>>  1 file changed, 1 insertion(+)
>> >>>
>> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> >>> index 9f44ee8..e6e9bec 100644
>> >>> --- a/drivers/iommu/of_iommu.c
>> >>> +++ b/drivers/iommu/of_iommu.c
>> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>> >>> device_node *np)
>> >>>
>> >>>         ops = iommu_ops_from_fwnode(fwnode);
>> >>>         if ((ops && !ops->of_xlate) ||
>> >>> +           !of_device_is_available(iommu_spec->np) ||
>> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >>>                 return NULL;
>> >>
>> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>> >> and thus will always be considered as absent.
>> >>
>> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>> >> have been nice to check existing IOMMU drivers before merging this patch
>> >> series...
>> >
>> > Please pardon the question, but has this patch series been tested on
>> > ARM32 ?
>> >
>> > When the device is probed the arch_setup_dma_ops() function is called.
>> > It sets the device's dma_ops and the mapping (in
>> > __arm_iommu_attach_device()). If probe is deferred,
>> > arch_teardown_dma_ops() is called which in turn calls
>> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>> > out immediately as the dma_ops are already set, leaving us with a device
>> > bound to IOMMU operations but with no mapping. This oopses later as soon
>> > as the kernel tries to map memory for the device through the IOMMU.
>> 
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's 
>> patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>> 
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct 
>> device
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, 
> but
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will 
> override
> them. To be safe you should only set them to NULL if they have been set 
> by
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should 
> probably
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option 
> would be
> to add a field to struct dev_archdata to store that information. To 
> avoid
> growing the structure, which is embedded in every struct device, you 
> could
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
> 
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need 
> to be
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the 
> better.
> 

ho, yet another combination. This seems to be a problem with 
exynos_iommu,
ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with 
its
own custom mapping. They are calling arm_iommu_attach_device from the
add_device callback and that is not always replayed when the reprobe 
happens
and these archs are storing the old mapping data in private structures 
which
might not be cleared in the teardown path. I will post the fix that you 
have
suggested.

Regards,
  Sricharan


>> > I might be missing something obvious, but I don't see how this can
>> > work.
>> >
>> >>>>> This can be fixed by either:
>> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>>>
>> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi Laurent,

On 2017-05-16 12:47, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>>
>> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>> >>>>>> need to be handled separately from the .of_xlate() failures to
>> >>>>>> support deferred probing.
>> >>>>>>
>> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>>>> having been deferred, or having failed.
>> >>>>>>
>> >>>>>> The first case occurs when the device tree describes the bus master
>> >>>>>> and IOMMU topology correctly but no device driver exists for the
>> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>> >>>>>> the caller will configure the device without an IOMMU.
>> >>>>>>
>> >>>>>> The second and third cases are handled by deferring the probe of the
>> >>>>>> bus master device which will eventually get reprobed after the
>> >>>>>> IOMMU.
>> >>>>>>
>> >>>>>> The last case is currently handled by deferring the probe of the bus
>> >>>>>> master device as well. A mechanism to either configure the bus
>> >>>>>> master device without an IOMMU or to fail the bus master device probe
>> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
>> >>>>>> good enhancement.
>> >>>>>>
>> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>>>> Signed-off-by: Laurent Pichart
>> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>>>
>> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>> >>>>> iommus properties in DT now fail to probe.
>> >>>>
>> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>> >>>> ops registered then they should merely defer until we reach the point
>> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>> >>>> late-probing drivers or post-init module loads to kick the deferred
>> >>>> queue after that point? I did try to find a way to explicitly kick it
>> >>>> from a suitably late initcall, but there didn't seem to be any obvious
>> >>>> public interface - anyone have any suggestions?
>> >>>>
>> >>>> I think that's more of a general problem with the probe deferral
>> >>>> mechanism itself (I've seen the same thing happen with some of the
>> >>>> CoreSight stuff on Juno due to the number of inter-component
>> >>>> dependencies) rather than any specific fault of this series.
>> >>>
>> >>> I was thinking of an additional check like below to avoid the
>> >>> situation ?
>> >>>
>> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> >>> From: Sricharan R <sricharan@codeaurora.org>
>> >>> Date: Wed, 3 May 2017 13:16:59 +0530
>> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >>>
>> >>> While returning EPROBE_DEFER for iommu masters
>> >>> take in to account of iommu nodes that could be
>> >>> marked in DT as 'status=disabled', in which case
>> >>> simply return NULL and let the master's probe
>> >>> continue rather than deferring.
>> >>>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>> ---
>> >>>
>> >>>  drivers/iommu/of_iommu.c | 1 +
>> >>>  1 file changed, 1 insertion(+)
>> >>>
>> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> >>> index 9f44ee8..e6e9bec 100644
>> >>> --- a/drivers/iommu/of_iommu.c
>> >>> +++ b/drivers/iommu/of_iommu.c
>> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>> >>> device_node *np)
>> >>>
>> >>>         ops = iommu_ops_from_fwnode(fwnode);
>> >>>         if ((ops && !ops->of_xlate) ||
>> >>> +           !of_device_is_available(iommu_spec->np) ||
>> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >>>                 return NULL;
>> >>
>> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>> >> and thus will always be considered as absent.
>> >>
>> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>> >> have been nice to check existing IOMMU drivers before merging this patch
>> >> series...
>> >
>> > Please pardon the question, but has this patch series been tested on
>> > ARM32 ?
>> >
>> > When the device is probed the arch_setup_dma_ops() function is called.
>> > It sets the device's dma_ops and the mapping (in
>> > __arm_iommu_attach_device()). If probe is deferred,
>> > arch_teardown_dma_ops() is called which in turn calls
>> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>> > out immediately as the dma_ops are already set, leaving us with a device
>> > bound to IOMMU operations but with no mapping. This oopses later as soon
>> > as the kernel tries to map memory for the device through the IOMMU.
>> 
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's 
>> patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>> 
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct 
>> device
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, 
> but
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will 
> override
> them. To be safe you should only set them to NULL if they have been set 
> by
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should 
> probably
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option 
> would be
> to add a field to struct dev_archdata to store that information. To 
> avoid
> growing the structure, which is embedded in every struct device, you 
> could
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
> 
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need 
> to be
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the 
> better.
> 

ho, yet another combination. This seems to be a problem with 
exynos_iommu,
ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with 
its
own custom mapping. They are calling arm_iommu_attach_device from the
add_device callback and that is not always replayed when the reprobe 
happens
and these archs are storing the old mapping data in private structures 
which
might not be cleared in the teardown path. I will post the fix that you 
have
suggested.

Regards,
  Sricharan


>> > I might be missing something obvious, but I don't see how this can
>> > work.
>> >
>> >>>>> This can be fixed by either:
>> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>>>
>> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan at codeaurora.org]

Hi Laurent,

On 2017-05-16 12:47, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>> > On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>> >> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>> >>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>> >>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>> >>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>> >>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>>
>> >>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>> >>>>>> need to be handled separately from the .of_xlate() failures to
>> >>>>>> support deferred probing.
>> >>>>>>
>> >>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>> >>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>> >>>>>> having been deferred, or having failed.
>> >>>>>>
>> >>>>>> The first case occurs when the device tree describes the bus master
>> >>>>>> and IOMMU topology correctly but no device driver exists for the
>> >>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>> >>>>>> the caller will configure the device without an IOMMU.
>> >>>>>>
>> >>>>>> The second and third cases are handled by deferring the probe of the
>> >>>>>> bus master device which will eventually get reprobed after the
>> >>>>>> IOMMU.
>> >>>>>>
>> >>>>>> The last case is currently handled by deferring the probe of the bus
>> >>>>>> master device as well. A mechanism to either configure the bus
>> >>>>>> master device without an IOMMU or to fail the bus master device probe
>> >>>>>> depending on whether the IOMMU is optional or mandatory would be a
>> >>>>>> good enhancement.
>> >>>>>>
>> >>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> >>>>>> Signed-off-by: Laurent Pichart
>> >>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>> >>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>>>>
>> >>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>> >>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>> >>>>> iommus properties in DT now fail to probe.
>> >>>>
>> >>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>> >>>> ops registered then they should merely defer until we reach the point
>> >>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>> >>>> late-probing drivers or post-init module loads to kick the deferred
>> >>>> queue after that point? I did try to find a way to explicitly kick it
>> >>>> from a suitably late initcall, but there didn't seem to be any obvious
>> >>>> public interface - anyone have any suggestions?
>> >>>>
>> >>>> I think that's more of a general problem with the probe deferral
>> >>>> mechanism itself (I've seen the same thing happen with some of the
>> >>>> CoreSight stuff on Juno due to the number of inter-component
>> >>>> dependencies) rather than any specific fault of this series.
>> >>>
>> >>> I was thinking of an additional check like below to avoid the
>> >>> situation ?
>> >>>
>> >>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> >>> From: Sricharan R <sricharan@codeaurora.org>
>> >>> Date: Wed, 3 May 2017 13:16:59 +0530
>> >>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>> >>>
>> >>> While returning EPROBE_DEFER for iommu masters
>> >>> take in to account of iommu nodes that could be
>> >>> marked in DT as 'status=disabled', in which case
>> >>> simply return NULL and let the master's probe
>> >>> continue rather than deferring.
>> >>>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>> ---
>> >>>
>> >>>  drivers/iommu/of_iommu.c | 1 +
>> >>>  1 file changed, 1 insertion(+)
>> >>>
>> >>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> >>> index 9f44ee8..e6e9bec 100644
>> >>> --- a/drivers/iommu/of_iommu.c
>> >>> +++ b/drivers/iommu/of_iommu.c
>> >>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>> >>> device_node *np)
>> >>>
>> >>>         ops = iommu_ops_from_fwnode(fwnode);
>> >>>         if ((ops && !ops->of_xlate) ||
>> >>> +           !of_device_is_available(iommu_spec->np) ||
>> >>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>> >>>                 return NULL;
>> >>
>> >> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>> >> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>> >> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>> >> and thus will always be considered as absent.
>> >>
>> >> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>> >> have been nice to check existing IOMMU drivers before merging this patch
>> >> series...
>> >
>> > Please pardon the question, but has this patch series been tested on
>> > ARM32 ?
>> >
>> > When the device is probed the arch_setup_dma_ops() function is called.
>> > It sets the device's dma_ops and the mapping (in
>> > __arm_iommu_attach_device()). If probe is deferred,
>> > arch_teardown_dma_ops() is called which in turn calls
>> > arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>> > dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>> > out immediately as the dma_ops are already set, leaving us with a device
>> > bound to IOMMU operations but with no mapping. This oopses later as soon
>> > as the kernel tries to map memory for the device through the IOMMU.
>> 
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's 
>> patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>> 
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>> 
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct 
>> device
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, 
> but
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will 
> override
> them. To be safe you should only set them to NULL if they have been set 
> by
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should 
> probably
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by
> arm_iommu_attach_device() and not arch_teardown_dma_ops(). One option 
> would be
> to add a field to struct dev_archdata to store that information. To 
> avoid
> growing the structure, which is embedded in every struct device, you 
> could
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
> 
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need 
> to be
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the 
> better.
> 

ho, yet another combination. This seems to be a problem with 
exynos_iommu,
ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with 
its
own custom mapping. They are calling arm_iommu_attach_device from the
add_device callback and that is not always replayed when the reprobe 
happens
and these archs are storing the old mapping data in private structures 
which
might not be cleared in the teardown path. I will post the fix that you 
have
suggested.

Regards,
  Sricharan


>> > I might be missing something obvious, but I don't see how this can
>> > work.
>> >
>> >>>>> This can be fixed by either:
>> >>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>> >>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>> >>>>>
>> >>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>> >>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Laurent,

On 16/05/17 08:17, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>>>>>>
>>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>>>>>>>> need to be handled separately from the .of_xlate() failures to
>>>>>>>> support deferred probing.
>>>>>>>>
>>>>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>>>>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>>>>>>>> having been deferred, or having failed.
>>>>>>>>
>>>>>>>> The first case occurs when the device tree describes the bus master
>>>>>>>> and IOMMU topology correctly but no device driver exists for the
>>>>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>>>>>>>> the caller will configure the device without an IOMMU.
>>>>>>>>
>>>>>>>> The second and third cases are handled by deferring the probe of the
>>>>>>>> bus master device which will eventually get reprobed after the
>>>>>>>> IOMMU.
>>>>>>>>
>>>>>>>> The last case is currently handled by deferring the probe of the bus
>>>>>>>> master device as well. A mechanism to either configure the bus
>>>>>>>> master device without an IOMMU or to fail the bus master device probe
>>>>>>>> depending on whether the IOMMU is optional or mandatory would be a
>>>>>>>> good enhancement.
>>>>>>>>
>>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>>>>>> Signed-off-by: Laurent Pichart
>>>>>>>> <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>>>>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>>>>>
>>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>>>>>>> iommus properties in DT now fail to probe.
>>>>>>
>>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>>>>>> ops registered then they should merely defer until we reach the point
>>>>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>>>>>> late-probing drivers or post-init module loads to kick the deferred
>>>>>> queue after that point? I did try to find a way to explicitly kick it
>>>>>> from a suitably late initcall, but there didn't seem to be any obvious
>>>>>> public interface - anyone have any suggestions?
>>>>>>
>>>>>> I think that's more of a general problem with the probe deferral
>>>>>> mechanism itself (I've seen the same thing happen with some of the
>>>>>> CoreSight stuff on Juno due to the number of inter-component
>>>>>> dependencies) rather than any specific fault of this series.
>>>>>
>>>>> I was thinking of an additional check like below to avoid the
>>>>> situation ?
>>>>>
>>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>>>>> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>>> Date: Wed, 3 May 2017 13:16:59 +0530
>>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>>>>
>>>>> While returning EPROBE_DEFER for iommu masters
>>>>> take in to account of iommu nodes that could be
>>>>> marked in DT as 'status=disabled', in which case
>>>>> simply return NULL and let the master's probe
>>>>> continue rather than deferring.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>>> ---
>>>>>
>>>>>  drivers/iommu/of_iommu.c | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>>>> index 9f44ee8..e6e9bec 100644
>>>>> --- a/drivers/iommu/of_iommu.c
>>>>> +++ b/drivers/iommu/of_iommu.c
>>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>>>>> device_node *np)
>>>>>
>>>>>         ops = iommu_ops_from_fwnode(fwnode);
>>>>>         if ((ops && !ops->of_xlate) ||
>>>>> +           !of_device_is_available(iommu_spec->np) ||
>>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>>>>                 return NULL;
>>>>
>>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>>>> and thus will always be considered as absent.
>>>>
>>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>>>> have been nice to check existing IOMMU drivers before merging this patch
>>>> series...
>>>
>>> Please pardon the question, but has this patch series been tested on
>>> ARM32 ?
>>>
>>> When the device is probed the arch_setup_dma_ops() function is called.
>>> It sets the device's dma_ops and the mapping (in
>>> __arm_iommu_attach_device()). If probe is deferred,
>>> arch_teardown_dma_ops() is called which in turn calls
>>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>>> out immediately as the dma_ops are already set, leaving us with a device
>>> bound to IOMMU operations but with no mapping. This oopses later as soon
>>> as the kernel tries to map memory for the device through the IOMMU.
>>
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>>
>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops().

Under what circumstances is that an issue? We'll only be tearing down
the DMA ops when unbinding the driver, and I think it would be erroneous
to expect the device to retain much state after that. Everything else
would be set up from scratch again if it get reprobed later, so why not
the DMA ops?

Robin.

> One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the better.
> 
>>> I might be missing something obvious, but I don't see how this can
>>> work.
>>>
>>>>>>> This can be fixed by either:
>>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>>>>>>
>>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.
> 

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Laurent,

On 16/05/17 08:17, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>>
>>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>>>>>>>> need to be handled separately from the .of_xlate() failures to
>>>>>>>> support deferred probing.
>>>>>>>>
>>>>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>>>>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>>>>>>>> having been deferred, or having failed.
>>>>>>>>
>>>>>>>> The first case occurs when the device tree describes the bus master
>>>>>>>> and IOMMU topology correctly but no device driver exists for the
>>>>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>>>>>>>> the caller will configure the device without an IOMMU.
>>>>>>>>
>>>>>>>> The second and third cases are handled by deferring the probe of the
>>>>>>>> bus master device which will eventually get reprobed after the
>>>>>>>> IOMMU.
>>>>>>>>
>>>>>>>> The last case is currently handled by deferring the probe of the bus
>>>>>>>> master device as well. A mechanism to either configure the bus
>>>>>>>> master device without an IOMMU or to fail the bus master device probe
>>>>>>>> depending on whether the IOMMU is optional or mandatory would be a
>>>>>>>> good enhancement.
>>>>>>>>
>>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>>>>> Signed-off-by: Laurent Pichart
>>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>>>>>>> iommus properties in DT now fail to probe.
>>>>>>
>>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>>>>>> ops registered then they should merely defer until we reach the point
>>>>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>>>>>> late-probing drivers or post-init module loads to kick the deferred
>>>>>> queue after that point? I did try to find a way to explicitly kick it
>>>>>> from a suitably late initcall, but there didn't seem to be any obvious
>>>>>> public interface - anyone have any suggestions?
>>>>>>
>>>>>> I think that's more of a general problem with the probe deferral
>>>>>> mechanism itself (I've seen the same thing happen with some of the
>>>>>> CoreSight stuff on Juno due to the number of inter-component
>>>>>> dependencies) rather than any specific fault of this series.
>>>>>
>>>>> I was thinking of an additional check like below to avoid the
>>>>> situation ?
>>>>>
>>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>> Date: Wed, 3 May 2017 13:16:59 +0530
>>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>>>>
>>>>> While returning EPROBE_DEFER for iommu masters
>>>>> take in to account of iommu nodes that could be
>>>>> marked in DT as 'status=disabled', in which case
>>>>> simply return NULL and let the master's probe
>>>>> continue rather than deferring.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> ---
>>>>>
>>>>>  drivers/iommu/of_iommu.c | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>>>> index 9f44ee8..e6e9bec 100644
>>>>> --- a/drivers/iommu/of_iommu.c
>>>>> +++ b/drivers/iommu/of_iommu.c
>>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>>>>> device_node *np)
>>>>>
>>>>>         ops = iommu_ops_from_fwnode(fwnode);
>>>>>         if ((ops && !ops->of_xlate) ||
>>>>> +           !of_device_is_available(iommu_spec->np) ||
>>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>>>>                 return NULL;
>>>>
>>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>>>> and thus will always be considered as absent.
>>>>
>>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>>>> have been nice to check existing IOMMU drivers before merging this patch
>>>> series...
>>>
>>> Please pardon the question, but has this patch series been tested on
>>> ARM32 ?
>>>
>>> When the device is probed the arch_setup_dma_ops() function is called.
>>> It sets the device's dma_ops and the mapping (in
>>> __arm_iommu_attach_device()). If probe is deferred,
>>> arch_teardown_dma_ops() is called which in turn calls
>>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>>> out immediately as the dma_ops are already set, leaving us with a device
>>> bound to IOMMU operations but with no mapping. This oopses later as soon
>>> as the kernel tries to map memory for the device through the IOMMU.
>>
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops().

Under what circumstances is that an issue? We'll only be tearing down
the DMA ops when unbinding the driver, and I think it would be erroneous
to expect the device to retain much state after that. Everything else
would be set up from scratch again if it get reprobed later, so why not
the DMA ops?

Robin.

> One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the better.
> 
>>> I might be missing something obvious, but I don't see how this can
>>> work.
>>>
>>>>>>> This can be fixed by either:
>>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>>>>>>
>>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.
> 

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Laurent,

On 16/05/17 08:17, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>>
>>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>>>>>>>> need to be handled separately from the .of_xlate() failures to
>>>>>>>> support deferred probing.
>>>>>>>>
>>>>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>>>>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>>>>>>>> having been deferred, or having failed.
>>>>>>>>
>>>>>>>> The first case occurs when the device tree describes the bus master
>>>>>>>> and IOMMU topology correctly but no device driver exists for the
>>>>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>>>>>>>> the caller will configure the device without an IOMMU.
>>>>>>>>
>>>>>>>> The second and third cases are handled by deferring the probe of the
>>>>>>>> bus master device which will eventually get reprobed after the
>>>>>>>> IOMMU.
>>>>>>>>
>>>>>>>> The last case is currently handled by deferring the probe of the bus
>>>>>>>> master device as well. A mechanism to either configure the bus
>>>>>>>> master device without an IOMMU or to fail the bus master device probe
>>>>>>>> depending on whether the IOMMU is optional or mandatory would be a
>>>>>>>> good enhancement.
>>>>>>>>
>>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>>>>> Signed-off-by: Laurent Pichart
>>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>>>>>>> iommus properties in DT now fail to probe.
>>>>>>
>>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>>>>>> ops registered then they should merely defer until we reach the point
>>>>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>>>>>> late-probing drivers or post-init module loads to kick the deferred
>>>>>> queue after that point? I did try to find a way to explicitly kick it
>>>>>> from a suitably late initcall, but there didn't seem to be any obvious
>>>>>> public interface - anyone have any suggestions?
>>>>>>
>>>>>> I think that's more of a general problem with the probe deferral
>>>>>> mechanism itself (I've seen the same thing happen with some of the
>>>>>> CoreSight stuff on Juno due to the number of inter-component
>>>>>> dependencies) rather than any specific fault of this series.
>>>>>
>>>>> I was thinking of an additional check like below to avoid the
>>>>> situation ?
>>>>>
>>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>> Date: Wed, 3 May 2017 13:16:59 +0530
>>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>>>>
>>>>> While returning EPROBE_DEFER for iommu masters
>>>>> take in to account of iommu nodes that could be
>>>>> marked in DT as 'status=disabled', in which case
>>>>> simply return NULL and let the master's probe
>>>>> continue rather than deferring.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> ---
>>>>>
>>>>>  drivers/iommu/of_iommu.c | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>>>> index 9f44ee8..e6e9bec 100644
>>>>> --- a/drivers/iommu/of_iommu.c
>>>>> +++ b/drivers/iommu/of_iommu.c
>>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>>>>> device_node *np)
>>>>>
>>>>>         ops = iommu_ops_from_fwnode(fwnode);
>>>>>         if ((ops && !ops->of_xlate) ||
>>>>> +           !of_device_is_available(iommu_spec->np) ||
>>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>>>>                 return NULL;
>>>>
>>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>>>> and thus will always be considered as absent.
>>>>
>>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>>>> have been nice to check existing IOMMU drivers before merging this patch
>>>> series...
>>>
>>> Please pardon the question, but has this patch series been tested on
>>> ARM32 ?
>>>
>>> When the device is probed the arch_setup_dma_ops() function is called.
>>> It sets the device's dma_ops and the mapping (in
>>> __arm_iommu_attach_device()). If probe is deferred,
>>> arch_teardown_dma_ops() is called which in turn calls
>>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>>> out immediately as the dma_ops are already set, leaving us with a device
>>> bound to IOMMU operations but with no mapping. This oopses later as soon
>>> as the kernel tries to map memory for the device through the IOMMU.
>>
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops().

Under what circumstances is that an issue? We'll only be tearing down
the DMA ops when unbinding the driver, and I think it would be erroneous
to expect the device to retain much state after that. Everything else
would be set up from scratch again if it get reprobed later, so why not
the DMA ops?

Robin.

> One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the better.
> 
>>> I might be missing something obvious, but I don't see how this can
>>> work.
>>>
>>>>>>> This can be fixed by either:
>>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>>>>>>
>>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

Hi Laurent,

On 16/05/17 08:17, Laurent Pinchart wrote:
> Hi Sricharan,
> 
> On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
>> On 2017-05-16 03:04, Laurent Pinchart wrote:
>>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
>>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
>>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
>>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>>
>>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
>>>>>>>> need to be handled separately from the .of_xlate() failures to
>>>>>>>> support deferred probing.
>>>>>>>>
>>>>>>>> The lack of a registered IOMMU can be caused by the lack of a driver
>>>>>>>> for the IOMMU, the IOMMU device probe not having been performed yet,
>>>>>>>> having been deferred, or having failed.
>>>>>>>>
>>>>>>>> The first case occurs when the device tree describes the bus master
>>>>>>>> and IOMMU topology correctly but no device driver exists for the
>>>>>>>> IOMMU yet or the device driver has not been compiled in. Return NULL,
>>>>>>>> the caller will configure the device without an IOMMU.
>>>>>>>>
>>>>>>>> The second and third cases are handled by deferring the probe of the
>>>>>>>> bus master device which will eventually get reprobed after the
>>>>>>>> IOMMU.
>>>>>>>>
>>>>>>>> The last case is currently handled by deferring the probe of the bus
>>>>>>>> master device as well. A mechanism to either configure the bus
>>>>>>>> master device without an IOMMU or to fail the bus master device probe
>>>>>>>> depending on whether the IOMMU is optional or mandatory would be a
>>>>>>>> good enhancement.
>>>>>>>>
>>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>>>>> Signed-off-by: Laurent Pichart
>>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
>>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
>>>>>>> iommus properties in DT now fail to probe.
>>>>>>
>>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
>>>>>> ops registered then they should merely defer until we reach the point
>>>>>> of giving up and ignoring the IOMMU. Is it just that you have no other
>>>>>> late-probing drivers or post-init module loads to kick the deferred
>>>>>> queue after that point? I did try to find a way to explicitly kick it
>>>>>> from a suitably late initcall, but there didn't seem to be any obvious
>>>>>> public interface - anyone have any suggestions?
>>>>>>
>>>>>> I think that's more of a general problem with the probe deferral
>>>>>> mechanism itself (I've seen the same thing happen with some of the
>>>>>> CoreSight stuff on Juno due to the number of inter-component
>>>>>> dependencies) rather than any specific fault of this series.
>>>>>
>>>>> I was thinking of an additional check like below to avoid the
>>>>> situation ?
>>>>>
>>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>> Date: Wed, 3 May 2017 13:16:59 +0530
>>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>>>>
>>>>> While returning EPROBE_DEFER for iommu masters
>>>>> take in to account of iommu nodes that could be
>>>>> marked in DT as 'status=disabled', in which case
>>>>> simply return NULL and let the master's probe
>>>>> continue rather than deferring.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> ---
>>>>>
>>>>>  drivers/iommu/of_iommu.c | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>>>>> index 9f44ee8..e6e9bec 100644
>>>>> --- a/drivers/iommu/of_iommu.c
>>>>> +++ b/drivers/iommu/of_iommu.c
>>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
>>>>> device_node *np)
>>>>>
>>>>>         ops = iommu_ops_from_fwnode(fwnode);
>>>>>         if ((ops && !ops->of_xlate) ||
>>>>> +           !of_device_is_available(iommu_spec->np) ||
>>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>>>>                 return NULL;
>>>>
>>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
>>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
>>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't IOMMU_OF_DECLARE(),
>>>> and thus will always be considered as absent.
>>>>
>>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
>>>> have been nice to check existing IOMMU drivers before merging this patch
>>>> series...
>>>
>>> Please pardon the question, but has this patch series been tested on
>>> ARM32 ?
>>>
>>> When the device is probed the arch_setup_dma_ops() function is called.
>>> It sets the device's dma_ops and the mapping (in
>>> __arm_iommu_attach_device()). If probe is deferred,
>>> arch_teardown_dma_ops() is called which in turn calls
>>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
>>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
>>> out immediately as the dma_ops are already set, leaving us with a device
>>> bound to IOMMU operations but with no mapping. This oopses later as soon
>>> as the kernel tries to map memory for the device through the IOMMU.
>>
>> Resetting the dma_ops for arm32 was added in this patch [1], which I
>> missed to send in the original series, but now have added to Russell's patch
>> tracking system.
> 
> Thank you. I fear that won't be enough though.
> 
>> [1] https://patchwork.kernel.org/patch/9434105/
> 
> Quoting the patch:
> 
>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> ,dma_ops should be cleared in the teardown path. Otherwise
>> this causes problem when the probe of device is retried after
>> being deferred. The device's iommu structures are cleared
>> after EPROBEDEFER error, but on the next try dma_ops will still
>> be set to old value, which is not right.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> ---
>>   arch/arm/mm/dma-mapping.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index ab4f745..a40f03e 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct device 
> *dev)
>>   	__arm_iommu_detach_device(dev);
>>   	arm_iommu_release_mapping(mapping);
>> +	set_dma_ops(dev, NULL);
>>   }
>>   #else
> 
> The subject mentions arch_teardown_dma_ops(), which I think is correct, but 
> the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> 
> However, the situation is perhaps more complex. Note the check at the 
> beginning of arch_setup_dma_ops():
> 
> 	/*
> 	 * Don't override the dma_ops if they have already been set. Ideally
> 	 * this should be the only location where dma_ops are set, remove this
> 	 * check when all other callers of set_dma_ops will have disappeared.
> 	 */
> 	if (dev->dma_ops)
> 		return;
> 
> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or 
> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will override 
> them. To be safe you should only set them to NULL if they have been set by 
> arch_setup_dma_ops(). More than that, arch_teardown_dma_ops() should probably 
> not call arm_teardown_iommu_dma_ops() at all if the dma_ops were set by 
> arm_iommu_attach_device() and not arch_teardown_dma_ops().

Under what circumstances is that an issue? We'll only be tearing down
the DMA ops when unbinding the driver, and I think it would be erroneous
to expect the device to retain much state after that. Everything else
would be set up from scratch again if it get reprobed later, so why not
the DMA ops?

Robin.

> One option would be 
> to add a field to struct dev_archdata to store that information. To avoid 
> growing the structure, which is embedded in every struct device, you could 
> possibly turn the dma_coherent bool into a bitfield.
> 
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	bool dma_coherent:1;
> +	bool dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> 
> I haven't checked, however, whether the dma_coherent field would need to be 
> accessed atomically, so this might be a bad idea.
> 
> Last but not least, a fix must be merged in v4.12, and the sooner the better.
> 
>>> I might be missing something obvious, but I don't see how this can
>>> work.
>>>
>>>>>>> This can be fixed by either:
>>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
>>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
>>>>>>>
>>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
>>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.
> 

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:10:03 sricharan@codeaurora.org wrote:
> On 2017-05-16 12:47, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> >> On 2017-05-16 03:04, Laurent Pinchart wrote:
> >>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> 
> >>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>>>> support deferred probing.
> >>>>>>>> 
> >>>>>>>> The lack of a registered IOMMU can be caused by the lack of a
> >>>>>>>> driver for the IOMMU, the IOMMU device probe not having been
> >>>>>>>> performed yet, having been deferred, or having failed.
> >>>>>>>> 
> >>>>>>>> The first case occurs when the device tree describes the bus
> >>>>>>>> master and IOMMU topology correctly but no device driver exists for
> >>>>>>>> the IOMMU yet or the device driver has not been compiled in. Return
> >>>>>>>> NULL, the caller will configure the device without an IOMMU.
> >>>>>>>> 
> >>>>>>>> The second and third cases are handled by deferring the probe of
> >>>>>>>> the bus master device which will eventually get reprobed after the
> >>>>>>>> IOMMU.
> >>>>>>>> 
> >>>>>>>> The last case is currently handled by deferring the probe of the
> >>>>>>>> bus master device as well. A mechanism to either configure the bus
> >>>>>>>> master device without an IOMMU or to fail the bus master device
> >>>>>>>> probe depending on whether the IOMMU is optional or mandatory would
> >>>>>>>> be a good enhancement.
> >>>>>>>> 
> >>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>>>> Signed-off-by: Laurent Pichart
> >>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>>>> 
> >>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>>>> iommus properties in DT now fail to probe.
> >>>>>> 
> >>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>>>> ops registered then they should merely defer until we reach the
> >>>>>> point of giving up and ignoring the IOMMU. Is it just that you have
> >>>>>> no other late-probing drivers or post-init module loads to kick the
> >>>>>> deferred queue after that point? I did try to find a way to
> >>>>>> explicitly kick it from a suitably late initcall, but there didn't
> >>>>>> seem to be any obvious public interface - anyone have any
> >>>>>> suggestions?
> >>>>>> 
> >>>>>> I think that's more of a general problem with the probe deferral
> >>>>>> mechanism itself (I've seen the same thing happen with some of the
> >>>>>> CoreSight stuff on Juno due to the number of inter-component
> >>>>>> dependencies) rather than any specific fault of this series.
> >>>>> 
> >>>>> I was thinking of an additional check like below to avoid the
> >>>>> situation ?
> >>>>> 
> >>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00
> >>>>> 2001
> >>>>> From: Sricharan R <sricharan@codeaurora.org>
> >>>>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>>>> 
> >>>>> While returning EPROBE_DEFER for iommu masters
> >>>>> take in to account of iommu nodes that could be
> >>>>> marked in DT as 'status=disabled', in which case
> >>>>> simply return NULL and let the master's probe
> >>>>> continue rather than deferring.
> >>>>> 
> >>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> ---
> >>>>> 
> >>>>>  drivers/iommu/of_iommu.c | 1 +
> >>>>>  1 file changed, 1 insertion(+)
> >>>>> 
> >>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>>>> index 9f44ee8..e6e9bec 100644
> >>>>> --- a/drivers/iommu/of_iommu.c
> >>>>> +++ b/drivers/iommu/of_iommu.c
> >>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>>>> device_node *np)
> >>>>> 
> >>>>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>>>         if ((ops && !ops->of_xlate) ||
> >>>>> +           !of_device_is_available(iommu_spec->np) ||
> >>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>>>                 return NULL;
> >>>> 
> >>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't
> >>>> IOMMU_OF_DECLARE(),
> >>>> and thus will always be considered as absent.
> >>>> 
> >>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >>>> have been nice to check existing IOMMU drivers before merging this
> >>>> patch series...
> >>> 
> >>> Please pardon the question, but has this patch series been tested on
> >>> ARM32 ?
> >>> 
> >>> When the device is probed the arch_setup_dma_ops() function is called.
> >>> It sets the device's dma_ops and the mapping (in
> >>> __arm_iommu_attach_device()). If probe is deferred,
> >>> arch_teardown_dma_ops() is called which in turn calls
> >>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> >>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> >>> out immediately as the dma_ops are already set, leaving us with a
> >>> device bound to IOMMU operations but with no mapping. This oopses later
> >>> as soon as the kernel tries to map memory for the device through the
> >>> IOMMU.
> >> 
> >> Resetting the dma_ops for arm32 was added in this patch [1], which I
> >> missed to send in the original series, but now have added to Russell's
> >> patch tracking system.
> > 
> > Thank you. I fear that won't be enough though.
> > 
> >> [1] https://patchwork.kernel.org/patch/9434105/
> > 
> > Quoting the patch:
> >
> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > 
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops(). One option would be to add a field to struct
> > dev_archdata to store that information. To avoid growing the structure,
> > which is embedded in every struct device, you could possibly turn the
> > dma_coherent bool into a bitfield.
> > 
> > @@ -19,7 +19,8 @@ struct dev_archdata {
> >  #ifdef CONFIG_XEN
> >  	const struct dma_map_ops *dev_dma_ops;
> >  #endif
> > -	bool dma_coherent;
> > +	bool dma_coherent:1;
> > +	bool dma_ops_setup:1;
> >  };
> >  
> >  struct omap_device;
> > 
> > I haven't checked, however, whether the dma_coherent field would need
> > to be accessed atomically, so this might be a bad idea.
> > 
> > Last but not least, a fix must be merged in v4.12, and the sooner the
> > better.
> 
> ho, yet another combination. This seems to be a problem with exynos_iommu,
> ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with its
> own custom mapping. They are calling arm_iommu_attach_device from the
> add_device callback and that is not always replayed when the reprobe happens
> and these archs are storing the old mapping data in private structures which
> might not be cleared in the teardown path.

Yes, I know, it's messy :-/ There's a handful of non-IOMMU drivers calling 
arm_iommu_attach_device() directly too. All these should be fixed, but in the 
meantime, let's try not to break them.

> I will post the fix that you have suggested.

Thank you. You might want to use an unsigned int bitfield instead of a bool 
bitfield as Sakari suggested. It would be nice to check the code setting the 
dma_coherent field to make sure there will be no race with code setting the 
new dma_ops_setup field (which might not be the best name, feel free to rename 
it).

I have successfully test the patch, let me know if there's anything else I can 
do to help.

> >>> I might be missing something obvious, but I don't see how this can
> >>> work.
> >>> 
> >>>>>>> This can be fixed by either:
> >>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>>>> 
> >>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:10:03 sricharan@codeaurora.org wrote:
> On 2017-05-16 12:47, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> >> On 2017-05-16 03:04, Laurent Pinchart wrote:
> >>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> 
> >>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>>>> support deferred probing.
> >>>>>>>> 
> >>>>>>>> The lack of a registered IOMMU can be caused by the lack of a
> >>>>>>>> driver for the IOMMU, the IOMMU device probe not having been
> >>>>>>>> performed yet, having been deferred, or having failed.
> >>>>>>>> 
> >>>>>>>> The first case occurs when the device tree describes the bus
> >>>>>>>> master and IOMMU topology correctly but no device driver exists for
> >>>>>>>> the IOMMU yet or the device driver has not been compiled in. Return
> >>>>>>>> NULL, the caller will configure the device without an IOMMU.
> >>>>>>>> 
> >>>>>>>> The second and third cases are handled by deferring the probe of
> >>>>>>>> the bus master device which will eventually get reprobed after the
> >>>>>>>> IOMMU.
> >>>>>>>> 
> >>>>>>>> The last case is currently handled by deferring the probe of the
> >>>>>>>> bus master device as well. A mechanism to either configure the bus
> >>>>>>>> master device without an IOMMU or to fail the bus master device
> >>>>>>>> probe depending on whether the IOMMU is optional or mandatory would
> >>>>>>>> be a good enhancement.
> >>>>>>>> 
> >>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>>>> Signed-off-by: Laurent Pichart
> >>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>>>> 
> >>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>>>> iommus properties in DT now fail to probe.
> >>>>>> 
> >>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>>>> ops registered then they should merely defer until we reach the
> >>>>>> point of giving up and ignoring the IOMMU. Is it just that you have
> >>>>>> no other late-probing drivers or post-init module loads to kick the
> >>>>>> deferred queue after that point? I did try to find a way to
> >>>>>> explicitly kick it from a suitably late initcall, but there didn't
> >>>>>> seem to be any obvious public interface - anyone have any
> >>>>>> suggestions?
> >>>>>> 
> >>>>>> I think that's more of a general problem with the probe deferral
> >>>>>> mechanism itself (I've seen the same thing happen with some of the
> >>>>>> CoreSight stuff on Juno due to the number of inter-component
> >>>>>> dependencies) rather than any specific fault of this series.
> >>>>> 
> >>>>> I was thinking of an additional check like below to avoid the
> >>>>> situation ?
> >>>>> 
> >>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00
> >>>>> 2001
> >>>>> From: Sricharan R <sricharan@codeaurora.org>
> >>>>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>>>> 
> >>>>> While returning EPROBE_DEFER for iommu masters
> >>>>> take in to account of iommu nodes that could be
> >>>>> marked in DT as 'status=disabled', in which case
> >>>>> simply return NULL and let the master's probe
> >>>>> continue rather than deferring.
> >>>>> 
> >>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> ---
> >>>>> 
> >>>>>  drivers/iommu/of_iommu.c | 1 +
> >>>>>  1 file changed, 1 insertion(+)
> >>>>> 
> >>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>>>> index 9f44ee8..e6e9bec 100644
> >>>>> --- a/drivers/iommu/of_iommu.c
> >>>>> +++ b/drivers/iommu/of_iommu.c
> >>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>>>> device_node *np)
> >>>>> 
> >>>>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>>>         if ((ops && !ops->of_xlate) ||
> >>>>> +           !of_device_is_available(iommu_spec->np) ||
> >>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>>>                 return NULL;
> >>>> 
> >>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't
> >>>> IOMMU_OF_DECLARE(),
> >>>> and thus will always be considered as absent.
> >>>> 
> >>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >>>> have been nice to check existing IOMMU drivers before merging this
> >>>> patch series...
> >>> 
> >>> Please pardon the question, but has this patch series been tested on
> >>> ARM32 ?
> >>> 
> >>> When the device is probed the arch_setup_dma_ops() function is called.
> >>> It sets the device's dma_ops and the mapping (in
> >>> __arm_iommu_attach_device()). If probe is deferred,
> >>> arch_teardown_dma_ops() is called which in turn calls
> >>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> >>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> >>> out immediately as the dma_ops are already set, leaving us with a
> >>> device bound to IOMMU operations but with no mapping. This oopses later
> >>> as soon as the kernel tries to map memory for the device through the
> >>> IOMMU.
> >> 
> >> Resetting the dma_ops for arm32 was added in this patch [1], which I
> >> missed to send in the original series, but now have added to Russell's
> >> patch tracking system.
> > 
> > Thank you. I fear that won't be enough though.
> > 
> >> [1] https://patchwork.kernel.org/patch/9434105/
> > 
> > Quoting the patch:
> >
> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > 
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops(). One option would be to add a field to struct
> > dev_archdata to store that information. To avoid growing the structure,
> > which is embedded in every struct device, you could possibly turn the
> > dma_coherent bool into a bitfield.
> > 
> > @@ -19,7 +19,8 @@ struct dev_archdata {
> >  #ifdef CONFIG_XEN
> >  	const struct dma_map_ops *dev_dma_ops;
> >  #endif
> > -	bool dma_coherent;
> > +	bool dma_coherent:1;
> > +	bool dma_ops_setup:1;
> >  };
> >  
> >  struct omap_device;
> > 
> > I haven't checked, however, whether the dma_coherent field would need
> > to be accessed atomically, so this might be a bad idea.
> > 
> > Last but not least, a fix must be merged in v4.12, and the sooner the
> > better.
> 
> ho, yet another combination. This seems to be a problem with exynos_iommu,
> ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with its
> own custom mapping. They are calling arm_iommu_attach_device from the
> add_device callback and that is not always replayed when the reprobe happens
> and these archs are storing the old mapping data in private structures which
> might not be cleared in the teardown path.

Yes, I know, it's messy :-/ There's a handful of non-IOMMU drivers calling 
arm_iommu_attach_device() directly too. All these should be fixed, but in the 
meantime, let's try not to break them.

> I will post the fix that you have suggested.

Thank you. You might want to use an unsigned int bitfield instead of a bool 
bitfield as Sakari suggested. It would be nice to check the code setting the 
dma_coherent field to make sure there will be no race with code setting the 
new dma_ops_setup field (which might not be the best name, feel free to rename 
it).

I have successfully test the patch, let me know if there's anything else I can 
do to help.

> >>> I might be missing something obvious, but I don't see how this can
> >>> work.
> >>> 
> >>>>>>> This can be fixed by either:
> >>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>>>> 
> >>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:10:03 sricharan at codeaurora.org wrote:
> On 2017-05-16 12:47, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> >> On 2017-05-16 03:04, Laurent Pinchart wrote:
> >>> On Monday 15 May 2017 23:37:16 Laurent Pinchart wrote:
> >>>> On Wednesday 03 May 2017 15:54:59 Sricharan R wrote:
> >>>>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
> >>>>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
> >>>>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R wrote:
> >>>>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> 
> >>>>>>>> Failures to look up an IOMMU when parsing the DT iommus property
> >>>>>>>> need to be handled separately from the .of_xlate() failures to
> >>>>>>>> support deferred probing.
> >>>>>>>> 
> >>>>>>>> The lack of a registered IOMMU can be caused by the lack of a
> >>>>>>>> driver for the IOMMU, the IOMMU device probe not having been
> >>>>>>>> performed yet, having been deferred, or having failed.
> >>>>>>>> 
> >>>>>>>> The first case occurs when the device tree describes the bus
> >>>>>>>> master and IOMMU topology correctly but no device driver exists for
> >>>>>>>> the IOMMU yet or the device driver has not been compiled in. Return
> >>>>>>>> NULL, the caller will configure the device without an IOMMU.
> >>>>>>>> 
> >>>>>>>> The second and third cases are handled by deferring the probe of
> >>>>>>>> the bus master device which will eventually get reprobed after the
> >>>>>>>> IOMMU.
> >>>>>>>> 
> >>>>>>>> The last case is currently handled by deferring the probe of the
> >>>>>>>> bus master device as well. A mechanism to either configure the bus
> >>>>>>>> master device without an IOMMU or to fail the bus master device
> >>>>>>>> probe depending on whether the IOMMU is optional or mandatory would
> >>>>>>>> be a good enhancement.
> >>>>>>>> 
> >>>>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> >>>>>>>> Signed-off-by: Laurent Pichart
> >>>>>>>> <laurent.pinchart+renesas@ideasonboard.com>
> >>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>>>> 
> >>>>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
> >>>>>>> As the IOMMU nodes in DT are not yet enabled, all devices having
> >>>>>>> iommus properties in DT now fail to probe.
> >>>>>> 
> >>>>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no
> >>>>>> ops registered then they should merely defer until we reach the
> >>>>>> point of giving up and ignoring the IOMMU. Is it just that you have
> >>>>>> no other late-probing drivers or post-init module loads to kick the
> >>>>>> deferred queue after that point? I did try to find a way to
> >>>>>> explicitly kick it from a suitably late initcall, but there didn't
> >>>>>> seem to be any obvious public interface - anyone have any
> >>>>>> suggestions?
> >>>>>> 
> >>>>>> I think that's more of a general problem with the probe deferral
> >>>>>> mechanism itself (I've seen the same thing happen with some of the
> >>>>>> CoreSight stuff on Juno due to the number of inter-component
> >>>>>> dependencies) rather than any specific fault of this series.
> >>>>> 
> >>>>> I was thinking of an additional check like below to avoid the
> >>>>> situation ?
> >>>>> 
> >>>>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00
> >>>>> 2001
> >>>>> From: Sricharan R <sricharan@codeaurora.org>
> >>>>> Date: Wed, 3 May 2017 13:16:59 +0530
> >>>>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
> >>>>> 
> >>>>> While returning EPROBE_DEFER for iommu masters
> >>>>> take in to account of iommu nodes that could be
> >>>>> marked in DT as 'status=disabled', in which case
> >>>>> simply return NULL and let the master's probe
> >>>>> continue rather than deferring.
> >>>>> 
> >>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>>> ---
> >>>>> 
> >>>>>  drivers/iommu/of_iommu.c | 1 +
> >>>>>  1 file changed, 1 insertion(+)
> >>>>> 
> >>>>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> >>>>> index 9f44ee8..e6e9bec 100644
> >>>>> --- a/drivers/iommu/of_iommu.c
> >>>>> +++ b/drivers/iommu/of_iommu.c
> >>>>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct
> >>>>> device_node *np)
> >>>>> 
> >>>>>         ops = iommu_ops_from_fwnode(fwnode);
> >>>>>         if ((ops && !ops->of_xlate) ||
> >>>>> +           !of_device_is_available(iommu_spec->np) ||
> >>>>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
> >>>>>                 return NULL;
> >>>> 
> >>>> This looks good to me, but won't be enough. The ipmmu-vmsa driver in
> >>>> v4.12-rc1 doesn't call iommu_device_register() and thus won't be found
> >>>> by iommu_ops_from_fwnode(). Furthermore, it doesn't
> >>>> IOMMU_OF_DECLARE(),
> >>>> and thus will always be considered as absent.
> >>>> 
> >>>> I agree that the ipmmu-vmsa driver needs to be fixed, but it would
> >>>> have been nice to check existing IOMMU drivers before merging this
> >>>> patch series...
> >>> 
> >>> Please pardon the question, but has this patch series been tested on
> >>> ARM32 ?
> >>> 
> >>> When the device is probed the arch_setup_dma_ops() function is called.
> >>> It sets the device's dma_ops and the mapping (in
> >>> __arm_iommu_attach_device()). If probe is deferred,
> >>> arch_teardown_dma_ops() is called which in turn calls
> >>> arch_teardown_dma_ops(). This removes the mapping but doesn't touch the
> >>> dma_ops. The next time the device is probed, arch_setup_dma_ops() bails
> >>> out immediately as the dma_ops are already set, leaving us with a
> >>> device bound to IOMMU operations but with no mapping. This oopses later
> >>> as soon as the kernel tries to map memory for the device through the
> >>> IOMMU.
> >> 
> >> Resetting the dma_ops for arm32 was added in this patch [1], which I
> >> missed to send in the original series, but now have added to Russell's
> >> patch tracking system.
> > 
> > Thank you. I fear that won't be enough though.
> > 
> >> [1] https://patchwork.kernel.org/patch/9434105/
> > 
> > Quoting the patch:
> >
> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > 
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops(). One option would be to add a field to struct
> > dev_archdata to store that information. To avoid growing the structure,
> > which is embedded in every struct device, you could possibly turn the
> > dma_coherent bool into a bitfield.
> > 
> > @@ -19,7 +19,8 @@ struct dev_archdata {
> >  #ifdef CONFIG_XEN
> >  	const struct dma_map_ops *dev_dma_ops;
> >  #endif
> > -	bool dma_coherent;
> > +	bool dma_coherent:1;
> > +	bool dma_ops_setup:1;
> >  };
> >  
> >  struct omap_device;
> > 
> > I haven't checked, however, whether the dma_coherent field would need
> > to be accessed atomically, so this might be a bad idea.
> > 
> > Last but not least, a fix must be merged in v4.12, and the sooner the
> > better.
> 
> ho, yet another combination. This seems to be a problem with exynos_iommu,
> ipmmu-vmsa, mtk_iommu_v1 which calls the arm_iommu_attach_device with its
> own custom mapping. They are calling arm_iommu_attach_device from the
> add_device callback and that is not always replayed when the reprobe happens
> and these archs are storing the old mapping data in private structures which
> might not be cleared in the teardown path.

Yes, I know, it's messy :-/ There's a handful of non-IOMMU drivers calling 
arm_iommu_attach_device() directly too. All these should be fixed, but in the 
meantime, let's try not to break them.

> I will post the fix that you have suggested.

Thank you. You might want to use an unsigned int bitfield instead of a bool 
bitfield as Sakari suggested. It would be nice to check the code setting the 
dma_coherent field to make sure there will be no race with code setting the 
new dma_ops_setup field (which might not be the best name, feel free to rename 
it).

I have successfully test the patch, let me know if there's anything else I can 
do to help.

> >>> I might be missing something obvious, but I don't see how this can
> >>> work.
> >>> 
> >>>>>>> This can be fixed by either:
> >>>>>>>   - Disabling CONFIG_IPMMU_VMSA, or
> >>>>>>>   - Reverting commit 7b07cbefb68d486f (but keeping "int ret = 0;").
> >>>>>>> 
> >>>>>>> Note that this was a bit hard to investigate, as R-Car Gen3 support
> >>>>>>> wasn't upstreamed yet, so bisection pointed to a merge commit.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Robin,

On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> On 16/05/17 08:17, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:

[snip]

> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops().
> 
> Under what circumstances is that an issue? We'll only be tearing down
> the DMA ops when unbinding the driver,

Or when deferring probe.

> and I think it would be erroneous to expect the device to retain much state
> after that. Everything else would be set up from scratch again if it get
> reprobed later, so why not the DMA ops?

Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
If you look at the patch that added the above warning, its commit message 
states

commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Date:   Fri May 15 02:00:02 2015 +0300

    arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
    
    The arch_setup_dma_ops() function is in charge of setting dma_ops with a
    call to set_dma_ops(). set_dma_ops() is also called from
    
    - highbank and mvebu bus notifiers
    - dmabounce (to be replaced with swiotlb)
    - arm_iommu_attach_device
    
    (arm_iommu_attach_device is itself called from IOMMU and bus master
    device drivers)
    
    To allow the arch_setup_dma_ops() call to be moved from device add time
    to device probe time we must ensure that dma_ops already setup by any of
    the above callers will not be overriden.
    
    Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
    of_xlate and taking care of highbank and mvebu, the workaround should be
    removed.

I'm concerned about potentially breaking these if we unconditionally remove 
the DMA ops and mapping.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Robin,

On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> On 16/05/17 08:17, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:

[snip]

> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops().
> 
> Under what circumstances is that an issue? We'll only be tearing down
> the DMA ops when unbinding the driver,

Or when deferring probe.

> and I think it would be erroneous to expect the device to retain much state
> after that. Everything else would be set up from scratch again if it get
> reprobed later, so why not the DMA ops?

Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
If you look at the patch that added the above warning, its commit message 
states

commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Date:   Fri May 15 02:00:02 2015 +0300

    arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
    
    The arch_setup_dma_ops() function is in charge of setting dma_ops with a
    call to set_dma_ops(). set_dma_ops() is also called from
    
    - highbank and mvebu bus notifiers
    - dmabounce (to be replaced with swiotlb)
    - arm_iommu_attach_device
    
    (arm_iommu_attach_device is itself called from IOMMU and bus master
    device drivers)
    
    To allow the arch_setup_dma_ops() call to be moved from device add time
    to device probe time we must ensure that dma_ops already setup by any of
    the above callers will not be overriden.
    
    Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
    of_xlate and taking care of highbank and mvebu, the workaround should be
    removed.

I'm concerned about potentially breaking these if we unconditionally remove 
the DMA ops and mapping.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Robin,

On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> On 16/05/17 08:17, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:

[snip]

> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >> ,dma_ops should be cleared in the teardown path. Otherwise
> >> this causes problem when the probe of device is retried after
> >> being deferred. The device's iommu structures are cleared
> >> after EPROBEDEFER error, but on the next try dma_ops will still
> >> be set to old value, which is not right.
> >> 
> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >> ---
> >> 
> >>   arch/arm/mm/dma-mapping.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >> index ab4f745..a40f03e 100644
> >> --- a/arch/arm/mm/dma-mapping.c
> >> +++ b/arch/arm/mm/dma-mapping.c
> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >> device *dev)
> > 
> >>   	__arm_iommu_detach_device(dev);
> >>   	arm_iommu_release_mapping(mapping);
> >> +	set_dma_ops(dev, NULL);
> >>   }
> >>   #else
> > 
> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
> > 
> > However, the situation is perhaps more complex. Note the check at the
> > beginning of arch_setup_dma_ops():
> > 	/*
> > 	 * Don't override the dma_ops if they have already been set. Ideally
> > 	 * this should be the only location where dma_ops are set, remove this
> > 	 * check when all other callers of set_dma_ops will have disappeared.
> > 	 */
> > 	if (dev->dma_ops)
> > 		return;
> > 
> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> > override them. To be safe you should only set them to NULL if they have
> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
> > should probably not call arm_teardown_iommu_dma_ops() at all if the
> > dma_ops were set by arm_iommu_attach_device() and not
> > arch_teardown_dma_ops().
> 
> Under what circumstances is that an issue? We'll only be tearing down
> the DMA ops when unbinding the driver,

Or when deferring probe.

> and I think it would be erroneous to expect the device to retain much state
> after that. Everything else would be set up from scratch again if it get
> reprobed later, so why not the DMA ops?

Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
If you look at the patch that added the above warning, its commit message 
states

commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Date:   Fri May 15 02:00:02 2015 +0300

    arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
    
    The arch_setup_dma_ops() function is in charge of setting dma_ops with a
    call to set_dma_ops(). set_dma_ops() is also called from
    
    - highbank and mvebu bus notifiers
    - dmabounce (to be replaced with swiotlb)
    - arm_iommu_attach_device
    
    (arm_iommu_attach_device is itself called from IOMMU and bus master
    device drivers)
    
    To allow the arch_setup_dma_ops() call to be moved from device add time
    to device probe time we must ensure that dma_ops already setup by any of
    the above callers will not be overriden.
    
    Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
    of_xlate and taking care of highbank and mvebu, the workaround should be
    removed.

I'm concerned about potentially breaking these if we unconditionally remove 
the DMA ops and mapping.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan-sgV2jX0FEOL9JmXXK+q4OQ]

Hi,

On 2017-05-16 19:40, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>> > On Tuesday 16 May 2017 07:53:57 sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org wrote:
> 
> [snip]
> 
>> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> >> ,dma_ops should be cleared in the teardown path. Otherwise
>> >> this causes problem when the probe of device is retried after
>> >> being deferred. The device's iommu structures are cleared
>> >> after EPROBEDEFER error, but on the next try dma_ops will still
>> >> be set to old value, which is not right.
>> >>
>> >> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>> >> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>> >> ---
>> >>
>> >>   arch/arm/mm/dma-mapping.c | 1 +
>> >>   1 file changed, 1 insertion(+)
>> >>
>> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> >> index ab4f745..a40f03e 100644
>> >> --- a/arch/arm/mm/dma-mapping.c
>> >> +++ b/arch/arm/mm/dma-mapping.c
>> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>> >> device *dev)
>> >
>> >>   	__arm_iommu_detach_device(dev);
>> >>   	arm_iommu_release_mapping(mapping);
>> >> +	set_dma_ops(dev, NULL);
>> >>   }
>> >>   #else
>> >
>> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
>> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>> >
>> > However, the situation is perhaps more complex. Note the check at the
>> > beginning of arch_setup_dma_ops():
>> > 	/*
>> > 	 * Don't override the dma_ops if they have already been set. Ideally
>> > 	 * this should be the only location where dma_ops are set, remove this
>> > 	 * check when all other callers of set_dma_ops will have disappeared.
>> > 	 */
>> > 	if (dev->dma_ops)
>> > 		return;
>> >
>> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>> > override them. To be safe you should only set them to NULL if they have
>> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>> > should probably not call arm_teardown_iommu_dma_ops() at all if the
>> > dma_ops were set by arm_iommu_attach_device() and not
>> > arch_teardown_dma_ops().
>> 
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much 
>> state
>> after that. Everything else would be set up from scratch again if it 
>> get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than 
> arch_setup_dma_ops().
> If you look at the patch that added the above warning, its commit 
> message
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> 
>     The arch_setup_dma_ops() function is in charge of setting dma_ops 
> with a
>     call to set_dma_ops(). set_dma_ops() is also called from
> 
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
> 
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
> 
>     To allow the arch_setup_dma_ops() call to be moved from device add 
> time
>     to device probe time we must ensure that dma_ops already setup by 
> any of
>     the above callers will not be overriden.
> 
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers 
> to
>     of_xlate and taking care of highbank and mvebu, the workaround 
> should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally 
> remove
> the DMA ops and mapping.

arch_teardown_dma_ops does nothing if there is
no mapping (not behind iommu), dma_ops without iommu is ok.
But when the arm_iommu_create_mapping/arm_iommu_attach_device
was called previously in the iommu driver, after we teardown,
that path in the iommu driver which called those functions is not
replayed.

Regards,

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi,

On 2017-05-16 19:40, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> 
> [snip]
> 
>> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> >> ,dma_ops should be cleared in the teardown path. Otherwise
>> >> this causes problem when the probe of device is retried after
>> >> being deferred. The device's iommu structures are cleared
>> >> after EPROBEDEFER error, but on the next try dma_ops will still
>> >> be set to old value, which is not right.
>> >>
>> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> >> ---
>> >>
>> >>   arch/arm/mm/dma-mapping.c | 1 +
>> >>   1 file changed, 1 insertion(+)
>> >>
>> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> >> index ab4f745..a40f03e 100644
>> >> --- a/arch/arm/mm/dma-mapping.c
>> >> +++ b/arch/arm/mm/dma-mapping.c
>> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>> >> device *dev)
>> >
>> >>   	__arm_iommu_detach_device(dev);
>> >>   	arm_iommu_release_mapping(mapping);
>> >> +	set_dma_ops(dev, NULL);
>> >>   }
>> >>   #else
>> >
>> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
>> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>> >
>> > However, the situation is perhaps more complex. Note the check at the
>> > beginning of arch_setup_dma_ops():
>> > 	/*
>> > 	 * Don't override the dma_ops if they have already been set. Ideally
>> > 	 * this should be the only location where dma_ops are set, remove this
>> > 	 * check when all other callers of set_dma_ops will have disappeared.
>> > 	 */
>> > 	if (dev->dma_ops)
>> > 		return;
>> >
>> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>> > override them. To be safe you should only set them to NULL if they have
>> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>> > should probably not call arm_teardown_iommu_dma_ops() at all if the
>> > dma_ops were set by arm_iommu_attach_device() and not
>> > arch_teardown_dma_ops().
>> 
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much 
>> state
>> after that. Everything else would be set up from scratch again if it 
>> get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than 
> arch_setup_dma_ops().
> If you look at the patch that added the above warning, its commit 
> message
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> 
>     The arch_setup_dma_ops() function is in charge of setting dma_ops 
> with a
>     call to set_dma_ops(). set_dma_ops() is also called from
> 
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
> 
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
> 
>     To allow the arch_setup_dma_ops() call to be moved from device add 
> time
>     to device probe time we must ensure that dma_ops already setup by 
> any of
>     the above callers will not be overriden.
> 
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers 
> to
>     of_xlate and taking care of highbank and mvebu, the workaround 
> should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally 
> remove
> the DMA ops and mapping.

arch_teardown_dma_ops does nothing if there is
no mapping (not behind iommu), dma_ops without iommu is ok.
But when the arm_iommu_create_mapping/arm_iommu_attach_device
was called previously in the iommu driver, after we teardown,
that path in the iommu driver which called those functions is not
replayed.

Regards,

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan]

Hi,

On 2017-05-16 19:40, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> 
> [snip]
> 
>> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> >> ,dma_ops should be cleared in the teardown path. Otherwise
>> >> this causes problem when the probe of device is retried after
>> >> being deferred. The device's iommu structures are cleared
>> >> after EPROBEDEFER error, but on the next try dma_ops will still
>> >> be set to old value, which is not right.
>> >>
>> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> >> ---
>> >>
>> >>   arch/arm/mm/dma-mapping.c | 1 +
>> >>   1 file changed, 1 insertion(+)
>> >>
>> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> >> index ab4f745..a40f03e 100644
>> >> --- a/arch/arm/mm/dma-mapping.c
>> >> +++ b/arch/arm/mm/dma-mapping.c
>> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>> >> device *dev)
>> >
>> >>   	__arm_iommu_detach_device(dev);
>> >>   	arm_iommu_release_mapping(mapping);
>> >> +	set_dma_ops(dev, NULL);
>> >>   }
>> >>   #else
>> >
>> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
>> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>> >
>> > However, the situation is perhaps more complex. Note the check at the
>> > beginning of arch_setup_dma_ops():
>> > 	/*
>> > 	 * Don't override the dma_ops if they have already been set. Ideally
>> > 	 * this should be the only location where dma_ops are set, remove this
>> > 	 * check when all other callers of set_dma_ops will have disappeared.
>> > 	 */
>> > 	if (dev->dma_ops)
>> > 		return;
>> >
>> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>> > override them. To be safe you should only set them to NULL if they have
>> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>> > should probably not call arm_teardown_iommu_dma_ops() at all if the
>> > dma_ops were set by arm_iommu_attach_device() and not
>> > arch_teardown_dma_ops().
>> 
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much 
>> state
>> after that. Everything else would be set up from scratch again if it 
>> get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than 
> arch_setup_dma_ops().
> If you look at the patch that added the above warning, its commit 
> message
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> 
>     The arch_setup_dma_ops() function is in charge of setting dma_ops 
> with a
>     call to set_dma_ops(). set_dma_ops() is also called from
> 
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
> 
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
> 
>     To allow the arch_setup_dma_ops() call to be moved from device add 
> time
>     to device probe time we must ensure that dma_ops already setup by 
> any of
>     the above callers will not be overriden.
> 
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers 
> to
>     of_xlate and taking care of highbank and mvebu, the workaround 
> should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally 
> remove
> the DMA ops and mapping.

arch_teardown_dma_ops does nothing if there is
no mapping (not behind iommu), dma_ops without iommu is ok.
But when the arm_iommu_create_mapping/arm_iommu_attach_device
was called previously in the iommu driver, after we teardown,
that path in the iommu driver which called those functions is not
replayed.

Regards,

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[sricharan at codeaurora.org]

Hi,

On 2017-05-16 19:40, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>> > On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> 
> [snip]
> 
>> >> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>> >> ,dma_ops should be cleared in the teardown path. Otherwise
>> >> this causes problem when the probe of device is retried after
>> >> being deferred. The device's iommu structures are cleared
>> >> after EPROBEDEFER error, but on the next try dma_ops will still
>> >> be set to old value, which is not right.
>> >>
>> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>> >> ---
>> >>
>> >>   arch/arm/mm/dma-mapping.c | 1 +
>> >>   1 file changed, 1 insertion(+)
>> >>
>> >> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> >> index ab4f745..a40f03e 100644
>> >> --- a/arch/arm/mm/dma-mapping.c
>> >> +++ b/arch/arm/mm/dma-mapping.c
>> >> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>> >> device *dev)
>> >
>> >>   	__arm_iommu_detach_device(dev);
>> >>   	arm_iommu_release_mapping(mapping);
>> >> +	set_dma_ops(dev, NULL);
>> >>   }
>> >>   #else
>> >
>> > The subject mentions arch_teardown_dma_ops(), which I think is correct,
>> > but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>> >
>> > However, the situation is perhaps more complex. Note the check at the
>> > beginning of arch_setup_dma_ops():
>> > 	/*
>> > 	 * Don't override the dma_ops if they have already been set. Ideally
>> > 	 * this should be the only location where dma_ops are set, remove this
>> > 	 * check when all other callers of set_dma_ops will have disappeared.
>> > 	 */
>> > 	if (dev->dma_ops)
>> > 		return;
>> >
>> > If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>> > arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>> > override them. To be safe you should only set them to NULL if they have
>> > been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>> > should probably not call arm_teardown_iommu_dma_ops() at all if the
>> > dma_ops were set by arm_iommu_attach_device() and not
>> > arch_teardown_dma_ops().
>> 
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much 
>> state
>> after that. Everything else would be set up from scratch again if it 
>> get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than 
> arch_setup_dma_ops().
> If you look at the patch that added the above warning, its commit 
> message
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> 
>     The arch_setup_dma_ops() function is in charge of setting dma_ops 
> with a
>     call to set_dma_ops(). set_dma_ops() is also called from
> 
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
> 
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
> 
>     To allow the arch_setup_dma_ops() call to be moved from device add 
> time
>     to device probe time we must ensure that dma_ops already setup by 
> any of
>     the above callers will not be overriden.
> 
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers 
> to
>     of_xlate and taking care of highbank and mvebu, the workaround 
> should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally 
> remove
> the DMA ops and mapping.

arch_teardown_dma_ops does nothing if there is
no mapping (not behind iommu), dma_ops without iommu is ok.
But when the arm_iommu_create_mapping/arm_iommu_attach_device
was called previously in the iommu driver, after we teardown,
that path in the iommu driver which called those functions is not
replayed.

Regards,

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:59:01 sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org wrote:
> On 2017-05-16 19:40, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> >> On 16/05/17 08:17, Laurent Pinchart wrote:
> >> > On Tuesday 16 May 2017 07:53:57 sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org wrote:
> > [snip]
> > 
> >>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>>> dma_ops should be cleared in the teardown path. Otherwise
> >>>> this causes problem when the probe of device is retried after
> >>>> being deferred. The device's iommu structures are cleared
> >>>> after EPROBEDEFER error, but on the next try dma_ops will still
> >>>> be set to old value, which is not right.
> >>>> 
> >>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >>>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> >>>> ---
> >>>> 
> >>>>   arch/arm/mm/dma-mapping.c | 1 +
> >>>>   1 file changed, 1 insertion(+)
> >>>> 
> >>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >>>> index ab4f745..a40f03e 100644
> >>>> --- a/arch/arm/mm/dma-mapping.c
> >>>> +++ b/arch/arm/mm/dma-mapping.c
> >>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >>>> device *dev)
> >>>>
> >>>>   	__arm_iommu_detach_device(dev);
> >>>>   	arm_iommu_release_mapping(mapping);
> >>>> +	set_dma_ops(dev, NULL);
> >>>>   }
> >>>>   #else
> >>> 
> >>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
> >>> but the patch adds the set_dma_ops() call to
> >>> arm_teardown_iommu_dma_ops().
> >>> 
> >>> However, the situation is perhaps more complex. Note the check at the
> >>> beginning of arch_setup_dma_ops():
> >>> 
> >>> 	/*
> >>> 	 * Don't override the dma_ops if they have already been set. Ideally
> >>> 	 * this should be the only location where dma_ops are set, remove this
> >>> 	 * check when all other callers of set_dma_ops will have disappeared.
> >>> 	 */
> >>> 	if (dev->dma_ops)
> >>> 		return;
> >>> 
> >>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> >>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> >>> override them. To be safe you should only set them to NULL if they have
> >>> been set by arch_setup_dma_ops(). More than that,
> >>> arch_teardown_dma_ops()
> >>> should probably not call arm_teardown_iommu_dma_ops() at all if the
> >>> dma_ops were set by arm_iommu_attach_device() and not
> >>> arch_teardown_dma_ops().
> >> 
> >> Under what circumstances is that an issue? We'll only be tearing down
> >> the DMA ops when unbinding the driver,
> > 
> > Or when deferring probe.
> > 
> >> and I think it would be erroneous to expect the device to retain much
> >> state after that. Everything else would be set up from scratch again if
> >> it get reprobed later, so why not the DMA ops?
> > 
> > Because the DMA ops might have been set elsewhere than
> > arch_setup_dma_ops(). If you look at the patch that added the above
> > warning, its commit message states
> > 
> > commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> > Author: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> > Date:   Fri May 15 02:00:02 2015 +0300
> > 
> >     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> >     
> >     The arch_setup_dma_ops() function is in charge of setting dma_ops
> >     with a call to set_dma_ops(). set_dma_ops() is also called from
> >     
> >     - highbank and mvebu bus notifiers
> >     - dmabounce (to be replaced with swiotlb)
> >     - arm_iommu_attach_device
> >     
> >     (arm_iommu_attach_device is itself called from IOMMU and bus master
> >     device drivers)
> >     
> >     To allow the arch_setup_dma_ops() call to be moved from device add
> >     time to device probe time we must ensure that dma_ops already setup by
> >     any of the above callers will not be overriden.
> >     
> >     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers
> >     to of_xlate and taking care of highbank and mvebu, the workaround
> >     should be removed.
> > 
> > I'm concerned about potentially breaking these if we unconditionally
> > remove the DMA ops and mapping.
> 
> arch_teardown_dma_ops does nothing if there is no mapping (not behind
> iommu), dma_ops without iommu is ok. But when the
> arm_iommu_create_mapping/arm_iommu_attach_device was called previously in
> the iommu driver, after we teardown, that path in the iommu driver which
> called those functions is not replayed.

I've had a look at the code in more details, and I'm not sure how we've 
reached the current situation (as I haven't followed the multiple versions of 
this patch series due to lack of time), but it's a very big mess.

arch_setup_dma_ops() is currently not the only way to create a mapping and 
attach it to a device. Not only that, but with the current ARM32 dma-mapping 
implementation, it can't be.

arch_setup_dma_ops() will create a separate mapping for every device. That's 
certainly fine when every device has its own IOMMU instance (or at least its 
own TLB in a shared IOMMU), but that's far from being true in all cases. 
Renesas R-Car Gen2 hardware, for instance, share IOMMUs and TLBs between 
multiple devices. That's why the ipmmu-vmsa driver creates the mapping 
manually with arm_iommu_create_mapping() and attaches it to the device with 
arm_iommu_attach_device(). We need to keep supporting this mechanism until the 
ARM32 dma-mapping API is brought in line with the ARM64 implementation that 
lets the IOMMU core manage the IOVA address space.

We're unfortunately far from that, so, for the time being, we need to keep 
supporting arm_iommu_attach_device(). How that can work with deferred probing, 
I will let you figure it out, but we need to at least fix the breakage for 
v4.12.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:59:01 sricharan@codeaurora.org wrote:
> On 2017-05-16 19:40, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> >> On 16/05/17 08:17, Laurent Pinchart wrote:
> >> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> > [snip]
> > 
> >>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>>> dma_ops should be cleared in the teardown path. Otherwise
> >>>> this causes problem when the probe of device is retried after
> >>>> being deferred. The device's iommu structures are cleared
> >>>> after EPROBEDEFER error, but on the next try dma_ops will still
> >>>> be set to old value, which is not right.
> >>>> 
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >>>> ---
> >>>> 
> >>>>   arch/arm/mm/dma-mapping.c | 1 +
> >>>>   1 file changed, 1 insertion(+)
> >>>> 
> >>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >>>> index ab4f745..a40f03e 100644
> >>>> --- a/arch/arm/mm/dma-mapping.c
> >>>> +++ b/arch/arm/mm/dma-mapping.c
> >>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >>>> device *dev)
> >>>>
> >>>>   	__arm_iommu_detach_device(dev);
> >>>>   	arm_iommu_release_mapping(mapping);
> >>>> +	set_dma_ops(dev, NULL);
> >>>>   }
> >>>>   #else
> >>> 
> >>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
> >>> but the patch adds the set_dma_ops() call to
> >>> arm_teardown_iommu_dma_ops().
> >>> 
> >>> However, the situation is perhaps more complex. Note the check at the
> >>> beginning of arch_setup_dma_ops():
> >>> 
> >>> 	/*
> >>> 	 * Don't override the dma_ops if they have already been set. Ideally
> >>> 	 * this should be the only location where dma_ops are set, remove this
> >>> 	 * check when all other callers of set_dma_ops will have disappeared.
> >>> 	 */
> >>> 	if (dev->dma_ops)
> >>> 		return;
> >>> 
> >>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> >>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> >>> override them. To be safe you should only set them to NULL if they have
> >>> been set by arch_setup_dma_ops(). More than that,
> >>> arch_teardown_dma_ops()
> >>> should probably not call arm_teardown_iommu_dma_ops() at all if the
> >>> dma_ops were set by arm_iommu_attach_device() and not
> >>> arch_teardown_dma_ops().
> >> 
> >> Under what circumstances is that an issue? We'll only be tearing down
> >> the DMA ops when unbinding the driver,
> > 
> > Or when deferring probe.
> > 
> >> and I think it would be erroneous to expect the device to retain much
> >> state after that. Everything else would be set up from scratch again if
> >> it get reprobed later, so why not the DMA ops?
> > 
> > Because the DMA ops might have been set elsewhere than
> > arch_setup_dma_ops(). If you look at the patch that added the above
> > warning, its commit message states
> > 
> > commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> > Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > Date:   Fri May 15 02:00:02 2015 +0300
> > 
> >     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> >     
> >     The arch_setup_dma_ops() function is in charge of setting dma_ops
> >     with a call to set_dma_ops(). set_dma_ops() is also called from
> >     
> >     - highbank and mvebu bus notifiers
> >     - dmabounce (to be replaced with swiotlb)
> >     - arm_iommu_attach_device
> >     
> >     (arm_iommu_attach_device is itself called from IOMMU and bus master
> >     device drivers)
> >     
> >     To allow the arch_setup_dma_ops() call to be moved from device add
> >     time to device probe time we must ensure that dma_ops already setup by
> >     any of the above callers will not be overriden.
> >     
> >     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers
> >     to of_xlate and taking care of highbank and mvebu, the workaround
> >     should be removed.
> > 
> > I'm concerned about potentially breaking these if we unconditionally
> > remove the DMA ops and mapping.
> 
> arch_teardown_dma_ops does nothing if there is no mapping (not behind
> iommu), dma_ops without iommu is ok. But when the
> arm_iommu_create_mapping/arm_iommu_attach_device was called previously in
> the iommu driver, after we teardown, that path in the iommu driver which
> called those functions is not replayed.

I've had a look at the code in more details, and I'm not sure how we've 
reached the current situation (as I haven't followed the multiple versions of 
this patch series due to lack of time), but it's a very big mess.

arch_setup_dma_ops() is currently not the only way to create a mapping and 
attach it to a device. Not only that, but with the current ARM32 dma-mapping 
implementation, it can't be.

arch_setup_dma_ops() will create a separate mapping for every device. That's 
certainly fine when every device has its own IOMMU instance (or at least its 
own TLB in a shared IOMMU), but that's far from being true in all cases. 
Renesas R-Car Gen2 hardware, for instance, share IOMMUs and TLBs between 
multiple devices. That's why the ipmmu-vmsa driver creates the mapping 
manually with arm_iommu_create_mapping() and attaches it to the device with 
arm_iommu_attach_device(). We need to keep supporting this mechanism until the 
ARM32 dma-mapping API is brought in line with the ARM64 implementation that 
lets the IOMMU core manage the IOVA address space.

We're unfortunately far from that, so, for the time being, we need to keep 
supporting arm_iommu_attach_device(). How that can work with deferred probing, 
I will let you figure it out, but we need to at least fix the breakage for 
v4.12.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:59:01 sricharan@codeaurora.org wrote:
> On 2017-05-16 19:40, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> >> On 16/05/17 08:17, Laurent Pinchart wrote:
> >> > On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> > [snip]
> > 
> >>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>>> dma_ops should be cleared in the teardown path. Otherwise
> >>>> this causes problem when the probe of device is retried after
> >>>> being deferred. The device's iommu structures are cleared
> >>>> after EPROBEDEFER error, but on the next try dma_ops will still
> >>>> be set to old value, which is not right.
> >>>> 
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >>>> ---
> >>>> 
> >>>>   arch/arm/mm/dma-mapping.c | 1 +
> >>>>   1 file changed, 1 insertion(+)
> >>>> 
> >>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >>>> index ab4f745..a40f03e 100644
> >>>> --- a/arch/arm/mm/dma-mapping.c
> >>>> +++ b/arch/arm/mm/dma-mapping.c
> >>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >>>> device *dev)
> >>>>
> >>>>   	__arm_iommu_detach_device(dev);
> >>>>   	arm_iommu_release_mapping(mapping);
> >>>> +	set_dma_ops(dev, NULL);
> >>>>   }
> >>>>   #else
> >>> 
> >>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
> >>> but the patch adds the set_dma_ops() call to
> >>> arm_teardown_iommu_dma_ops().
> >>> 
> >>> However, the situation is perhaps more complex. Note the check at the
> >>> beginning of arch_setup_dma_ops():
> >>> 
> >>> 	/*
> >>> 	 * Don't override the dma_ops if they have already been set. Ideally
> >>> 	 * this should be the only location where dma_ops are set, remove this
> >>> 	 * check when all other callers of set_dma_ops will have disappeared.
> >>> 	 */
> >>> 	if (dev->dma_ops)
> >>> 		return;
> >>> 
> >>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> >>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> >>> override them. To be safe you should only set them to NULL if they have
> >>> been set by arch_setup_dma_ops(). More than that,
> >>> arch_teardown_dma_ops()
> >>> should probably not call arm_teardown_iommu_dma_ops() at all if the
> >>> dma_ops were set by arm_iommu_attach_device() and not
> >>> arch_teardown_dma_ops().
> >> 
> >> Under what circumstances is that an issue? We'll only be tearing down
> >> the DMA ops when unbinding the driver,
> > 
> > Or when deferring probe.
> > 
> >> and I think it would be erroneous to expect the device to retain much
> >> state after that. Everything else would be set up from scratch again if
> >> it get reprobed later, so why not the DMA ops?
> > 
> > Because the DMA ops might have been set elsewhere than
> > arch_setup_dma_ops(). If you look at the patch that added the above
> > warning, its commit message states
> > 
> > commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> > Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > Date:   Fri May 15 02:00:02 2015 +0300
> > 
> >     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> >     
> >     The arch_setup_dma_ops() function is in charge of setting dma_ops
> >     with a call to set_dma_ops(). set_dma_ops() is also called from
> >     
> >     - highbank and mvebu bus notifiers
> >     - dmabounce (to be replaced with swiotlb)
> >     - arm_iommu_attach_device
> >     
> >     (arm_iommu_attach_device is itself called from IOMMU and bus master
> >     device drivers)
> >     
> >     To allow the arch_setup_dma_ops() call to be moved from device add
> >     time to device probe time we must ensure that dma_ops already setup by
> >     any of the above callers will not be overriden.
> >     
> >     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers
> >     to of_xlate and taking care of highbank and mvebu, the workaround
> >     should be removed.
> > 
> > I'm concerned about potentially breaking these if we unconditionally
> > remove the DMA ops and mapping.
> 
> arch_teardown_dma_ops does nothing if there is no mapping (not behind
> iommu), dma_ops without iommu is ok. But when the
> arm_iommu_create_mapping/arm_iommu_attach_device was called previously in
> the iommu driver, after we teardown, that path in the iommu driver which
> called those functions is not replayed.

I've had a look at the code in more details, and I'm not sure how we've 
reached the current situation (as I haven't followed the multiple versions of 
this patch series due to lack of time), but it's a very big mess.

arch_setup_dma_ops() is currently not the only way to create a mapping and 
attach it to a device. Not only that, but with the current ARM32 dma-mapping 
implementation, it can't be.

arch_setup_dma_ops() will create a separate mapping for every device. That's 
certainly fine when every device has its own IOMMU instance (or at least its 
own TLB in a shared IOMMU), but that's far from being true in all cases. 
Renesas R-Car Gen2 hardware, for instance, share IOMMUs and TLBs between 
multiple devices. That's why the ipmmu-vmsa driver creates the mapping 
manually with arm_iommu_create_mapping() and attaches it to the device with 
arm_iommu_attach_device(). We need to keep supporting this mechanism until the 
ARM32 dma-mapping API is brought in line with the ARM64 implementation that 
lets the IOMMU core manage the IOVA address space.

We're unfortunately far from that, so, for the time being, we need to keep 
supporting arm_iommu_attach_device(). How that can work with deferred probing, 
I will let you figure it out, but we need to at least fix the breakage for 
v4.12.

-- 
Regards,

Laurent Pinchart


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Laurent Pinchart]

Hi Sricharan,

On Tuesday 16 May 2017 19:59:01 sricharan at codeaurora.org wrote:
> On 2017-05-16 19:40, Laurent Pinchart wrote:
> > On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
> >> On 16/05/17 08:17, Laurent Pinchart wrote:
> >> > On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> > [snip]
> > 
> >>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>>> dma_ops should be cleared in the teardown path. Otherwise
> >>>> this causes problem when the probe of device is retried after
> >>>> being deferred. The device's iommu structures are cleared
> >>>> after EPROBEDEFER error, but on the next try dma_ops will still
> >>>> be set to old value, which is not right.
> >>>> 
> >>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> >>>> ---
> >>>> 
> >>>>   arch/arm/mm/dma-mapping.c | 1 +
> >>>>   1 file changed, 1 insertion(+)
> >>>> 
> >>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> >>>> index ab4f745..a40f03e 100644
> >>>> --- a/arch/arm/mm/dma-mapping.c
> >>>> +++ b/arch/arm/mm/dma-mapping.c
> >>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
> >>>> device *dev)
> >>>>
> >>>>   	__arm_iommu_detach_device(dev);
> >>>>   	arm_iommu_release_mapping(mapping);
> >>>> +	set_dma_ops(dev, NULL);
> >>>>   }
> >>>>   #else
> >>> 
> >>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
> >>> but the patch adds the set_dma_ops() call to
> >>> arm_teardown_iommu_dma_ops().
> >>> 
> >>> However, the situation is perhaps more complex. Note the check at the
> >>> beginning of arch_setup_dma_ops():
> >>> 
> >>> 	/*
> >>> 	 * Don't override the dma_ops if they have already been set. Ideally
> >>> 	 * this should be the only location where dma_ops are set, remove this
> >>> 	 * check when all other callers of set_dma_ops will have disappeared.
> >>> 	 */
> >>> 	if (dev->dma_ops)
> >>> 		return;
> >>> 
> >>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
> >>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
> >>> override them. To be safe you should only set them to NULL if they have
> >>> been set by arch_setup_dma_ops(). More than that,
> >>> arch_teardown_dma_ops()
> >>> should probably not call arm_teardown_iommu_dma_ops() at all if the
> >>> dma_ops were set by arm_iommu_attach_device() and not
> >>> arch_teardown_dma_ops().
> >> 
> >> Under what circumstances is that an issue? We'll only be tearing down
> >> the DMA ops when unbinding the driver,
> > 
> > Or when deferring probe.
> > 
> >> and I think it would be erroneous to expect the device to retain much
> >> state after that. Everything else would be set up from scratch again if
> >> it get reprobed later, so why not the DMA ops?
> > 
> > Because the DMA ops might have been set elsewhere than
> > arch_setup_dma_ops(). If you look at the patch that added the above
> > warning, its commit message states
> > 
> > commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> > Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > Date:   Fri May 15 02:00:02 2015 +0300
> > 
> >     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
> >     
> >     The arch_setup_dma_ops() function is in charge of setting dma_ops
> >     with a call to set_dma_ops(). set_dma_ops() is also called from
> >     
> >     - highbank and mvebu bus notifiers
> >     - dmabounce (to be replaced with swiotlb)
> >     - arm_iommu_attach_device
> >     
> >     (arm_iommu_attach_device is itself called from IOMMU and bus master
> >     device drivers)
> >     
> >     To allow the arch_setup_dma_ops() call to be moved from device add
> >     time to device probe time we must ensure that dma_ops already setup by
> >     any of the above callers will not be overriden.
> >     
> >     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers
> >     to of_xlate and taking care of highbank and mvebu, the workaround
> >     should be removed.
> > 
> > I'm concerned about potentially breaking these if we unconditionally
> > remove the DMA ops and mapping.
> 
> arch_teardown_dma_ops does nothing if there is no mapping (not behind
> iommu), dma_ops without iommu is ok. But when the
> arm_iommu_create_mapping/arm_iommu_attach_device was called previously in
> the iommu driver, after we teardown, that path in the iommu driver which
> called those functions is not replayed.

I've had a look at the code in more details, and I'm not sure how we've 
reached the current situation (as I haven't followed the multiple versions of 
this patch series due to lack of time), but it's a very big mess.

arch_setup_dma_ops() is currently not the only way to create a mapping and 
attach it to a device. Not only that, but with the current ARM32 dma-mapping 
implementation, it can't be.

arch_setup_dma_ops() will create a separate mapping for every device. That's 
certainly fine when every device has its own IOMMU instance (or at least its 
own TLB in a shared IOMMU), but that's far from being true in all cases. 
Renesas R-Car Gen2 hardware, for instance, share IOMMUs and TLBs between 
multiple devices. That's why the ipmmu-vmsa driver creates the mapping 
manually with arm_iommu_create_mapping() and attaches it to the device with 
arm_iommu_attach_device(). We need to keep supporting this mechanism until the 
ARM32 dma-mapping API is brought in line with the ARM64 implementation that 
lets the IOMMU core manage the IOVA address space.

We're unfortunately far from that, so, for the time being, we need to keep 
supporting arm_iommu_attach_device(). How that can work with deferred probing, 
I will let you figure it out, but we need to at least fix the breakage for 
v4.12.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 16/05/17 15:10, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>>> On Tuesday 16 May 2017 07:53:57 sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org wrote:
> 
> [snip]
> 
>>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>>>> ,dma_ops should be cleared in the teardown path. Otherwise
>>>> this causes problem when the probe of device is retried after
>>>> being deferred. The device's iommu structures are cleared
>>>> after EPROBEDEFER error, but on the next try dma_ops will still
>>>> be set to old value, which is not right.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>>>> ---
>>>>
>>>>   arch/arm/mm/dma-mapping.c | 1 +
>>>>   1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index ab4f745..a40f03e 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>>>> device *dev)
>>>
>>>>   	__arm_iommu_detach_device(dev);
>>>>   	arm_iommu_release_mapping(mapping);
>>>> +	set_dma_ops(dev, NULL);
>>>>   }
>>>>   #else
>>>
>>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
>>> but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>>>
>>> However, the situation is perhaps more complex. Note the check at the
>>> beginning of arch_setup_dma_ops():
>>> 	/*
>>> 	 * Don't override the dma_ops if they have already been set. Ideally
>>> 	 * this should be the only location where dma_ops are set, remove this
>>> 	 * check when all other callers of set_dma_ops will have disappeared.
>>> 	 */
>>> 	if (dev->dma_ops)
>>> 		return;
>>>
>>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>>> override them. To be safe you should only set them to NULL if they have
>>> been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>>> should probably not call arm_teardown_iommu_dma_ops() at all if the
>>> dma_ops were set by arm_iommu_attach_device() and not
>>> arch_teardown_dma_ops().
>>
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much state
>> after that. Everything else would be set up from scratch again if it get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
> If you look at the patch that added the above warning, its commit message 
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>     
>     The arch_setup_dma_ops() function is in charge of setting dma_ops with a
>     call to set_dma_ops(). set_dma_ops() is also called from
>     
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
>     
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
>     
>     To allow the arch_setup_dma_ops() call to be moved from device add time
>     to device probe time we must ensure that dma_ops already setup by any of
>     the above callers will not be overriden.
>     
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
>     of_xlate and taking care of highbank and mvebu, the workaround should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally remove 
> the DMA ops and mapping.

Ah, sorry, I see now - it was taking a long time to page the 32-bit code
back in, and I'd forgotten the specifics of the mess of competing
"default domain" notions. Indeed, it's not the device's driver expecting
any state to be preserved as I got stuck on, it's the IOMMU driver,
which does "know better" to an extent, expecting its changes to the
struct device to stick for the lifetime of that structure.

I agree there shouldn't be a disparity - arch_setup_dma_ops() only does
things given certain circumstances, so arch_teardown_dma_ops() should
only undo them under the same. With probe-deferral in place I'll be
reviving my work to convert this path over to IOMMU API default domains,
which will make some of these issues go away again, but in the meantime
I also agree that the most expedient fix is indeed to add a flag to say
whether the dma ops were automatically set or not (this is implicitly
true on arm64, which was partly what was tripping me up).

Robin.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 16/05/17 15:10, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>>> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> 
> [snip]
> 
>>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>>>> ,dma_ops should be cleared in the teardown path. Otherwise
>>>> this causes problem when the probe of device is retried after
>>>> being deferred. The device's iommu structures are cleared
>>>> after EPROBEDEFER error, but on the next try dma_ops will still
>>>> be set to old value, which is not right.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>>> ---
>>>>
>>>>   arch/arm/mm/dma-mapping.c | 1 +
>>>>   1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index ab4f745..a40f03e 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>>>> device *dev)
>>>
>>>>   	__arm_iommu_detach_device(dev);
>>>>   	arm_iommu_release_mapping(mapping);
>>>> +	set_dma_ops(dev, NULL);
>>>>   }
>>>>   #else
>>>
>>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
>>> but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>>>
>>> However, the situation is perhaps more complex. Note the check at the
>>> beginning of arch_setup_dma_ops():
>>> 	/*
>>> 	 * Don't override the dma_ops if they have already been set. Ideally
>>> 	 * this should be the only location where dma_ops are set, remove this
>>> 	 * check when all other callers of set_dma_ops will have disappeared.
>>> 	 */
>>> 	if (dev->dma_ops)
>>> 		return;
>>>
>>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>>> override them. To be safe you should only set them to NULL if they have
>>> been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>>> should probably not call arm_teardown_iommu_dma_ops() at all if the
>>> dma_ops were set by arm_iommu_attach_device() and not
>>> arch_teardown_dma_ops().
>>
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much state
>> after that. Everything else would be set up from scratch again if it get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
> If you look at the patch that added the above warning, its commit message 
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>     
>     The arch_setup_dma_ops() function is in charge of setting dma_ops with a
>     call to set_dma_ops(). set_dma_ops() is also called from
>     
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
>     
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
>     
>     To allow the arch_setup_dma_ops() call to be moved from device add time
>     to device probe time we must ensure that dma_ops already setup by any of
>     the above callers will not be overriden.
>     
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
>     of_xlate and taking care of highbank and mvebu, the workaround should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally remove 
> the DMA ops and mapping.

Ah, sorry, I see now - it was taking a long time to page the 32-bit code
back in, and I'd forgotten the specifics of the mess of competing
"default domain" notions. Indeed, it's not the device's driver expecting
any state to be preserved as I got stuck on, it's the IOMMU driver,
which does "know better" to an extent, expecting its changes to the
struct device to stick for the lifetime of that structure.

I agree there shouldn't be a disparity - arch_setup_dma_ops() only does
things given certain circumstances, so arch_teardown_dma_ops() should
only undo them under the same. With probe-deferral in place I'll be
reviving my work to convert this path over to IOMMU API default domains,
which will make some of these issues go away again, but in the meantime
I also agree that the most expedient fix is indeed to add a flag to say
whether the dma ops were automatically set or not (this is implicitly
true on arm64, which was partly what was tripping me up).

Robin.

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 16/05/17 15:10, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>>> On Tuesday 16 May 2017 07:53:57 sricharan@codeaurora.org wrote:
> 
> [snip]
> 
>>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>>>> ,dma_ops should be cleared in the teardown path. Otherwise
>>>> this causes problem when the probe of device is retried after
>>>> being deferred. The device's iommu structures are cleared
>>>> after EPROBEDEFER error, but on the next try dma_ops will still
>>>> be set to old value, which is not right.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>>> ---
>>>>
>>>>   arch/arm/mm/dma-mapping.c | 1 +
>>>>   1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index ab4f745..a40f03e 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>>>> device *dev)
>>>
>>>>   	__arm_iommu_detach_device(dev);
>>>>   	arm_iommu_release_mapping(mapping);
>>>> +	set_dma_ops(dev, NULL);
>>>>   }
>>>>   #else
>>>
>>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
>>> but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>>>
>>> However, the situation is perhaps more complex. Note the check at the
>>> beginning of arch_setup_dma_ops():
>>> 	/*
>>> 	 * Don't override the dma_ops if they have already been set. Ideally
>>> 	 * this should be the only location where dma_ops are set, remove this
>>> 	 * check when all other callers of set_dma_ops will have disappeared.
>>> 	 */
>>> 	if (dev->dma_ops)
>>> 		return;
>>>
>>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>>> override them. To be safe you should only set them to NULL if they have
>>> been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>>> should probably not call arm_teardown_iommu_dma_ops() at all if the
>>> dma_ops were set by arm_iommu_attach_device() and not
>>> arch_teardown_dma_ops().
>>
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much state
>> after that. Everything else would be set up from scratch again if it get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
> If you look at the patch that added the above warning, its commit message 
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>     
>     The arch_setup_dma_ops() function is in charge of setting dma_ops with a
>     call to set_dma_ops(). set_dma_ops() is also called from
>     
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
>     
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
>     
>     To allow the arch_setup_dma_ops() call to be moved from device add time
>     to device probe time we must ensure that dma_ops already setup by any of
>     the above callers will not be overriden.
>     
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
>     of_xlate and taking care of highbank and mvebu, the workaround should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally remove 
> the DMA ops and mapping.

Ah, sorry, I see now - it was taking a long time to page the 32-bit code
back in, and I'd forgotten the specifics of the mess of competing
"default domain" notions. Indeed, it's not the device's driver expecting
any state to be preserved as I got stuck on, it's the IOMMU driver,
which does "know better" to an extent, expecting its changes to the
struct device to stick for the lifetime of that structure.

I agree there shouldn't be a disparity - arch_setup_dma_ops() only does
things given certain circumstances, so arch_teardown_dma_ops() should
only undo them under the same. With probe-deferral in place I'll be
reviving my work to convert this path over to IOMMU API default domains,
which will make some of these issues go away again, but in the meantime
I also agree that the most expedient fix is indeed to add a flag to say
whether the dma ops were automatically set or not (this is implicitly
true on arm64, which was partly what was tripping me up).

Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 16/05/17 15:10, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 15:04:55 Robin Murphy wrote:
>> On 16/05/17 08:17, Laurent Pinchart wrote:
>>> On Tuesday 16 May 2017 07:53:57 sricharan at codeaurora.org wrote:
> 
> [snip]
> 
>>>> arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>>>> ,dma_ops should be cleared in the teardown path. Otherwise
>>>> this causes problem when the probe of device is retried after
>>>> being deferred. The device's iommu structures are cleared
>>>> after EPROBEDEFER error, but on the next try dma_ops will still
>>>> be set to old value, which is not right.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>>> ---
>>>>
>>>>   arch/arm/mm/dma-mapping.c | 1 +
>>>>   1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index ab4f745..a40f03e 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2358,6 +2358,7 @@ static void arm_teardown_iommu_dma_ops(struct
>>>> device *dev)
>>>
>>>>   	__arm_iommu_detach_device(dev);
>>>>   	arm_iommu_release_mapping(mapping);
>>>> +	set_dma_ops(dev, NULL);
>>>>   }
>>>>   #else
>>>
>>> The subject mentions arch_teardown_dma_ops(), which I think is correct,
>>> but the patch adds the set_dma_ops() call to arm_teardown_iommu_dma_ops().
>>>
>>> However, the situation is perhaps more complex. Note the check at the
>>> beginning of arch_setup_dma_ops():
>>> 	/*
>>> 	 * Don't override the dma_ops if they have already been set. Ideally
>>> 	 * this should be the only location where dma_ops are set, remove this
>>> 	 * check when all other callers of set_dma_ops will have disappeared.
>>> 	 */
>>> 	if (dev->dma_ops)
>>> 		return;
>>>
>>> If you set the dma_ops to NULL in arm_teardown_iommu_dma_ops() or
>>> arch_teardown_dma_ops(), the next call to arch_setup_dma_ops() will
>>> override them. To be safe you should only set them to NULL if they have
>>> been set by arch_setup_dma_ops(). More than that, arch_teardown_dma_ops()
>>> should probably not call arm_teardown_iommu_dma_ops() at all if the
>>> dma_ops were set by arm_iommu_attach_device() and not
>>> arch_teardown_dma_ops().
>>
>> Under what circumstances is that an issue? We'll only be tearing down
>> the DMA ops when unbinding the driver,
> 
> Or when deferring probe.
> 
>> and I think it would be erroneous to expect the device to retain much state
>> after that. Everything else would be set up from scratch again if it get
>> reprobed later, so why not the DMA ops?
> 
> Because the DMA ops might have been set elsewhere than arch_setup_dma_ops(). 
> If you look at the patch that added the above warning, its commit message 
> states
> 
> commit 26b37b946a5c2658dbc37dd5d6df40aaa9685d70 (iommu-joerg/arm/core)
> Author: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> Date:   Fri May 15 02:00:02 2015 +0300
> 
>     arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>     
>     The arch_setup_dma_ops() function is in charge of setting dma_ops with a
>     call to set_dma_ops(). set_dma_ops() is also called from
>     
>     - highbank and mvebu bus notifiers
>     - dmabounce (to be replaced with swiotlb)
>     - arm_iommu_attach_device
>     
>     (arm_iommu_attach_device is itself called from IOMMU and bus master
>     device drivers)
>     
>     To allow the arch_setup_dma_ops() call to be moved from device add time
>     to device probe time we must ensure that dma_ops already setup by any of
>     the above callers will not be overriden.
>     
>     Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
>     of_xlate and taking care of highbank and mvebu, the workaround should be
>     removed.
> 
> I'm concerned about potentially breaking these if we unconditionally remove 
> the DMA ops and mapping.

Ah, sorry, I see now - it was taking a long time to page the 32-bit code
back in, and I'd forgotten the specifics of the mess of competing
"default domain" notions. Indeed, it's not the device's driver expecting
any state to be preserved as I got stuck on, it's the IOMMU driver,
which does "know better" to an extent, expecting its changes to the
struct device to stick for the lifetime of that structure.

I agree there shouldn't be a disparity - arch_setup_dma_ops() only does
things given certain circumstances, so arch_teardown_dma_ops() should
only undo them under the same. With probe-deferral in place I'll be
reviving my work to convert this path over to IOMMU API default domains,
which will make some of these issues go away again, but in the meantime
I also agree that the most expedient fix is indeed to add a flag to say
whether the dma ops were automatically set or not (this is implicitly
true on arm64, which was partly what was tripping me up).

Robin.

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Laurent Pinchart]

arch_setup_dma_ops() is used in device probe code paths to create an
IOMMU mapping and attach it to the device. The function assumes that the
device is attached to a device-specific IOMMU instance (or at least a
device-specific TLB in a shared IOMMU instance) and thus creates a
separate mapping for every device.

On several systems (Renesas R-Car Gen2 being one of them), that
assumption is not true, and IOMMU mappings must be shared between
multiple devices. In those cases the IOMMU driver knows better than the
generic ARM dma-mapping layer and attaches mapping to devices manually
with arm_iommu_attach_device(), which sets the DMA ops for the device.

The arch_setup_dma_ops() function takes this into account and bails out
immediately if the device already has DMA ops assigned. However, the
corresponding arch_teardown_dma_ops() function, called from driver
unbind code paths (including probe deferral), will tear the mapping down
regardless of who created it. When the device is reprobed
arch_setup_dma_ops() will be called again but won't perform any
operation as the DMA ops will still be set.

We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
However, we can't do so unconditionally, as then a new mapping would be
created by arch_setup_dma_ops() when the device is reprobed, regardless
of whether the device needs to share a mapping or not. We must thus keep
track of whether arch_setup_dma_ops() created the mapping, and only in
that case tear it down in arch_teardown_dma_ops().

Keep track of that information in the dev_archdata structure. As the
structure is embedded in all instances of struct device let's not grow
it, but turn the existing dma_coherent bool field into a bitfield that
can be used for other purposes.

Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred probing or error")
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 arch/arm/include/asm/device.h | 3 ++-
 arch/arm/mm/dma-mapping.c     | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
index 36ec9c8f6e16..3234fe9bba6e 100644
--- a/arch/arm/include/asm/device.h
+++ b/arch/arm/include/asm/device.h
@@ -19,7 +19,8 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
+	unsigned int dma_coherent:1;
+	unsigned int dma_ops_setup:1;
 };
 
 struct omap_device;
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index c742dfd2967b..e0272f9140e2 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 		dev->dma_ops = xen_dma_ops;
 	}
 #endif
+	dev->archdata.dma_ops_setup = true;
 }
 
 void arch_teardown_dma_ops(struct device *dev)
 {
+	if (!dev->archdata.dma_ops_setup)
+		return;
+
 	arm_teardown_iommu_dma_ops(dev);
+	set_dma_ops(dev, NULL);
 }
-- 
Regards,

Laurent Pinchart

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Laurent Pinchart]

arch_setup_dma_ops() is used in device probe code paths to create an
IOMMU mapping and attach it to the device. The function assumes that the
device is attached to a device-specific IOMMU instance (or at least a
device-specific TLB in a shared IOMMU instance) and thus creates a
separate mapping for every device.

On several systems (Renesas R-Car Gen2 being one of them), that
assumption is not true, and IOMMU mappings must be shared between
multiple devices. In those cases the IOMMU driver knows better than the
generic ARM dma-mapping layer and attaches mapping to devices manually
with arm_iommu_attach_device(), which sets the DMA ops for the device.

The arch_setup_dma_ops() function takes this into account and bails out
immediately if the device already has DMA ops assigned. However, the
corresponding arch_teardown_dma_ops() function, called from driver
unbind code paths (including probe deferral), will tear the mapping down
regardless of who created it. When the device is reprobed
arch_setup_dma_ops() will be called again but won't perform any
operation as the DMA ops will still be set.

We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
However, we can't do so unconditionally, as then a new mapping would be
created by arch_setup_dma_ops() when the device is reprobed, regardless
of whether the device needs to share a mapping or not. We must thus keep
track of whether arch_setup_dma_ops() created the mapping, and only in
that case tear it down in arch_teardown_dma_ops().

Keep track of that information in the dev_archdata structure. As the
structure is embedded in all instances of struct device let's not grow
it, but turn the existing dma_coherent bool field into a bitfield that
can be used for other purposes.

Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred probing or error")
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 arch/arm/include/asm/device.h | 3 ++-
 arch/arm/mm/dma-mapping.c     | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
index 36ec9c8f6e16..3234fe9bba6e 100644
--- a/arch/arm/include/asm/device.h
+++ b/arch/arm/include/asm/device.h
@@ -19,7 +19,8 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
+	unsigned int dma_coherent:1;
+	unsigned int dma_ops_setup:1;
 };
 
 struct omap_device;
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index c742dfd2967b..e0272f9140e2 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 		dev->dma_ops = xen_dma_ops;
 	}
 #endif
+	dev->archdata.dma_ops_setup = true;
 }
 
 void arch_teardown_dma_ops(struct device *dev)
 {
+	if (!dev->archdata.dma_ops_setup)
+		return;
+
 	arm_teardown_iommu_dma_ops(dev);
+	set_dma_ops(dev, NULL);
 }
-- 
Regards,

Laurent Pinchart

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Robin Murphy]

On 16/05/17 16:14, Laurent Pinchart wrote:
> arch_setup_dma_ops() is used in device probe code paths to create an
> IOMMU mapping and attach it to the device. The function assumes that the
> device is attached to a device-specific IOMMU instance (or at least a
> device-specific TLB in a shared IOMMU instance) and thus creates a
> separate mapping for every device.
> 
> On several systems (Renesas R-Car Gen2 being one of them), that
> assumption is not true, and IOMMU mappings must be shared between
> multiple devices. In those cases the IOMMU driver knows better than the
> generic ARM dma-mapping layer and attaches mapping to devices manually
> with arm_iommu_attach_device(), which sets the DMA ops for the device.
> 
> The arch_setup_dma_ops() function takes this into account and bails out
> immediately if the device already has DMA ops assigned. However, the
> corresponding arch_teardown_dma_ops() function, called from driver
> unbind code paths (including probe deferral), will tear the mapping down
> regardless of who created it. When the device is reprobed
> arch_setup_dma_ops() will be called again but won't perform any
> operation as the DMA ops will still be set.
> 
> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
> However, we can't do so unconditionally, as then a new mapping would be
> created by arch_setup_dma_ops() when the device is reprobed, regardless
> of whether the device needs to share a mapping or not. We must thus keep
> track of whether arch_setup_dma_ops() created the mapping, and only in
> that case tear it down in arch_teardown_dma_ops().
> 
> Keep track of that information in the dev_archdata structure. As the
> structure is embedded in all instances of struct device let's not grow
> it, but turn the existing dma_coherent bool field into a bitfield that
> can be used for other purposes.
> 
> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred probing or error")
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  arch/arm/include/asm/device.h | 3 ++-
>  arch/arm/mm/dma-mapping.c     | 5 +++++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
> index 36ec9c8f6e16..3234fe9bba6e 100644
> --- a/arch/arm/include/asm/device.h
> +++ b/arch/arm/include/asm/device.h
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	unsigned int dma_coherent:1;

This should only ever be accessed by the Xen DMA code via the
is_device_dma_coherent() helper, so I can't see the change of storage
type causing any problems.

> +	unsigned int dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index c742dfd2967b..e0272f9140e2 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  		dev->dma_ops = xen_dma_ops;
>  	}
>  #endif
> +	dev->archdata.dma_ops_setup = true;
>  }
>  
>  void arch_teardown_dma_ops(struct device *dev)
>  {
> +	if (!dev->archdata.dma_ops_setup)
> +		return;
> +
>  	arm_teardown_iommu_dma_ops(dev);
> +	set_dma_ops(dev, NULL);

Should we clear dma_ops_setup here for symmetry? I guess in practice
it's down to the IOMMU driver so will never change after the first
probe, but it still feels like a bit of a nagging loose end.

With that (or firm reassurance that it's OK not to),

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

Apologies for being too arm64-focused in the earlier reviews and
overlooking this. Should the patch supersede 8674/1 currently in
Russell's incoming box?

Robin.

>  }
> 

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Robin Murphy]

On 16/05/17 16:14, Laurent Pinchart wrote:
> arch_setup_dma_ops() is used in device probe code paths to create an
> IOMMU mapping and attach it to the device. The function assumes that the
> device is attached to a device-specific IOMMU instance (or at least a
> device-specific TLB in a shared IOMMU instance) and thus creates a
> separate mapping for every device.
> 
> On several systems (Renesas R-Car Gen2 being one of them), that
> assumption is not true, and IOMMU mappings must be shared between
> multiple devices. In those cases the IOMMU driver knows better than the
> generic ARM dma-mapping layer and attaches mapping to devices manually
> with arm_iommu_attach_device(), which sets the DMA ops for the device.
> 
> The arch_setup_dma_ops() function takes this into account and bails out
> immediately if the device already has DMA ops assigned. However, the
> corresponding arch_teardown_dma_ops() function, called from driver
> unbind code paths (including probe deferral), will tear the mapping down
> regardless of who created it. When the device is reprobed
> arch_setup_dma_ops() will be called again but won't perform any
> operation as the DMA ops will still be set.
> 
> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
> However, we can't do so unconditionally, as then a new mapping would be
> created by arch_setup_dma_ops() when the device is reprobed, regardless
> of whether the device needs to share a mapping or not. We must thus keep
> track of whether arch_setup_dma_ops() created the mapping, and only in
> that case tear it down in arch_teardown_dma_ops().
> 
> Keep track of that information in the dev_archdata structure. As the
> structure is embedded in all instances of struct device let's not grow
> it, but turn the existing dma_coherent bool field into a bitfield that
> can be used for other purposes.
> 
> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred probing or error")
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> ---
>  arch/arm/include/asm/device.h | 3 ++-
>  arch/arm/mm/dma-mapping.c     | 5 +++++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
> index 36ec9c8f6e16..3234fe9bba6e 100644
> --- a/arch/arm/include/asm/device.h
> +++ b/arch/arm/include/asm/device.h
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	unsigned int dma_coherent:1;

This should only ever be accessed by the Xen DMA code via the
is_device_dma_coherent() helper, so I can't see the change of storage
type causing any problems.

> +	unsigned int dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index c742dfd2967b..e0272f9140e2 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  		dev->dma_ops = xen_dma_ops;
>  	}
>  #endif
> +	dev->archdata.dma_ops_setup = true;
>  }
>  
>  void arch_teardown_dma_ops(struct device *dev)
>  {
> +	if (!dev->archdata.dma_ops_setup)
> +		return;
> +
>  	arm_teardown_iommu_dma_ops(dev);
> +	set_dma_ops(dev, NULL);

Should we clear dma_ops_setup here for symmetry? I guess in practice
it's down to the IOMMU driver so will never change after the first
probe, but it still feels like a bit of a nagging loose end.

With that (or firm reassurance that it's OK not to),

Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>

Apologies for being too arm64-focused in the earlier reviews and
overlooking this. Should the patch supersede 8674/1 currently in
Russell's incoming box?

Robin.

>  }
> 

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Robin Murphy]

On 16/05/17 16:14, Laurent Pinchart wrote:
> arch_setup_dma_ops() is used in device probe code paths to create an
> IOMMU mapping and attach it to the device. The function assumes that the
> device is attached to a device-specific IOMMU instance (or at least a
> device-specific TLB in a shared IOMMU instance) and thus creates a
> separate mapping for every device.
> 
> On several systems (Renesas R-Car Gen2 being one of them), that
> assumption is not true, and IOMMU mappings must be shared between
> multiple devices. In those cases the IOMMU driver knows better than the
> generic ARM dma-mapping layer and attaches mapping to devices manually
> with arm_iommu_attach_device(), which sets the DMA ops for the device.
> 
> The arch_setup_dma_ops() function takes this into account and bails out
> immediately if the device already has DMA ops assigned. However, the
> corresponding arch_teardown_dma_ops() function, called from driver
> unbind code paths (including probe deferral), will tear the mapping down
> regardless of who created it. When the device is reprobed
> arch_setup_dma_ops() will be called again but won't perform any
> operation as the DMA ops will still be set.
> 
> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
> However, we can't do so unconditionally, as then a new mapping would be
> created by arch_setup_dma_ops() when the device is reprobed, regardless
> of whether the device needs to share a mapping or not. We must thus keep
> track of whether arch_setup_dma_ops() created the mapping, and only in
> that case tear it down in arch_teardown_dma_ops().
> 
> Keep track of that information in the dev_archdata structure. As the
> structure is embedded in all instances of struct device let's not grow
> it, but turn the existing dma_coherent bool field into a bitfield that
> can be used for other purposes.
> 
> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred probing or error")
> Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> ---
>  arch/arm/include/asm/device.h | 3 ++-
>  arch/arm/mm/dma-mapping.c     | 5 +++++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
> index 36ec9c8f6e16..3234fe9bba6e 100644
> --- a/arch/arm/include/asm/device.h
> +++ b/arch/arm/include/asm/device.h
> @@ -19,7 +19,8 @@ struct dev_archdata {
>  #ifdef CONFIG_XEN
>  	const struct dma_map_ops *dev_dma_ops;
>  #endif
> -	bool dma_coherent;
> +	unsigned int dma_coherent:1;

This should only ever be accessed by the Xen DMA code via the
is_device_dma_coherent() helper, so I can't see the change of storage
type causing any problems.

> +	unsigned int dma_ops_setup:1;
>  };
>  
>  struct omap_device;
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index c742dfd2967b..e0272f9140e2 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  		dev->dma_ops = xen_dma_ops;
>  	}
>  #endif
> +	dev->archdata.dma_ops_setup = true;
>  }
>  
>  void arch_teardown_dma_ops(struct device *dev)
>  {
> +	if (!dev->archdata.dma_ops_setup)
> +		return;
> +
>  	arm_teardown_iommu_dma_ops(dev);
> +	set_dma_ops(dev, NULL);

Should we clear dma_ops_setup here for symmetry? I guess in practice
it's down to the IOMMU driver so will never change after the first
probe, but it still feels like a bit of a nagging loose end.

With that (or firm reassurance that it's OK not to),

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

Apologies for being too arm64-focused in the earlier reviews and
overlooking this. Should the patch supersede 8674/1 currently in
Russell's incoming box?

Robin.

>  }
> 

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Laurent Pinchart]

Hi Robin,

On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
> On 16/05/17 16:14, Laurent Pinchart wrote:
> > arch_setup_dma_ops() is used in device probe code paths to create an
> > IOMMU mapping and attach it to the device. The function assumes that the
> > device is attached to a device-specific IOMMU instance (or at least a
> > device-specific TLB in a shared IOMMU instance) and thus creates a
> > separate mapping for every device.
> > 
> > On several systems (Renesas R-Car Gen2 being one of them), that
> > assumption is not true, and IOMMU mappings must be shared between
> > multiple devices. In those cases the IOMMU driver knows better than the
> > generic ARM dma-mapping layer and attaches mapping to devices manually
> > with arm_iommu_attach_device(), which sets the DMA ops for the device.
> > 
> > The arch_setup_dma_ops() function takes this into account and bails out
> > immediately if the device already has DMA ops assigned. However, the
> > corresponding arch_teardown_dma_ops() function, called from driver
> > unbind code paths (including probe deferral), will tear the mapping down
> > regardless of who created it. When the device is reprobed
> > arch_setup_dma_ops() will be called again but won't perform any
> > operation as the DMA ops will still be set.
> > 
> > We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
> > However, we can't do so unconditionally, as then a new mapping would be
> > created by arch_setup_dma_ops() when the device is reprobed, regardless
> > of whether the device needs to share a mapping or not. We must thus keep
> > track of whether arch_setup_dma_ops() created the mapping, and only in
> > that case tear it down in arch_teardown_dma_ops().
> > 
> > Keep track of that information in the dev_archdata structure. As the
> > structure is embedded in all instances of struct device let's not grow
> > it, but turn the existing dma_coherent bool field into a bitfield that
> > can be used for other purposes.
> > 
> > Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred
> > probing or error") Signed-off-by: Laurent Pinchart
> > <laurent.pinchart+renesas@ideasonboard.com> ---
> > 
> >  arch/arm/include/asm/device.h | 3 ++-
> >  arch/arm/mm/dma-mapping.c     | 5 +++++
> >  2 files changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
> > index 36ec9c8f6e16..3234fe9bba6e 100644
> > --- a/arch/arm/include/asm/device.h
> > +++ b/arch/arm/include/asm/device.h
> > @@ -19,7 +19,8 @@ struct dev_archdata {
> >  #ifdef CONFIG_XEN
> >  	const struct dma_map_ops *dev_dma_ops;
> >  #endif
> > -	bool dma_coherent;
> > +	unsigned int dma_coherent:1;
> 
> This should only ever be accessed by the Xen DMA code via the
> is_device_dma_coherent() helper, so I can't see the change of storage
> type causing any problems.

Thank you for double-checking. I agree with your analysis.

> > +	unsigned int dma_ops_setup:1;
> >  };
> >  
> >  struct omap_device;
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index c742dfd2967b..e0272f9140e2 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64
> > dma_base, u64 size,
> >  		dev->dma_ops = xen_dma_ops;
> >  	}
> >  #endif
> > +	dev->archdata.dma_ops_setup = true;
> >  }
> >  
> >  void arch_teardown_dma_ops(struct device *dev)
> >  {
> > +	if (!dev->archdata.dma_ops_setup)
> > +		return;
> > +
> >  	arm_teardown_iommu_dma_ops(dev);
> > +	set_dma_ops(dev, NULL);
> 
> Should we clear dma_ops_setup here for symmetry? I guess in practice
> it's down to the IOMMU driver so will never change after the first
> probe, but it still feels like a bit of a nagging loose end.

To make a difference, we would need an IOMMU driver that creates a mapping 
after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() calls, 
follow by a second round. I don't think this could happen, but if it did, I 
believe we'd be screwed already, as there would be a time were an incorrect 
mapping (created by arch_setup_dma_ops() while the IOMMU driver needs to take 
care of mapping creation) exists.

> With that (or firm reassurance that it's OK not to),
> 
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> 
> Apologies for being too arm64-focused in the earlier reviews and
> overlooking this. Should the patch supersede 8674/1 currently in
> Russell's incoming box?

Yes I think it should. Could you please take care of that ?

You can also add my

Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

as I've tested that this paptch restores proper IOMMU operation on the Renesas 
R-Car H2 Lager board. I believe the problem related to Sricharan's patch 
reported by Geert still affects us and needs to be addressed separately.

-- 
Regards,

Laurent Pinchart

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Laurent Pinchart]

Hi Robin,

On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
> On 16/05/17 16:14, Laurent Pinchart wrote:
> > arch_setup_dma_ops() is used in device probe code paths to create an
> > IOMMU mapping and attach it to the device. The function assumes that the
> > device is attached to a device-specific IOMMU instance (or at least a
> > device-specific TLB in a shared IOMMU instance) and thus creates a
> > separate mapping for every device.
> > 
> > On several systems (Renesas R-Car Gen2 being one of them), that
> > assumption is not true, and IOMMU mappings must be shared between
> > multiple devices. In those cases the IOMMU driver knows better than the
> > generic ARM dma-mapping layer and attaches mapping to devices manually
> > with arm_iommu_attach_device(), which sets the DMA ops for the device.
> > 
> > The arch_setup_dma_ops() function takes this into account and bails out
> > immediately if the device already has DMA ops assigned. However, the
> > corresponding arch_teardown_dma_ops() function, called from driver
> > unbind code paths (including probe deferral), will tear the mapping down
> > regardless of who created it. When the device is reprobed
> > arch_setup_dma_ops() will be called again but won't perform any
> > operation as the DMA ops will still be set.
> > 
> > We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
> > However, we can't do so unconditionally, as then a new mapping would be
> > created by arch_setup_dma_ops() when the device is reprobed, regardless
> > of whether the device needs to share a mapping or not. We must thus keep
> > track of whether arch_setup_dma_ops() created the mapping, and only in
> > that case tear it down in arch_teardown_dma_ops().
> > 
> > Keep track of that information in the dev_archdata structure. As the
> > structure is embedded in all instances of struct device let's not grow
> > it, but turn the existing dma_coherent bool field into a bitfield that
> > can be used for other purposes.
> > 
> > Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred
> > probing or error") Signed-off-by: Laurent Pinchart
> > <laurent.pinchart+renesas@ideasonboard.com> ---
> > 
> >  arch/arm/include/asm/device.h | 3 ++-
> >  arch/arm/mm/dma-mapping.c     | 5 +++++
> >  2 files changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
> > index 36ec9c8f6e16..3234fe9bba6e 100644
> > --- a/arch/arm/include/asm/device.h
> > +++ b/arch/arm/include/asm/device.h
> > @@ -19,7 +19,8 @@ struct dev_archdata {
> >  #ifdef CONFIG_XEN
> >  	const struct dma_map_ops *dev_dma_ops;
> >  #endif
> > -	bool dma_coherent;
> > +	unsigned int dma_coherent:1;
> 
> This should only ever be accessed by the Xen DMA code via the
> is_device_dma_coherent() helper, so I can't see the change of storage
> type causing any problems.

Thank you for double-checking. I agree with your analysis.

> > +	unsigned int dma_ops_setup:1;
> >  };
> >  
> >  struct omap_device;
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index c742dfd2967b..e0272f9140e2 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64
> > dma_base, u64 size,
> >  		dev->dma_ops = xen_dma_ops;
> >  	}
> >  #endif
> > +	dev->archdata.dma_ops_setup = true;
> >  }
> >  
> >  void arch_teardown_dma_ops(struct device *dev)
> >  {
> > +	if (!dev->archdata.dma_ops_setup)
> > +		return;
> > +
> >  	arm_teardown_iommu_dma_ops(dev);
> > +	set_dma_ops(dev, NULL);
> 
> Should we clear dma_ops_setup here for symmetry? I guess in practice
> it's down to the IOMMU driver so will never change after the first
> probe, but it still feels like a bit of a nagging loose end.

To make a difference, we would need an IOMMU driver that creates a mapping 
after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() calls, 
follow by a second round. I don't think this could happen, but if it did, I 
believe we'd be screwed already, as there would be a time were an incorrect 
mapping (created by arch_setup_dma_ops() while the IOMMU driver needs to take 
care of mapping creation) exists.

> With that (or firm reassurance that it's OK not to),
> 
> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> 
> Apologies for being too arm64-focused in the earlier reviews and
> overlooking this. Should the patch supersede 8674/1 currently in
> Russell's incoming box?

Yes I think it should. Could you please take care of that ?

You can also add my

Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

as I've tested that this paptch restores proper IOMMU operation on the Renesas 
R-Car H2 Lager board. I believe the problem related to Sricharan's patch 
reported by Geert still affects us and needs to be addressed separately.

-- 
Regards,

Laurent Pinchart

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Sricharan R]

Hi Laurent/Robin,

On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>> IOMMU mapping and attach it to the device. The function assumes that the
>>> device is attached to a device-specific IOMMU instance (or at least a
>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>> separate mapping for every device.
>>>
>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>> assumption is not true, and IOMMU mappings must be shared between
>>> multiple devices. In those cases the IOMMU driver knows better than the
>>> generic ARM dma-mapping layer and attaches mapping to devices manually
>>> with arm_iommu_attach_device(), which sets the DMA ops for the device.
>>>
>>> The arch_setup_dma_ops() function takes this into account and bails out
>>> immediately if the device already has DMA ops assigned. However, the
>>> corresponding arch_teardown_dma_ops() function, called from driver
>>> unbind code paths (including probe deferral), will tear the mapping down
>>> regardless of who created it. When the device is reprobed
>>> arch_setup_dma_ops() will be called again but won't perform any
>>> operation as the DMA ops will still be set.
>>>
>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>> However, we can't do so unconditionally, as then a new mapping would be
>>> created by arch_setup_dma_ops() when the device is reprobed, regardless
>>> of whether the device needs to share a mapping or not. We must thus keep
>>> track of whether arch_setup_dma_ops() created the mapping, and only in
>>> that case tear it down in arch_teardown_dma_ops().
>>>
>>> Keep track of that information in the dev_archdata structure. As the
>>> structure is embedded in all instances of struct device let's not grow
>>> it, but turn the existing dma_coherent bool field into a bitfield that
>>> can be used for other purposes.
>>>
>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred
>>> probing or error") Signed-off-by: Laurent Pinchart
>>> <laurent.pinchart+renesas@ideasonboard.com> ---
>>>
>>>  arch/arm/include/asm/device.h | 3 ++-
>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>> --- a/arch/arm/include/asm/device.h
>>> +++ b/arch/arm/include/asm/device.h
>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>  #ifdef CONFIG_XEN
>>>  	const struct dma_map_ops *dev_dma_ops;
>>>  #endif
>>> -	bool dma_coherent;
>>> +	unsigned int dma_coherent:1;
>>
>> This should only ever be accessed by the Xen DMA code via the
>> is_device_dma_coherent() helper, so I can't see the change of storage
>> type causing any problems.
> 
> Thank you for double-checking. I agree with your analysis.
> 
>>> +	unsigned int dma_ops_setup:1;
>>>  };
>>>  
>>>  struct omap_device;
>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>> index c742dfd2967b..e0272f9140e2 100644
>>> --- a/arch/arm/mm/dma-mapping.c
>>> +++ b/arch/arm/mm/dma-mapping.c
>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64
>>> dma_base, u64 size,
>>>  		dev->dma_ops = xen_dma_ops;
>>>  	}
>>>  #endif
>>> +	dev->archdata.dma_ops_setup = true;
>>>  }
>>>  
>>>  void arch_teardown_dma_ops(struct device *dev)
>>>  {
>>> +	if (!dev->archdata.dma_ops_setup)
>>> +		return;
>>> +
>>>  	arm_teardown_iommu_dma_ops(dev);
>>> +	set_dma_ops(dev, NULL);
>>
>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>> it's down to the IOMMU driver so will never change after the first
>> probe, but it still feels like a bit of a nagging loose end.
> 
> To make a difference, we would need an IOMMU driver that creates a mapping 
> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() calls, 
> follow by a second round. I don't think this could happen, but if it did, I 
> believe we'd be screwed already, as there would be a time were an incorrect 
> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs to take 
> care of mapping creation) exists.
> 

Feels correct not to reset this, the iommu drivers in question, seems to
creating mapping/attaching in add_device path (which gets called before the
clients gets probed) and when the iommu client gets deferred/reprobed that
does not happen again even after the first round.

>> With that (or firm reassurance that it's OK not to),
>>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>
>> Apologies for being too arm64-focused in the earlier reviews and
>> overlooking this. Should the patch supersede 8674/1 currently in
>> Russell's incoming box?
> 
> Yes I think it should. Could you please take care of that ?
> 
> You can also add my
> was
> Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> as I've tested that this paptch restores proper IOMMU operation on the Renesas 
> R-Car H2 Lager board. I believe the problem related to Sricharan's patch 
> reported by Geert still affects us and needs to be addressed separately.

Thanks for the above, i had the same thing to be posted, was just testing it once.
There are three patches [1][2], already posted and third one for the issue that Geert
pointed i did below (Geert had a patch little differently to ignore -ENODEV).
I had this question previously for not propagating errors apart from EPROBE_DEFER,
did not have an issue reported at that time. Anyways if the below is ok, i will
just send the 3 patches in one set for easy picking up ?

[1] https://lkml.org/lkml/2017/5/16/25  
[2] The above one that you have. 
[3] The below one, if its fine ?

>From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Wed, 3 May 2017 14:54:11 +0530
Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER

While deferring the probe of iommu masters,
xlate and add_device callback can passback error values
like -ENODEV, which means iommu cannot be connected
with that master for real reasons. So rather than
killing the master's probe for such errors, just
ignore the errors and let the master work without
an iommu.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index e6e9bec..750ab07 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                        ops = ERR_PTR(err);
        }

+       /* Ignore all other errors apart from EPROBE_DEFER */
+       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
+               ops = NULL;
+
        return ops;
 }

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation



> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Regards,
 Sricharan

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Sricharan R]

Hi Laurent/Robin,

On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>> IOMMU mapping and attach it to the device. The function assumes that the
>>> device is attached to a device-specific IOMMU instance (or at least a
>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>> separate mapping for every device.
>>>
>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>> assumption is not true, and IOMMU mappings must be shared between
>>> multiple devices. In those cases the IOMMU driver knows better than the
>>> generic ARM dma-mapping layer and attaches mapping to devices manually
>>> with arm_iommu_attach_device(), which sets the DMA ops for the device.
>>>
>>> The arch_setup_dma_ops() function takes this into account and bails out
>>> immediately if the device already has DMA ops assigned. However, the
>>> corresponding arch_teardown_dma_ops() function, called from driver
>>> unbind code paths (including probe deferral), will tear the mapping down
>>> regardless of who created it. When the device is reprobed
>>> arch_setup_dma_ops() will be called again but won't perform any
>>> operation as the DMA ops will still be set.
>>>
>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>> However, we can't do so unconditionally, as then a new mapping would be
>>> created by arch_setup_dma_ops() when the device is reprobed, regardless
>>> of whether the device needs to share a mapping or not. We must thus keep
>>> track of whether arch_setup_dma_ops() created the mapping, and only in
>>> that case tear it down in arch_teardown_dma_ops().
>>>
>>> Keep track of that information in the dev_archdata structure. As the
>>> structure is embedded in all instances of struct device let's not grow
>>> it, but turn the existing dma_coherent bool field into a bitfield that
>>> can be used for other purposes.
>>>
>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred
>>> probing or error") Signed-off-by: Laurent Pinchart
>>> <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org> ---
>>>
>>>  arch/arm/include/asm/device.h | 3 ++-
>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>> --- a/arch/arm/include/asm/device.h
>>> +++ b/arch/arm/include/asm/device.h
>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>  #ifdef CONFIG_XEN
>>>  	const struct dma_map_ops *dev_dma_ops;
>>>  #endif
>>> -	bool dma_coherent;
>>> +	unsigned int dma_coherent:1;
>>
>> This should only ever be accessed by the Xen DMA code via the
>> is_device_dma_coherent() helper, so I can't see the change of storage
>> type causing any problems.
> 
> Thank you for double-checking. I agree with your analysis.
> 
>>> +	unsigned int dma_ops_setup:1;
>>>  };
>>>  
>>>  struct omap_device;
>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>> index c742dfd2967b..e0272f9140e2 100644
>>> --- a/arch/arm/mm/dma-mapping.c
>>> +++ b/arch/arm/mm/dma-mapping.c
>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64
>>> dma_base, u64 size,
>>>  		dev->dma_ops = xen_dma_ops;
>>>  	}
>>>  #endif
>>> +	dev->archdata.dma_ops_setup = true;
>>>  }
>>>  
>>>  void arch_teardown_dma_ops(struct device *dev)
>>>  {
>>> +	if (!dev->archdata.dma_ops_setup)
>>> +		return;
>>> +
>>>  	arm_teardown_iommu_dma_ops(dev);
>>> +	set_dma_ops(dev, NULL);
>>
>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>> it's down to the IOMMU driver so will never change after the first
>> probe, but it still feels like a bit of a nagging loose end.
> 
> To make a difference, we would need an IOMMU driver that creates a mapping 
> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() calls, 
> follow by a second round. I don't think this could happen, but if it did, I 
> believe we'd be screwed already, as there would be a time were an incorrect 
> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs to take 
> care of mapping creation) exists.
> 

Feels correct not to reset this, the iommu drivers in question, seems to
creating mapping/attaching in add_device path (which gets called before the
clients gets probed) and when the iommu client gets deferred/reprobed that
does not happen again even after the first round.

>> With that (or firm reassurance that it's OK not to),
>>
>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>>
>> Apologies for being too arm64-focused in the earlier reviews and
>> overlooking this. Should the patch supersede 8674/1 currently in
>> Russell's incoming box?
> 
> Yes I think it should. Could you please take care of that ?
> 
> You can also add my
> was
> Tested-by: Laurent Pinchart <laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> 
> as I've tested that this paptch restores proper IOMMU operation on the Renesas 
> R-Car H2 Lager board. I believe the problem related to Sricharan's patch 
> reported by Geert still affects us and needs to be addressed separately.

Thanks for the above, i had the same thing to be posted, was just testing it once.
There are three patches [1][2], already posted and third one for the issue that Geert
pointed i did below (Geert had a patch little differently to ignore -ENODEV).
I had this question previously for not propagating errors apart from EPROBE_DEFER,
did not have an issue reported at that time. Anyways if the below is ok, i will
just send the 3 patches in one set for easy picking up ?

[1] https://lkml.org/lkml/2017/5/16/25  
[2] The above one that you have. 
[3] The below one, if its fine ?

>From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
Date: Wed, 3 May 2017 14:54:11 +0530
Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER

While deferring the probe of iommu masters,
xlate and add_device callback can passback error values
like -ENODEV, which means iommu cannot be connected
with that master for real reasons. So rather than
killing the master's probe for such errors, just
ignore the errors and let the master work without
an iommu.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/iommu/of_iommu.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index e6e9bec..750ab07 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                        ops = ERR_PTR(err);
        }

+       /* Ignore all other errors apart from EPROBE_DEFER */
+       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
+               ops = NULL;
+
        return ops;
 }

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation



> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Regards,
 Sricharan

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[Sricharan R]

Hi Laurent/Robin,

On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
> Hi Robin,
> 
> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>> IOMMU mapping and attach it to the device. The function assumes that the
>>> device is attached to a device-specific IOMMU instance (or at least a
>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>> separate mapping for every device.
>>>
>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>> assumption is not true, and IOMMU mappings must be shared between
>>> multiple devices. In those cases the IOMMU driver knows better than the
>>> generic ARM dma-mapping layer and attaches mapping to devices manually
>>> with arm_iommu_attach_device(), which sets the DMA ops for the device.
>>>
>>> The arch_setup_dma_ops() function takes this into account and bails out
>>> immediately if the device already has DMA ops assigned. However, the
>>> corresponding arch_teardown_dma_ops() function, called from driver
>>> unbind code paths (including probe deferral), will tear the mapping down
>>> regardless of who created it. When the device is reprobed
>>> arch_setup_dma_ops() will be called again but won't perform any
>>> operation as the DMA ops will still be set.
>>>
>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>> However, we can't do so unconditionally, as then a new mapping would be
>>> created by arch_setup_dma_ops() when the device is reprobed, regardless
>>> of whether the device needs to share a mapping or not. We must thus keep
>>> track of whether arch_setup_dma_ops() created the mapping, and only in
>>> that case tear it down in arch_teardown_dma_ops().
>>>
>>> Keep track of that information in the dev_archdata structure. As the
>>> structure is embedded in all instances of struct device let's not grow
>>> it, but turn the existing dma_coherent bool field into a bitfield that
>>> can be used for other purposes.
>>>
>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with deferred
>>> probing or error") Signed-off-by: Laurent Pinchart
>>> <laurent.pinchart+renesas@ideasonboard.com> ---
>>>
>>>  arch/arm/include/asm/device.h | 3 ++-
>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h
>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>> --- a/arch/arm/include/asm/device.h
>>> +++ b/arch/arm/include/asm/device.h
>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>  #ifdef CONFIG_XEN
>>>  	const struct dma_map_ops *dev_dma_ops;
>>>  #endif
>>> -	bool dma_coherent;
>>> +	unsigned int dma_coherent:1;
>>
>> This should only ever be accessed by the Xen DMA code via the
>> is_device_dma_coherent() helper, so I can't see the change of storage
>> type causing any problems.
> 
> Thank you for double-checking. I agree with your analysis.
> 
>>> +	unsigned int dma_ops_setup:1;
>>>  };
>>>  
>>>  struct omap_device;
>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>> index c742dfd2967b..e0272f9140e2 100644
>>> --- a/arch/arm/mm/dma-mapping.c
>>> +++ b/arch/arm/mm/dma-mapping.c
>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, u64
>>> dma_base, u64 size,
>>>  		dev->dma_ops = xen_dma_ops;
>>>  	}
>>>  #endif
>>> +	dev->archdata.dma_ops_setup = true;
>>>  }
>>>  
>>>  void arch_teardown_dma_ops(struct device *dev)
>>>  {
>>> +	if (!dev->archdata.dma_ops_setup)
>>> +		return;
>>> +
>>>  	arm_teardown_iommu_dma_ops(dev);
>>> +	set_dma_ops(dev, NULL);
>>
>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>> it's down to the IOMMU driver so will never change after the first
>> probe, but it still feels like a bit of a nagging loose end.
> 
> To make a difference, we would need an IOMMU driver that creates a mapping 
> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() calls, 
> follow by a second round. I don't think this could happen, but if it did, I 
> believe we'd be screwed already, as there would be a time were an incorrect 
> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs to take 
> care of mapping creation) exists.
> 

Feels correct not to reset this, the iommu drivers in question, seems to
creating mapping/attaching in add_device path (which gets called before the
clients gets probed) and when the iommu client gets deferred/reprobed that
does not happen again even after the first round.

>> With that (or firm reassurance that it's OK not to),
>>
>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>
>> Apologies for being too arm64-focused in the earlier reviews and
>> overlooking this. Should the patch supersede 8674/1 currently in
>> Russell's incoming box?
> 
> Yes I think it should. Could you please take care of that ?
> 
> You can also add my
> was
> Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> as I've tested that this paptch restores proper IOMMU operation on the Renesas 
> R-Car H2 Lager board. I believe the problem related to Sricharan's patch 
> reported by Geert still affects us and needs to be addressed separately.

Thanks for the above, i had the same thing to be posted, was just testing it once.
There are three patches [1][2], already posted and third one for the issue that Geert
pointed i did below (Geert had a patch little differently to ignore -ENODEV).
I had this question previously for not propagating errors apart from EPROBE_DEFER,
did not have an issue reported at that time. Anyways if the below is ok, i will
just send the 3 patches in one set for easy picking up ?

[1] https://lkml.org/lkml/2017/5/16/25  
[2] The above one that you have. 
[3] The below one, if its fine ?

>From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
From: Sricharan R <sricharan@codeaurora.org>
Date: Wed, 3 May 2017 14:54:11 +0530
Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER

While deferring the probe of iommu masters,
xlate and add_device callback can passback error values
like -ENODEV, which means iommu cannot be connected
with that master for real reasons. So rather than
killing the master's probe for such errors, just
ignore the errors and let the master work without
an iommu.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index e6e9bec..750ab07 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
                        ops = ERR_PTR(err);
        }

+       /* Ignore all other errors apart from EPROBE_DEFER */
+       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
+               ops = NULL;
+
        return ops;
 }

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation



> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Regards,
 Sricharan

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Magnus Damm]

Hi Geert, everyone,

On Fri, May 5, 2017 at 10:23 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> Hi Sricharan, Robin,
>
> On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>
>>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>>> be handled separately from the .of_xlate() failures to support deferred
>>>>> probing.
>>>>>
>>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>>> been deferred, or having failed.
>>>>>
>>>>> The first case occurs when the device tree describes the bus master and
>>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>>> will configure the device without an IOMMU.
>>>>>
>>>>> The second and third cases are handled by deferring the probe of the bus
>>>>> master device which will eventually get reprobed after the IOMMU.
>>>>>
>>>>> The last case is currently handled by deferring the probe of the bus
>>>>> master device as well. A mechanism to either configure the bus master
>>>>> device without an IOMMU or to fail the bus master device probe depending
>>>>> on whether the IOMMU is optional or mandatory would be a good
>>>>> enhancement.
>>>>>
>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>>> properties in DT now fail to probe.
>>>
>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>>> registered then they should merely defer until we reach the point of
>>> giving up and ignoring the IOMMU. Is it just that you have no other
>>> late-probing drivers or post-init module loads to kick the deferred
>>> queue after that point? I did try to find a way to explicitly kick it
>>> from a suitably late initcall, but there didn't seem to be any obvious
>>> public interface - anyone have any suggestions?
>>>
>>> I think that's more of a general problem with the probe deferral
>>> mechanism itself (I've seen the same thing happen with some of the
>>> CoreSight stuff on Juno due to the number of inter-component
>>> dependencies) rather than any specific fault of this series.
>
> I had a deeper look into the issue.
>
> What changed, is that of_dma_configure() now returns an error code,
> and dma_configure() looks at it.
>
> Actually there are two failure modes:
>   1. Devices with an iommus property pointing to a disabled IOMMU node.
>      These return -EPROBE_DEFER, and are now retried forever.
>   2. Devices that are blacklisted in the IPMMU driver, as we don't want to
>      use them with an IOMMU yet.
>      These return -ENODEV, due to ipmmu_of_xlate_dma().
>
>> I was thinking of an additional check like below to avoid the
>> situation ?
>>
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>>
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
>
> Thanks, this fixes the first class of failures.
>
> The second class can be worked around using:
>
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -196,6 +196,11 @@ static const struct iommu_ops
>                 ops = of_iommu_xlate(dev, &iommu_spec);
>                 of_node_put(iommu_spec.np);
>                 idx++;
> +               if (PTR_ERR(ops) == -ENODEV) {
> +                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
> +                                __func__);
> +                       return NULL;
> +               }
>                 if (IS_ERR_OR_NULL(ops))
>                         break;
>         }
>
> but obviously that's too hackish to apply...
> Magnus, do you have a suggestion?

Thanks for your efforts guys!

I've recently been working on up-porting the IPMMU patches and
addressing review comments. Now with my local driver stack on top of
v4.12-rc (a95cfad) I did not notice these issues initially since I
only tested devices with IPMMU enabled. Once these issues were pointed
out to me by Geert I have now reproduced the 64-bit ARM specific ones
on r8a7796 Salvator-X.

On my r8a7796 platform I'm using the following IOMMU and DMA Engine devices:

IOMMU device IPMMU-DS0 - Connected to SYS-DMAC0
IOMMU device IPMMU-DS1 - Connected to SYS-DMAC1 and SYS-DMAC2
IOMMU device IPMMU-MM - Root device serving IPMMU-DS0 and IPMMU-DS1

For testing the serial port SCIF1 is used with DMA Engine devices
SYS-DMAC1 or SYS-DMAC2.

The software environment is configured as follows:
- The DTS comes with all devices above enabled except IPMMU-DS0 which
comes with status = "disabled".
- The IPMMU driver is during run time only allowing use of SYS-DMAC2.
For other devices ->of_xlate() returns -ENODEV.

The above used to work just fine with v4.11 or earlier.

My observations for v4.12-rc:

1) Default state for a95cfad

The devices SYS-DMAC0 and SYS-DMAC1 will never probe. However SYS-DMAC2 is fine.

2) After applying "[PATCH] iommu: of: Fix check for returning EPROBE_DEFER" [1]

This fixes SYS-DMAC0 that now probes and operates without IPMMU-DS0 as
expected. Same as v4.11 or earlier.

3) After also applying "[PATCH] of: iommu: Ignore all errors except
EPROBE_DEFER" [2]

This fixes SYS-DMAC1 that now probes and operates without IPMMU-DS1 as
expected. Same as v4.11 or earlier.

With fix [1] and [2] things seem back to normal. Unless I'm mistaken
it also seems that [1] allows me to drop the similar patch "[PATCH/RFC
v2 1/4] iommu/of: Skip IOMMU devices disabled in DT".

Thanks for your help.

Cheers,

/ magnus


[1] https://lkml.org/lkml/2017/5/16/25
[2] https://www.spinics.net/lists/arm-kernel/msg581485.html
[3] https://patchwork.kernel.org/patch/9540605/

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Magnus Damm]

Hi Geert, everyone,

On Fri, May 5, 2017 at 10:23 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> Hi Sricharan, Robin,
>
> On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>
>>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>>> be handled separately from the .of_xlate() failures to support deferred
>>>>> probing.
>>>>>
>>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>>> been deferred, or having failed.
>>>>>
>>>>> The first case occurs when the device tree describes the bus master and
>>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>>> will configure the device without an IOMMU.
>>>>>
>>>>> The second and third cases are handled by deferring the probe of the bus
>>>>> master device which will eventually get reprobed after the IOMMU.
>>>>>
>>>>> The last case is currently handled by deferring the probe of the bus
>>>>> master device as well. A mechanism to either configure the bus master
>>>>> device without an IOMMU or to fail the bus master device probe depending
>>>>> on whether the IOMMU is optional or mandatory would be a good
>>>>> enhancement.
>>>>>
>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>>> properties in DT now fail to probe.
>>>
>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>>> registered then they should merely defer until we reach the point of
>>> giving up and ignoring the IOMMU. Is it just that you have no other
>>> late-probing drivers or post-init module loads to kick the deferred
>>> queue after that point? I did try to find a way to explicitly kick it
>>> from a suitably late initcall, but there didn't seem to be any obvious
>>> public interface - anyone have any suggestions?
>>>
>>> I think that's more of a general problem with the probe deferral
>>> mechanism itself (I've seen the same thing happen with some of the
>>> CoreSight stuff on Juno due to the number of inter-component
>>> dependencies) rather than any specific fault of this series.
>
> I had a deeper look into the issue.
>
> What changed, is that of_dma_configure() now returns an error code,
> and dma_configure() looks at it.
>
> Actually there are two failure modes:
>   1. Devices with an iommus property pointing to a disabled IOMMU node.
>      These return -EPROBE_DEFER, and are now retried forever.
>   2. Devices that are blacklisted in the IPMMU driver, as we don't want to
>      use them with an IOMMU yet.
>      These return -ENODEV, due to ipmmu_of_xlate_dma().
>
>> I was thinking of an additional check like below to avoid the
>> situation ?
>>
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>>
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
>
> Thanks, this fixes the first class of failures.
>
> The second class can be worked around using:
>
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -196,6 +196,11 @@ static const struct iommu_ops
>                 ops = of_iommu_xlate(dev, &iommu_spec);
>                 of_node_put(iommu_spec.np);
>                 idx++;
> +               if (PTR_ERR(ops) == -ENODEV) {
> +                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
> +                                __func__);
> +                       return NULL;
> +               }
>                 if (IS_ERR_OR_NULL(ops))
>                         break;
>         }
>
> but obviously that's too hackish to apply...
> Magnus, do you have a suggestion?

Thanks for your efforts guys!

I've recently been working on up-porting the IPMMU patches and
addressing review comments. Now with my local driver stack on top of
v4.12-rc (a95cfad) I did not notice these issues initially since I
only tested devices with IPMMU enabled. Once these issues were pointed
out to me by Geert I have now reproduced the 64-bit ARM specific ones
on r8a7796 Salvator-X.

On my r8a7796 platform I'm using the following IOMMU and DMA Engine devices:

IOMMU device IPMMU-DS0 - Connected to SYS-DMAC0
IOMMU device IPMMU-DS1 - Connected to SYS-DMAC1 and SYS-DMAC2
IOMMU device IPMMU-MM - Root device serving IPMMU-DS0 and IPMMU-DS1

For testing the serial port SCIF1 is used with DMA Engine devices
SYS-DMAC1 or SYS-DMAC2.

The software environment is configured as follows:
- The DTS comes with all devices above enabled except IPMMU-DS0 which
comes with status = "disabled".
- The IPMMU driver is during run time only allowing use of SYS-DMAC2.
For other devices ->of_xlate() returns -ENODEV.

The above used to work just fine with v4.11 or earlier.

My observations for v4.12-rc:

1) Default state for a95cfad

The devices SYS-DMAC0 and SYS-DMAC1 will never probe. However SYS-DMAC2 is fine.

2) After applying "[PATCH] iommu: of: Fix check for returning EPROBE_DEFER" [1]

This fixes SYS-DMAC0 that now probes and operates without IPMMU-DS0 as
expected. Same as v4.11 or earlier.

3) After also applying "[PATCH] of: iommu: Ignore all errors except
EPROBE_DEFER" [2]

This fixes SYS-DMAC1 that now probes and operates without IPMMU-DS1 as
expected. Same as v4.11 or earlier.

With fix [1] and [2] things seem back to normal. Unless I'm mistaken
it also seems that [1] allows me to drop the similar patch "[PATCH/RFC
v2 1/4] iommu/of: Skip IOMMU devices disabled in DT".

Thanks for your help.

Cheers,

/ magnus


[1] https://lkml.org/lkml/2017/5/16/25
[2] https://www.spinics.net/lists/arm-kernel/msg581485.html
[3] https://patchwork.kernel.org/patch/9540605/

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Magnus Damm]

Hi Geert, everyone,

On Fri, May 5, 2017 at 10:23 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> Hi Sricharan, Robin,
>
> On Wed, May 3, 2017 at 12:24 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>> On 5/3/2017 3:24 PM, Robin Murphy wrote:
>>> On 02/05/17 19:35, Geert Uytterhoeven wrote:
>>>> On Fri, Feb 3, 2017 at 4:48 PM, Sricharan R <sricharan@codeaurora.org> wrote:
>>>>> From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>>>>>
>>>>> Failures to look up an IOMMU when parsing the DT iommus property need to
>>>>> be handled separately from the .of_xlate() failures to support deferred
>>>>> probing.
>>>>>
>>>>> The lack of a registered IOMMU can be caused by the lack of a driver for
>>>>> the IOMMU, the IOMMU device probe not having been performed yet, having
>>>>> been deferred, or having failed.
>>>>>
>>>>> The first case occurs when the device tree describes the bus master and
>>>>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>>>>> or the device driver has not been compiled in. Return NULL, the caller
>>>>> will configure the device without an IOMMU.
>>>>>
>>>>> The second and third cases are handled by deferring the probe of the bus
>>>>> master device which will eventually get reprobed after the IOMMU.
>>>>>
>>>>> The last case is currently handled by deferring the probe of the bus
>>>>> master device as well. A mechanism to either configure the bus master
>>>>> device without an IOMMU or to fail the bus master device probe depending
>>>>> on whether the IOMMU is optional or mandatory would be a good
>>>>> enhancement.
>>>>>
>>>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>>> Signed-off-by: Laurent Pichart <laurent.pinchart+renesas@ideasonboard.com>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> This patch broke Renesas R-Car Gen3 platforms in renesas-drivers.
>>>> As the IOMMU nodes in DT are not yet enabled, all devices having iommus
>>>> properties in DT now fail to probe.
>>>
>>> How exactly do they fail to probe? Per d7b0558230e4, if there are no ops
>>> registered then they should merely defer until we reach the point of
>>> giving up and ignoring the IOMMU. Is it just that you have no other
>>> late-probing drivers or post-init module loads to kick the deferred
>>> queue after that point? I did try to find a way to explicitly kick it
>>> from a suitably late initcall, but there didn't seem to be any obvious
>>> public interface - anyone have any suggestions?
>>>
>>> I think that's more of a general problem with the probe deferral
>>> mechanism itself (I've seen the same thing happen with some of the
>>> CoreSight stuff on Juno due to the number of inter-component
>>> dependencies) rather than any specific fault of this series.
>
> I had a deeper look into the issue.
>
> What changed, is that of_dma_configure() now returns an error code,
> and dma_configure() looks at it.
>
> Actually there are two failure modes:
>   1. Devices with an iommus property pointing to a disabled IOMMU node.
>      These return -EPROBE_DEFER, and are now retried forever.
>   2. Devices that are blacklisted in the IPMMU driver, as we don't want to
>      use them with an IOMMU yet.
>      These return -ENODEV, due to ipmmu_of_xlate_dma().
>
>> I was thinking of an additional check like below to avoid the
>> situation ?
>>
>> From 499b6e662f60f23740b8880882b0a16f16434501 Mon Sep 17 00:00:00 2001
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Wed, 3 May 2017 13:16:59 +0530
>> Subject: [PATCH] iommu: of: Fix check for returning EPROBE_DEFER
>>
>> While returning EPROBE_DEFER for iommu masters
>> take in to account of iommu nodes that could be
>> marked in DT as 'status=disabled', in which case
>> simply return NULL and let the master's probe
>> continue rather than deferring.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/of_iommu.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 9f44ee8..e6e9bec 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -118,6 +118,7 @@ static bool of_iommu_driver_present(struct device_node *np)
>>
>>         ops = iommu_ops_from_fwnode(fwnode);
>>         if ((ops && !ops->of_xlate) ||
>> +           !of_device_is_available(iommu_spec->np) ||
>>             (!ops && !of_iommu_driver_present(iommu_spec->np)))
>>                 return NULL;
>
> Thanks, this fixes the first class of failures.
>
> The second class can be worked around using:
>
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -196,6 +196,11 @@ static const struct iommu_ops
>                 ops = of_iommu_xlate(dev, &iommu_spec);
>                 of_node_put(iommu_spec.np);
>                 idx++;
> +               if (PTR_ERR(ops) == -ENODEV) {
> +                       dev_info(dev, "%s: Ignoring -ENODEV => NULL\n",
> +                                __func__);
> +                       return NULL;
> +               }
>                 if (IS_ERR_OR_NULL(ops))
>                         break;
>         }
>
> but obviously that's too hackish to apply...
> Magnus, do you have a suggestion?

Thanks for your efforts guys!

I've recently been working on up-porting the IPMMU patches and
addressing review comments. Now with my local driver stack on top of
v4.12-rc (a95cfad) I did not notice these issues initially since I
only tested devices with IPMMU enabled. Once these issues were pointed
out to me by Geert I have now reproduced the 64-bit ARM specific ones
on r8a7796 Salvator-X.

On my r8a7796 platform I'm using the following IOMMU and DMA Engine devices:

IOMMU device IPMMU-DS0 - Connected to SYS-DMAC0
IOMMU device IPMMU-DS1 - Connected to SYS-DMAC1 and SYS-DMAC2
IOMMU device IPMMU-MM - Root device serving IPMMU-DS0 and IPMMU-DS1

For testing the serial port SCIF1 is used with DMA Engine devices
SYS-DMAC1 or SYS-DMAC2.

The software environment is configured as follows:
- The DTS comes with all devices above enabled except IPMMU-DS0 which
comes with status = "disabled".
- The IPMMU driver is during run time only allowing use of SYS-DMAC2.
For other devices ->of_xlate() returns -ENODEV.

The above used to work just fine with v4.11 or earlier.

My observations for v4.12-rc:

1) Default state for a95cfad

The devices SYS-DMAC0 and SYS-DMAC1 will never probe. However SYS-DMAC2 is fine.

2) After applying "[PATCH] iommu: of: Fix check for returning EPROBE_DEFER" [1]

This fixes SYS-DMAC0 that now probes and operates without IPMMU-DS0 as
expected. Same as v4.11 or earlier.

3) After also applying "[PATCH] of: iommu: Ignore all errors except
EPROBE_DEFER" [2]

This fixes SYS-DMAC1 that now probes and operates without IPMMU-DS1 as
expected. Same as v4.11 or earlier.

With fix [1] and [2] things seem back to normal. Unless I'm mistaken
it also seems that [1] allows me to drop the similar patch "[PATCH/RFC
v2 1/4] iommu/of: Skip IOMMU devices disabled in DT".

Thanks for your help.

Cheers,

/ magnus


[1] https://lkml.org/lkml/2017/5/16/25
[2] https://www.spinics.net/lists/arm-kernel/msg581485.html
[3] https://patchwork.kernel.org/patch/9540605/

Re: [PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Magnus,

<snip..>

>> Magnus, do you have a suggestion?
> 
> Thanks for your efforts guys!
> 
> I've recently been working on up-porting the IPMMU patches and
> addressing review comments. Now with my local driver stack on top of
> v4.12-rc (a95cfad) I did not notice these issues initially since I
> only tested devices with IPMMU enabled. Once these issues were pointed
> out to me by Geert I have now reproduced the 64-bit ARM specific ones
> on r8a7796 Salvator-X.
> 
> On my r8a7796 platform I'm using the following IOMMU and DMA Engine devices:
> 
> IOMMU device IPMMU-DS0 - Connected to SYS-DMAC0
> IOMMU device IPMMU-DS1 - Connected to SYS-DMAC1 and SYS-DMAC2
> IOMMU device IPMMU-MM - Root device serving IPMMU-DS0 and IPMMU-DS1
> 
> For testing the serial port SCIF1 is used with DMA Engine devices
> SYS-DMAC1 or SYS-DMAC2.
> 
> The software environment is configured as follows:
> - The DTS comes with all devices above enabled except IPMMU-DS0 which
> comes with status = "disabled".
> - The IPMMU driver is during run time only allowing use of SYS-DMAC2.
> For other devices ->of_xlate() returns -ENODEV.
> 
> The above used to work just fine with v4.11 or earlier.
> 
> My observations for v4.12-rc:
> 
> 1) Default state for a95cfad
> 
> The devices SYS-DMAC0 and SYS-DMAC1 will never probe. However SYS-DMAC2 is fine.
> 
> 2) After applying "[PATCH] iommu: of: Fix check for returning EPROBE_DEFER" [1]
> 
> This fixes SYS-DMAC0 that now probes and operates without IPMMU-DS0 as
> expected. Same as v4.11 or earlier.
> 
> 3) After also applying "[PATCH] of: iommu: Ignore all errors except
> EPROBE_DEFER" [2]
> 
> This fixes SYS-DMAC1 that now probes and operates without IPMMU-DS1 as
> expected. Same as v4.11 or earlier.
> 
> With fix [1] and [2] things seem back to normal. Unless I'm mistaken

Thanks, will use this as your Tested-by.

Regards,
 Sricharan

> it also seems that [1] allows me to drop the similar patch "[PATCH/RFC
> v2 1/4] iommu/of: Skip IOMMU devices disabled in DT".
> 
> Thanks for your help.
> 
> Cheers,
> 
> / magnus
> 
> 
> [1] https://lkml.org/lkml/2017/5/16/25
> [2] https://www.spinics.net/lists/arm-kernel/msg581485.html
> [3] https://patchwork.kernel.org/patch/9540605/
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V8 07/11] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

Hi Magnus,

<snip..>

>> Magnus, do you have a suggestion?
> 
> Thanks for your efforts guys!
> 
> I've recently been working on up-porting the IPMMU patches and
> addressing review comments. Now with my local driver stack on top of
> v4.12-rc (a95cfad) I did not notice these issues initially since I
> only tested devices with IPMMU enabled. Once these issues were pointed
> out to me by Geert I have now reproduced the 64-bit ARM specific ones
> on r8a7796 Salvator-X.
> 
> On my r8a7796 platform I'm using the following IOMMU and DMA Engine devices:
> 
> IOMMU device IPMMU-DS0 - Connected to SYS-DMAC0
> IOMMU device IPMMU-DS1 - Connected to SYS-DMAC1 and SYS-DMAC2
> IOMMU device IPMMU-MM - Root device serving IPMMU-DS0 and IPMMU-DS1
> 
> For testing the serial port SCIF1 is used with DMA Engine devices
> SYS-DMAC1 or SYS-DMAC2.
> 
> The software environment is configured as follows:
> - The DTS comes with all devices above enabled except IPMMU-DS0 which
> comes with status = "disabled".
> - The IPMMU driver is during run time only allowing use of SYS-DMAC2.
> For other devices ->of_xlate() returns -ENODEV.
> 
> The above used to work just fine with v4.11 or earlier.
> 
> My observations for v4.12-rc:
> 
> 1) Default state for a95cfad
> 
> The devices SYS-DMAC0 and SYS-DMAC1 will never probe. However SYS-DMAC2 is fine.
> 
> 2) After applying "[PATCH] iommu: of: Fix check for returning EPROBE_DEFER" [1]
> 
> This fixes SYS-DMAC0 that now probes and operates without IPMMU-DS0 as
> expected. Same as v4.11 or earlier.
> 
> 3) After also applying "[PATCH] of: iommu: Ignore all errors except
> EPROBE_DEFER" [2]
> 
> This fixes SYS-DMAC1 that now probes and operates without IPMMU-DS1 as
> expected. Same as v4.11 or earlier.
> 
> With fix [1] and [2] things seem back to normal. Unless I'm mistaken

Thanks, will use this as your Tested-by.

Regards,
 Sricharan

> it also seems that [1] allows me to drop the similar patch "[PATCH/RFC
> v2 1/4] iommu/of: Skip IOMMU devices disabled in DT".
> 
> Thanks for your help.
> 
> Cheers,
> 
> / magnus
> 
> 
> [1] https://lkml.org/lkml/2017/5/16/25
> [2] https://www.spinics.net/lists/arm-kernel/msg581485.html
> [3] https://patchwork.kernel.org/patch/9540605/
> 

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[sricharan]

On 2017-05-17 10:45, Sricharan R wrote:
> Hi Laurent/Robin,
> 
> On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
>> Hi Robin,
>> 
>> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>>> IOMMU mapping and attach it to the device. The function assumes that 
>>>> the
>>>> device is attached to a device-specific IOMMU instance (or at least 
>>>> a
>>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>>> separate mapping for every device.
>>>> 
>>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>>> assumption is not true, and IOMMU mappings must be shared between
>>>> multiple devices. In those cases the IOMMU driver knows better than 
>>>> the
>>>> generic ARM dma-mapping layer and attaches mapping to devices 
>>>> manually
>>>> with arm_iommu_attach_device(), which sets the DMA ops for the 
>>>> device.
>>>> 
>>>> The arch_setup_dma_ops() function takes this into account and bails 
>>>> out
>>>> immediately if the device already has DMA ops assigned. However, the
>>>> corresponding arch_teardown_dma_ops() function, called from driver
>>>> unbind code paths (including probe deferral), will tear the mapping 
>>>> down
>>>> regardless of who created it. When the device is reprobed
>>>> arch_setup_dma_ops() will be called again but won't perform any
>>>> operation as the DMA ops will still be set.
>>>> 
>>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>>> However, we can't do so unconditionally, as then a new mapping would 
>>>> be
>>>> created by arch_setup_dma_ops() when the device is reprobed, 
>>>> regardless
>>>> of whether the device needs to share a mapping or not. We must thus 
>>>> keep
>>>> track of whether arch_setup_dma_ops() created the mapping, and only 
>>>> in
>>>> that case tear it down in arch_teardown_dma_ops().
>>>> 
>>>> Keep track of that information in the dev_archdata structure. As the
>>>> structure is embedded in all instances of struct device let's not 
>>>> grow
>>>> it, but turn the existing dma_coherent bool field into a bitfield 
>>>> that
>>>> can be used for other purposes.
>>>> 
>>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with 
>>>> deferred
>>>> probing or error") Signed-off-by: Laurent Pinchart
>>>> <laurent.pinchart+renesas@ideasonboard.com> ---
>>>> 
>>>>  arch/arm/include/asm/device.h | 3 ++-
>>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>> 
>>>> diff --git a/arch/arm/include/asm/device.h 
>>>> b/arch/arm/include/asm/device.h
>>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>>> --- a/arch/arm/include/asm/device.h
>>>> +++ b/arch/arm/include/asm/device.h
>>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>>  #ifdef CONFIG_XEN
>>>>  	const struct dma_map_ops *dev_dma_ops;
>>>>  #endif
>>>> -	bool dma_coherent;
>>>> +	unsigned int dma_coherent:1;
>>> 
>>> This should only ever be accessed by the Xen DMA code via the
>>> is_device_dma_coherent() helper, so I can't see the change of storage
>>> type causing any problems.
>> 
>> Thank you for double-checking. I agree with your analysis.
>> 
>>>> +	unsigned int dma_ops_setup:1;
>>>>  };
>>>> 
>>>>  struct omap_device;
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index c742dfd2967b..e0272f9140e2 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, 
>>>> u64
>>>> dma_base, u64 size,
>>>>  		dev->dma_ops = xen_dma_ops;
>>>>  	}
>>>>  #endif
>>>> +	dev->archdata.dma_ops_setup = true;
>>>>  }
>>>> 
>>>>  void arch_teardown_dma_ops(struct device *dev)
>>>>  {
>>>> +	if (!dev->archdata.dma_ops_setup)
>>>> +		return;
>>>> +
>>>>  	arm_teardown_iommu_dma_ops(dev);
>>>> +	set_dma_ops(dev, NULL);
>>> 
>>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>>> it's down to the IOMMU driver so will never change after the first
>>> probe, but it still feels like a bit of a nagging loose end.
>> 
>> To make a difference, we would need an IOMMU driver that creates a 
>> mapping
>> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() 
>> calls,
>> follow by a second round. I don't think this could happen, but if it 
>> did, I
>> believe we'd be screwed already, as there would be a time were an 
>> incorrect
>> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs 
>> to take
>> care of mapping creation) exists.
>> 
> 
> Feels correct not to reset this, the iommu drivers in question, seems 
> to
> creating mapping/attaching in add_device path (which gets called before 
> the
> clients gets probed) and when the iommu client gets deferred/reprobed 
> that
> does not happen again even after the first round.

Please ignore the above comment. I said that because I was doing the
dma_ops_setup in arm_iommu_attach_device. I posted the three fixes now 
[1].
Accidentally removed you from CC, sorry for that.
Applied those patches on top of 8674/1 that Robin mentioned
below. So removed setting set_dma_ops(dev, NULL) from your patch.

Also please note that, I changed the Fixes: commit msg in your patch to
("of/acpi: Configure dma operations at probe time for platform/amba/pci 
bus devices")
because that was one which started to invoke the teardown on the driver
release path.

[1] https://lkml.org/lkml/2017/5/17/344

Regards,
  Sricharan


> 
>>> With that (or firm reassurance that it's OK not to),
>>> 
>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>> 
>>> Apologies for being too arm64-focused in the earlier reviews and
>>> overlooking this. Should the patch supersede 8674/1 currently in
>>> Russell's incoming box?
>> 
>> Yes I think it should. Could you please take care of that ?
>> 
>> You can also add my
>> was
>> Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> 
>> as I've tested that this paptch restores proper IOMMU operation on the 
>> Renesas
>> R-Car H2 Lager board. I believe the problem related to Sricharan's 
>> patch
>> reported by Geert still affects us and needs to be addressed 
>> separately.
> 
> Thanks for the above, i had the same thing to be posted, was just
> testing it once.
> There are three patches [1][2], already posted and third one for the
> issue that Geert
> pointed i did below (Geert had a patch little differently to ignore 
> -ENODEV).
> I had this question previously for not propagating errors apart from
> EPROBE_DEFER,
> did not have an issue reported at that time. Anyways if the below is 
> ok, i will
> just send the 3 patches in one set for easy picking up ?
> 
> [1] https://lkml.org/lkml/2017/5/16/25
> [2] The above one that you have.
> [3] The below one, if its fine ?
> 
> From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 14:54:11 +0530
> Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER
> 
> While deferring the probe of iommu masters,
> xlate and add_device callback can passback error values
> like -ENODEV, which means iommu cannot be connected
> with that master for real reasons. So rather than
> killing the master's probe for such errors, just
> ignore the errors and let the master work without
> an iommu.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index e6e9bec..750ab07 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct
> device *dev,
>                         ops = ERR_PTR(err);
>         }
> 
> +       /* Ignore all other errors apart from EPROBE_DEFER */
> +       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
> +               ops = NULL;
> +
>         return ops;
>  }
> 
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
> member of Code Aurora Forum, hosted by The Linux Foundation
> 
> 
> 
>> 

Re: [PATCH] ARM: dma-mapping: Don't tear third-party mappings

[sricharan-sgV2jX0FEOL9JmXXK+q4OQ]

On 2017-05-17 10:45, Sricharan R wrote:
> Hi Laurent/Robin,
> 
> On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
>> Hi Robin,
>> 
>> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>>> IOMMU mapping and attach it to the device. The function assumes that 
>>>> the
>>>> device is attached to a device-specific IOMMU instance (or at least 
>>>> a
>>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>>> separate mapping for every device.
>>>> 
>>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>>> assumption is not true, and IOMMU mappings must be shared between
>>>> multiple devices. In those cases the IOMMU driver knows better than 
>>>> the
>>>> generic ARM dma-mapping layer and attaches mapping to devices 
>>>> manually
>>>> with arm_iommu_attach_device(), which sets the DMA ops for the 
>>>> device.
>>>> 
>>>> The arch_setup_dma_ops() function takes this into account and bails 
>>>> out
>>>> immediately if the device already has DMA ops assigned. However, the
>>>> corresponding arch_teardown_dma_ops() function, called from driver
>>>> unbind code paths (including probe deferral), will tear the mapping 
>>>> down
>>>> regardless of who created it. When the device is reprobed
>>>> arch_setup_dma_ops() will be called again but won't perform any
>>>> operation as the DMA ops will still be set.
>>>> 
>>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>>> However, we can't do so unconditionally, as then a new mapping would 
>>>> be
>>>> created by arch_setup_dma_ops() when the device is reprobed, 
>>>> regardless
>>>> of whether the device needs to share a mapping or not. We must thus 
>>>> keep
>>>> track of whether arch_setup_dma_ops() created the mapping, and only 
>>>> in
>>>> that case tear it down in arch_teardown_dma_ops().
>>>> 
>>>> Keep track of that information in the dev_archdata structure. As the
>>>> structure is embedded in all instances of struct device let's not 
>>>> grow
>>>> it, but turn the existing dma_coherent bool field into a bitfield 
>>>> that
>>>> can be used for other purposes.
>>>> 
>>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with 
>>>> deferred
>>>> probing or error") Signed-off-by: Laurent Pinchart
>>>> <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org> ---
>>>> 
>>>>  arch/arm/include/asm/device.h | 3 ++-
>>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>> 
>>>> diff --git a/arch/arm/include/asm/device.h 
>>>> b/arch/arm/include/asm/device.h
>>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>>> --- a/arch/arm/include/asm/device.h
>>>> +++ b/arch/arm/include/asm/device.h
>>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>>  #ifdef CONFIG_XEN
>>>>  	const struct dma_map_ops *dev_dma_ops;
>>>>  #endif
>>>> -	bool dma_coherent;
>>>> +	unsigned int dma_coherent:1;
>>> 
>>> This should only ever be accessed by the Xen DMA code via the
>>> is_device_dma_coherent() helper, so I can't see the change of storage
>>> type causing any problems.
>> 
>> Thank you for double-checking. I agree with your analysis.
>> 
>>>> +	unsigned int dma_ops_setup:1;
>>>>  };
>>>> 
>>>>  struct omap_device;
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index c742dfd2967b..e0272f9140e2 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, 
>>>> u64
>>>> dma_base, u64 size,
>>>>  		dev->dma_ops = xen_dma_ops;
>>>>  	}
>>>>  #endif
>>>> +	dev->archdata.dma_ops_setup = true;
>>>>  }
>>>> 
>>>>  void arch_teardown_dma_ops(struct device *dev)
>>>>  {
>>>> +	if (!dev->archdata.dma_ops_setup)
>>>> +		return;
>>>> +
>>>>  	arm_teardown_iommu_dma_ops(dev);
>>>> +	set_dma_ops(dev, NULL);
>>> 
>>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>>> it's down to the IOMMU driver so will never change after the first
>>> probe, but it still feels like a bit of a nagging loose end.
>> 
>> To make a difference, we would need an IOMMU driver that creates a 
>> mapping
>> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() 
>> calls,
>> follow by a second round. I don't think this could happen, but if it 
>> did, I
>> believe we'd be screwed already, as there would be a time were an 
>> incorrect
>> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs 
>> to take
>> care of mapping creation) exists.
>> 
> 
> Feels correct not to reset this, the iommu drivers in question, seems 
> to
> creating mapping/attaching in add_device path (which gets called before 
> the
> clients gets probed) and when the iommu client gets deferred/reprobed 
> that
> does not happen again even after the first round.

Please ignore the above comment. I said that because I was doing the
dma_ops_setup in arm_iommu_attach_device. I posted the three fixes now 
[1].
Accidentally removed you from CC, sorry for that.
Applied those patches on top of 8674/1 that Robin mentioned
below. So removed setting set_dma_ops(dev, NULL) from your patch.

Also please note that, I changed the Fixes: commit msg in your patch to
("of/acpi: Configure dma operations at probe time for platform/amba/pci 
bus devices")
because that was one which started to invoke the teardown on the driver
release path.

[1] https://lkml.org/lkml/2017/5/17/344

Regards,
  Sricharan


> 
>>> With that (or firm reassurance that it's OK not to),
>>> 
>>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>>> 
>>> Apologies for being too arm64-focused in the earlier reviews and
>>> overlooking this. Should the patch supersede 8674/1 currently in
>>> Russell's incoming box?
>> 
>> Yes I think it should. Could you please take care of that ?
>> 
>> You can also add my
>> was
>> Tested-by: Laurent Pinchart <laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>> 
>> as I've tested that this paptch restores proper IOMMU operation on the 
>> Renesas
>> R-Car H2 Lager board. I believe the problem related to Sricharan's 
>> patch
>> reported by Geert still affects us and needs to be addressed 
>> separately.
> 
> Thanks for the above, i had the same thing to be posted, was just
> testing it once.
> There are three patches [1][2], already posted and third one for the
> issue that Geert
> pointed i did below (Geert had a patch little differently to ignore 
> -ENODEV).
> I had this question previously for not propagating errors apart from
> EPROBE_DEFER,
> did not have an issue reported at that time. Anyways if the below is 
> ok, i will
> just send the 3 patches in one set for easy picking up ?
> 
> [1] https://lkml.org/lkml/2017/5/16/25
> [2] The above one that you have.
> [3] The below one, if its fine ?
> 
> From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Wed, 3 May 2017 14:54:11 +0530
> Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER
> 
> While deferring the probe of iommu masters,
> xlate and add_device callback can passback error values
> like -ENODEV, which means iommu cannot be connected
> with that master for real reasons. So rather than
> killing the master's probe for such errors, just
> ignore the errors and let the master work without
> an iommu.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index e6e9bec..750ab07 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct
> device *dev,
>                         ops = ERR_PTR(err);
>         }
> 
> +       /* Ignore all other errors apart from EPROBE_DEFER */
> +       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
> +               ops = NULL;
> +
>         return ops;
>  }
> 
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
> member of Code Aurora Forum, hosted by The Linux Foundation
> 
> 
> 
>> 

[PATCH] ARM: dma-mapping: Don't tear third-party mappings

[sricharan at codeaurora.org]

On 2017-05-17 10:45, Sricharan R wrote:
> Hi Laurent/Robin,
> 
> On 5/16/2017 10:14 PM, Laurent Pinchart wrote:
>> Hi Robin,
>> 
>> On Tuesday 16 May 2017 16:47:36 Robin Murphy wrote:
>>> On 16/05/17 16:14, Laurent Pinchart wrote:
>>>> arch_setup_dma_ops() is used in device probe code paths to create an
>>>> IOMMU mapping and attach it to the device. The function assumes that 
>>>> the
>>>> device is attached to a device-specific IOMMU instance (or at least 
>>>> a
>>>> device-specific TLB in a shared IOMMU instance) and thus creates a
>>>> separate mapping for every device.
>>>> 
>>>> On several systems (Renesas R-Car Gen2 being one of them), that
>>>> assumption is not true, and IOMMU mappings must be shared between
>>>> multiple devices. In those cases the IOMMU driver knows better than 
>>>> the
>>>> generic ARM dma-mapping layer and attaches mapping to devices 
>>>> manually
>>>> with arm_iommu_attach_device(), which sets the DMA ops for the 
>>>> device.
>>>> 
>>>> The arch_setup_dma_ops() function takes this into account and bails 
>>>> out
>>>> immediately if the device already has DMA ops assigned. However, the
>>>> corresponding arch_teardown_dma_ops() function, called from driver
>>>> unbind code paths (including probe deferral), will tear the mapping 
>>>> down
>>>> regardless of who created it. When the device is reprobed
>>>> arch_setup_dma_ops() will be called again but won't perform any
>>>> operation as the DMA ops will still be set.
>>>> 
>>>> We need to reset the DMA ops in arch_teardown_dma_ops() to fix this.
>>>> However, we can't do so unconditionally, as then a new mapping would 
>>>> be
>>>> created by arch_setup_dma_ops() when the device is reprobed, 
>>>> regardless
>>>> of whether the device needs to share a mapping or not. We must thus 
>>>> keep
>>>> track of whether arch_setup_dma_ops() created the mapping, and only 
>>>> in
>>>> that case tear it down in arch_teardown_dma_ops().
>>>> 
>>>> Keep track of that information in the dev_archdata structure. As the
>>>> structure is embedded in all instances of struct device let's not 
>>>> grow
>>>> it, but turn the existing dma_coherent bool field into a bitfield 
>>>> that
>>>> can be used for other purposes.
>>>> 
>>>> Fixes: 7b07cbefb68d ("iommu: of: Handle IOMMU lookup failure with 
>>>> deferred
>>>> probing or error") Signed-off-by: Laurent Pinchart
>>>> <laurent.pinchart+renesas@ideasonboard.com> ---
>>>> 
>>>>  arch/arm/include/asm/device.h | 3 ++-
>>>>  arch/arm/mm/dma-mapping.c     | 5 +++++
>>>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>>> 
>>>> diff --git a/arch/arm/include/asm/device.h 
>>>> b/arch/arm/include/asm/device.h
>>>> index 36ec9c8f6e16..3234fe9bba6e 100644
>>>> --- a/arch/arm/include/asm/device.h
>>>> +++ b/arch/arm/include/asm/device.h
>>>> @@ -19,7 +19,8 @@ struct dev_archdata {
>>>>  #ifdef CONFIG_XEN
>>>>  	const struct dma_map_ops *dev_dma_ops;
>>>>  #endif
>>>> -	bool dma_coherent;
>>>> +	unsigned int dma_coherent:1;
>>> 
>>> This should only ever be accessed by the Xen DMA code via the
>>> is_device_dma_coherent() helper, so I can't see the change of storage
>>> type causing any problems.
>> 
>> Thank you for double-checking. I agree with your analysis.
>> 
>>>> +	unsigned int dma_ops_setup:1;
>>>>  };
>>>> 
>>>>  struct omap_device;
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index c742dfd2967b..e0272f9140e2 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2430,9 +2430,14 @@ void arch_setup_dma_ops(struct device *dev, 
>>>> u64
>>>> dma_base, u64 size,
>>>>  		dev->dma_ops = xen_dma_ops;
>>>>  	}
>>>>  #endif
>>>> +	dev->archdata.dma_ops_setup = true;
>>>>  }
>>>> 
>>>>  void arch_teardown_dma_ops(struct device *dev)
>>>>  {
>>>> +	if (!dev->archdata.dma_ops_setup)
>>>> +		return;
>>>> +
>>>>  	arm_teardown_iommu_dma_ops(dev);
>>>> +	set_dma_ops(dev, NULL);
>>> 
>>> Should we clear dma_ops_setup here for symmetry? I guess in practice
>>> it's down to the IOMMU driver so will never change after the first
>>> probe, but it still feels like a bit of a nagging loose end.
>> 
>> To make a difference, we would need an IOMMU driver that creates a 
>> mapping
>> after a first round of arch_setup_dma_ops() / arch_teardown_dma_ops() 
>> calls,
>> follow by a second round. I don't think this could happen, but if it 
>> did, I
>> believe we'd be screwed already, as there would be a time were an 
>> incorrect
>> mapping (created by arch_setup_dma_ops() while the IOMMU driver needs 
>> to take
>> care of mapping creation) exists.
>> 
> 
> Feels correct not to reset this, the iommu drivers in question, seems 
> to
> creating mapping/attaching in add_device path (which gets called before 
> the
> clients gets probed) and when the iommu client gets deferred/reprobed 
> that
> does not happen again even after the first round.

Please ignore the above comment. I said that because I was doing the
dma_ops_setup in arm_iommu_attach_device. I posted the three fixes now 
[1].
Accidentally removed you from CC, sorry for that.
Applied those patches on top of 8674/1 that Robin mentioned
below. So removed setting set_dma_ops(dev, NULL) from your patch.

Also please note that, I changed the Fixes: commit msg in your patch to
("of/acpi: Configure dma operations at probe time for platform/amba/pci 
bus devices")
because that was one which started to invoke the teardown on the driver
release path.

[1] https://lkml.org/lkml/2017/5/17/344

Regards,
  Sricharan


> 
>>> With that (or firm reassurance that it's OK not to),
>>> 
>>> Reviewed-by: Robin Murphy <robin.murphy@arm.com>
>>> 
>>> Apologies for being too arm64-focused in the earlier reviews and
>>> overlooking this. Should the patch supersede 8674/1 currently in
>>> Russell's incoming box?
>> 
>> Yes I think it should. Could you please take care of that ?
>> 
>> You can also add my
>> was
>> Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> 
>> as I've tested that this paptch restores proper IOMMU operation on the 
>> Renesas
>> R-Car H2 Lager board. I believe the problem related to Sricharan's 
>> patch
>> reported by Geert still affects us and needs to be addressed 
>> separately.
> 
> Thanks for the above, i had the same thing to be posted, was just
> testing it once.
> There are three patches [1][2], already posted and third one for the
> issue that Geert
> pointed i did below (Geert had a patch little differently to ignore 
> -ENODEV).
> I had this question previously for not propagating errors apart from
> EPROBE_DEFER,
> did not have an issue reported at that time. Anyways if the below is 
> ok, i will
> just send the 3 patches in one set for easy picking up ?
> 
> [1] https://lkml.org/lkml/2017/5/16/25
> [2] The above one that you have.
> [3] The below one, if its fine ?
> 
> From 4b379d4b852c41d7b5904c9a9e53deda94039f0a Mon Sep 17 00:00:00 2001
> From: Sricharan R <sricharan@codeaurora.org>
> Date: Wed, 3 May 2017 14:54:11 +0530
> Subject: [PATCH] of: iommu: Ignore all errors except EPROBE_DEFER
> 
> While deferring the probe of iommu masters,
> xlate and add_device callback can passback error values
> like -ENODEV, which means iommu cannot be connected
> with that master for real reasons. So rather than
> killing the master's probe for such errors, just
> ignore the errors and let the master work without
> an iommu.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/iommu/of_iommu.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index e6e9bec..750ab07 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -237,6 +237,10 @@ const struct iommu_ops *of_iommu_configure(struct
> device *dev,
>                         ops = ERR_PTR(err);
>         }
> 
> +       /* Ignore all other errors apart from EPROBE_DEFER */
> +       if (IS_ERR(ops) && (PTR_ERR(ops) != -EPROBE_DEFER))
> +               ops = NULL;
> +
>         return ops;
>  }
> 
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
> member of Code Aurora Forum, hosted by The Linux Foundation
> 
> 
> 
>>