[PATCH V3 0/8] IOMMU probe deferral support

[PATCH V3 0/8] IOMMU probe deferral support

[Sricharan R]

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.


pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

 Similarly on the device/driver_unregister path __device_release_driver is
 called which inturn calls dma_deconfigure.

 If the ACPI bus code follows the same, we can add acpi_dma_configure
 at the same place as of_dma_configure.

 This series is based on the recently merged Generic DT bindings for
 PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy@arm.com [2]

 This time tested this with platform and pci device for probe deferral
 and reprobe on arm64 based platform. There is an issue on the cleanup
 path for arm64 though, where there is WARN_ON if the dma_ops is reset while
 device is attached to an domain in arch_teardown_dma_ops.
 But with iommu_groups created from the iommu driver, the device is always
 attached to a domain/default_domain. So so the WARN has to be removed/handled
 probably.

 Previous post of this series [3].

 [V3]
     * Removed the patch to split dma_masks/dma_ops configuration separately
       based on review comments that both masks and ops are required only
       during the device probe time.

     * Reworked the series based on Generic DT bindings series [2].

     * Added call to iommu's remove_device in the cleanup path for arm and arm64.

     * Removed the notifier trick in arm64 to handle early device registration.

     * Added reset of dma_ops in cleanup path for arm based on comments.

     * Fixed the pci_iommu_configure path and tested with PCI device as well.
 
     * Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

     * Fixed few other cosmetic comments.
  
 [V2]
     * Updated the Initial post to call dma_configure/deconfigure from generic code
 
     * Added iommu add_device callback from of_iommu_configure path

 [V1]
     * Initial post

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
[3] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg13941.html
[4] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg13940.html



Laurent Pinchart (4):
  arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
  of: dma: Move range size workaround to of_dma_get_range()
  of: dma: Make of_dma_deconfigure() public
  iommu: of: Handle IOMMU lookup failure with deferred probing or error

Sricharan R (4):
  drivers: platform: Configure dma operations at probe time
  arm: dma-mapping: Reset the device's dma_ops
  arm/arm64: dma-mapping: Call iommu's remove_device callback during
    device detach
  arm64: dma-mapping: Remove the notifier trick to handle early setting
    of dma_ops

 arch/arm/mm/dma-mapping.c   |  18 ++++++++
 arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
 drivers/base/dd.c           |  10 +++++
 drivers/base/dma-mapping.c  |  11 +++++
 drivers/iommu/of_iommu.c    |  47 +++++++++++++++++--
 drivers/of/address.c        |  20 ++++++++-
 drivers/of/device.c         |  34 +++++++-------
 drivers/of/platform.c       |   9 ----
 drivers/pci/probe.c         |   5 +--
 include/linux/dma-mapping.h |   3 ++
 include/linux/of_device.h   |   7 ++-
 11 files changed, 138 insertions(+), 133 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V3 1/8] arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>

The arch_setup_dma_ops() function is in charge of setting dma_ops with a
call to set_dma_ops(). set_dma_ops() is also called from

- highbank and mvebu bus notifiers
- dmabounce (to be replaced with swiotlb)
- arm_iommu_attach_device

(arm_iommu_attach_device is itself called from IOMMU and bus master
device drivers)

To allow the arch_setup_dma_ops() call to be moved from device add time
to device probe time we must ensure that dma_ops already setup by any of
the above callers will not be overridden.

Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
of_xlate and taking care of highbank and mvebu, the workaround should be
removed.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
---
 arch/arm/mm/dma-mapping.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index c6834c0..dde6514 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2322,6 +2322,15 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 	struct dma_map_ops *dma_ops;
 
 	dev->archdata.dma_coherent = coherent;
+
+	/*
+	 * Don't override the dma_ops if they have already been set. Ideally
+	 * this should be the only location where dma_ops are set, remove this
+	 * check when all other callers of set_dma_ops will have disappeared.
+	 */
+	if (dev->archdata.dma_ops)
+		return;
+
 	if (arm_setup_iommu_dma_ops(dev, dma_base, size, iommu))
 		dma_ops = arm_get_iommu_dma_map_ops(coherent);
 	else
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

 Similarly on the device/driver_unregister path __device_release_driver is
 called which inturn calls dma_deconfigure.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/base/dd.c           | 10 ++++++++++
 drivers/base/dma-mapping.c  | 11 +++++++++++
 drivers/of/platform.c       |  4 ----
 drivers/pci/probe.c         |  5 +----
 include/linux/dma-mapping.h |  3 +++
 5 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(dmam_free_noncoherent);
 
+int dma_configure(struct device *dev)
+{
+	return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+}
+
 #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
 
 static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
 {
 	struct device *bridge = pci_get_host_bridge_device(dev);
 
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
+	if (has_acpi_companion(bridge)) {
 		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
 		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
 
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 66533e1..2766dbe 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -656,6 +656,9 @@ dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

[Robin Murphy]

+Lorenzo

On 04/10/16 18:03, Sricharan R wrote:
> Configuring DMA ops at probe time will allow deferring device probe when
> the IOMMU isn't available yet. The dma_configure for the device is now called
> from the generic device_attach callback just before the bus/driver probe
> is called. This way, configuring the dma ops for the device would be called
> at the same place for all bus_types, hence the deferred probing mechanism
> should work for all buses as well.
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
>  Similarly on the device/driver_unregister path __device_release_driver is
>  called which inturn calls dma_deconfigure.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  drivers/base/dd.c           | 10 ++++++++++
>  drivers/base/dma-mapping.c  | 11 +++++++++++
>  drivers/of/platform.c       |  4 ----
>  drivers/pci/probe.c         |  5 +----
>  include/linux/dma-mapping.h |  3 +++
>  5 files changed, 25 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index 16688f5..cfebd48 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -19,6 +19,7 @@
>  
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/kthread.h>
>  #include <linux/wait.h>
> @@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	if (ret)
>  		goto pinctrl_bind_failed;
>  
> +	ret = dma_configure(dev);
> +	if (ret)
> +		goto dma_failed;
> +
>  	if (driver_sysfs_add(dev)) {
>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>  			__func__, dev_name(dev));
> @@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>  	goto done;
>  
>  probe_failed:
> +	dma_deconfigure(dev);
> +dma_failed:
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> @@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
>  			dev->bus->remove(dev);
>  		else if (drv->remove)
>  			drv->remove(dev);
> +
> +		dma_deconfigure(dev);
> +
>  		devres_release_all(dev);
>  		dev->driver = NULL;
>  		dev_set_drvdata(dev, NULL);
> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> index d799662..54e87f5 100644
> --- a/drivers/base/dma-mapping.c
> +++ b/drivers/base/dma-mapping.c
> @@ -10,6 +10,7 @@
>  #include <linux/dma-mapping.h>
>  #include <linux/export.h>
>  #include <linux/gfp.h>
> +#include <linux/of_device.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
>  }
>  EXPORT_SYMBOL(dmam_free_noncoherent);
>  
> +int dma_configure(struct device *dev)
> +{
> +	return of_dma_configure(dev, dev->of_node);
> +}
> +
> +void dma_deconfigure(struct device *dev)
> +{
> +	of_dma_deconfigure(dev);
> +}
> +
>  #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
>  
>  static void dmam_coherent_decl_release(struct device *dev, void *res)
> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> index 9cb7090..adbd77c 100644
> --- a/drivers/of/platform.c
> +++ b/drivers/of/platform.c
> @@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
>  
>  	dev->dev.bus = &platform_bus_type;
>  	dev->dev.platform_data = platform_data;
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>  
>  	if (of_device_add(dev) != 0) {
> -		of_dma_deconfigure(&dev->dev);
>  		platform_device_put(dev);
>  		goto err_clear_flag;
>  	}
> @@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>  		dev_set_name(&dev->dev, "%s", bus_id);
>  	else
>  		of_device_make_bus_id(&dev->dev);
> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>  
>  	/* Allow the HW Peripheral ID to be overridden */
>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> @@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>  		amba_device_unregister(to_amba_device(dev));
>  #endif
>  
> -	of_dma_deconfigure(dev);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>  	return 0;
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 93f280d..85c9553 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
>  {
>  	struct device *bridge = pci_get_host_bridge_device(dev);
>  
> -	if (IS_ENABLED(CONFIG_OF) &&
> -		bridge->parent && bridge->parent->of_node) {
> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> -	} else if (has_acpi_companion(bridge)) {
> +	if (has_acpi_companion(bridge)) {
>  		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>  		enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA
configuration at device add, after we've allegedly moved DMA
configuration to driver probe. Lorenzo, do you foresee any issues if
this probe-time of_dma_configure() path were to also multiplex
acpi_dma_configure() in future, such that everything would be back in
the same place eventually?

Conversely, is there actually any issue with leaving pci_dma_configure()
unchanged, and simply moving the call from pci_device_add() into
dma_configure()?

Robin.

> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 66533e1..2766dbe 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -656,6 +656,9 @@ dma_mark_declared_memory_occupied(struct device *dev,
>  }
>  #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
>  
> +int dma_configure(struct device *dev);
> +void dma_deconfigure(struct device *dev);
> +
>  /*
>   * Managed DMA API
>   */
> 

RE: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

[Sricharan]

Hi Robin,

>On 04/10/16 18:03, Sricharan R wrote:
>> Configuring DMA ops at probe time will allow deferring device probe when
>> the IOMMU isn't available yet. The dma_configure for the device is now called
>> from the generic device_attach callback just before the bus/driver probe
>> is called. This way, configuring the dma ops for the device would be called
>> at the same place for all bus_types, hence the deferred probing mechanism
>> should work for all buses as well.
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>>  Similarly on the device/driver_unregister path __device_release_driver is
>>  called which inturn calls dma_deconfigure.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/base/dd.c           | 10 ++++++++++
>>  drivers/base/dma-mapping.c  | 11 +++++++++++
>>  drivers/of/platform.c       |  4 ----
>>  drivers/pci/probe.c         |  5 +----
>>  include/linux/dma-mapping.h |  3 +++
>>  5 files changed, 25 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> index 16688f5..cfebd48 100644
>> --- a/drivers/base/dd.c
>> +++ b/drivers/base/dd.c
>> @@ -19,6 +19,7 @@
>>
>>  #include <linux/device.h>
>>  #include <linux/delay.h>
>> +#include <linux/dma-mapping.h>
>>  #include <linux/module.h>
>>  #include <linux/kthread.h>
>>  #include <linux/wait.h>
>> @@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	if (ret)
>>  		goto pinctrl_bind_failed;
>>
>> +	ret = dma_configure(dev);
>> +	if (ret)
>> +		goto dma_failed;
>> +
>>  	if (driver_sysfs_add(dev)) {
>>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>>  			__func__, dev_name(dev));
>> @@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>>  	goto done;
>>
>>  probe_failed:
>> +	dma_deconfigure(dev);
>> +dma_failed:
>>  	if (dev->bus)
>>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>> @@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
>>  			dev->bus->remove(dev);
>>  		else if (drv->remove)
>>  			drv->remove(dev);
>> +
>> +		dma_deconfigure(dev);
>> +
>>  		devres_release_all(dev);
>>  		dev->driver = NULL;
>>  		dev_set_drvdata(dev, NULL);
>> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> index d799662..54e87f5 100644
>> --- a/drivers/base/dma-mapping.c
>> +++ b/drivers/base/dma-mapping.c
>> @@ -10,6 +10,7 @@
>>  #include <linux/dma-mapping.h>
>>  #include <linux/export.h>
>>  #include <linux/gfp.h>
>> +#include <linux/of_device.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>
>> @@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
>>  }
>>  EXPORT_SYMBOL(dmam_free_noncoherent);
>>
>> +int dma_configure(struct device *dev)
>> +{
>> +	return of_dma_configure(dev, dev->of_node);
>> +}
>> +
>> +void dma_deconfigure(struct device *dev)
>> +{
>> +	of_dma_deconfigure(dev);
>> +}
>> +
>>  #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
>>
>>  static void dmam_coherent_decl_release(struct device *dev, void *res)
>> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
>> index 9cb7090..adbd77c 100644
>> --- a/drivers/of/platform.c
>> +++ b/drivers/of/platform.c
>> @@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
>>
>>  	dev->dev.bus = &platform_bus_type;
>>  	dev->dev.platform_data = platform_data;
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>>
>>  	if (of_device_add(dev) != 0) {
>> -		of_dma_deconfigure(&dev->dev);
>>  		platform_device_put(dev);
>>  		goto err_clear_flag;
>>  	}
>> @@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>>  		dev_set_name(&dev->dev, "%s", bus_id);
>>  	else
>>  		of_device_make_bus_id(&dev->dev);
>> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>>
>>  	/* Allow the HW Peripheral ID to be overridden */
>>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
>> @@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>>  		amba_device_unregister(to_amba_device(dev));
>>  #endif
>>
>> -	of_dma_deconfigure(dev);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>>  	return 0;
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index 93f280d..85c9553 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
>>  {
>>  	struct device *bridge = pci_get_host_bridge_device(dev);
>>
>> -	if (IS_ENABLED(CONFIG_OF) &&
>> -		bridge->parent && bridge->parent->of_node) {
>> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
>> -	} else if (has_acpi_companion(bridge)) {
>> +	if (has_acpi_companion(bridge)) {
>>  		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>>  		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
>
>It seems a bit awkward leaving pci_dma_configure here, doing DMA
>configuration at device add, after we've allegedly moved DMA
>configuration to driver probe. Lorenzo, do you foresee any issues if
>this probe-time of_dma_configure() path were to also multiplex
>acpi_dma_configure() in future, such that everything would be back in
>the same place eventually?
>
>Conversely, is there actually any issue with leaving pci_dma_configure()
>unchanged, and simply moving the call from pci_device_add() into
>dma_configure()?

Ya i removed only the CONFIG_OF part out of this, and was hoping that
the acpi configure can also be called from the dma_configure during
probe later. I did not have an ACPI based platform though.
 As you said, if that looks right to the ACPI world, it can be
moved out from here. I can try mergin series (after fixing the vfio errors
 part) with the ACPI IO port and see how it behaves.

Regards,
 Sricharan
  

Re: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

[Lorenzo Pieralisi]

On Wed, Oct 26, 2016 at 08:34:59PM +0530, Sricharan wrote:
> Hi Robin,
> 
> >On 04/10/16 18:03, Sricharan R wrote:
> >> Configuring DMA ops at probe time will allow deferring device probe when
> >> the IOMMU isn't available yet. The dma_configure for the device is now called
> >> from the generic device_attach callback just before the bus/driver probe
> >> is called. This way, configuring the dma ops for the device would be called
> >> at the same place for all bus_types, hence the deferred probing mechanism
> >> should work for all buses as well.
> >>
> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >>        |                         |
> >> pci_bus_add_device     (device_add/driver_register)
> >>        |                         |
> >> device_attach           device_initial_probe
> >>        |                         |
> >> __device_attach_driver    __device_attach_driver
> >>        |
> >> driver_probe_device
> >>        |
> >> really_probe
> >>        |
> >> dma_configure
> >>
> >>  Similarly on the device/driver_unregister path __device_release_driver is
> >>  called which inturn calls dma_deconfigure.
> >>
> >> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >> ---
> >>  drivers/base/dd.c           | 10 ++++++++++
> >>  drivers/base/dma-mapping.c  | 11 +++++++++++
> >>  drivers/of/platform.c       |  4 ----
> >>  drivers/pci/probe.c         |  5 +----
> >>  include/linux/dma-mapping.h |  3 +++
> >>  5 files changed, 25 insertions(+), 8 deletions(-)
> >>
> >> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> >> index 16688f5..cfebd48 100644
> >> --- a/drivers/base/dd.c
> >> +++ b/drivers/base/dd.c
> >> @@ -19,6 +19,7 @@
> >>
> >>  #include <linux/device.h>
> >>  #include <linux/delay.h>
> >> +#include <linux/dma-mapping.h>
> >>  #include <linux/module.h>
> >>  #include <linux/kthread.h>
> >>  #include <linux/wait.h>
> >> @@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
> >>  	if (ret)
> >>  		goto pinctrl_bind_failed;
> >>
> >> +	ret = dma_configure(dev);
> >> +	if (ret)
> >> +		goto dma_failed;
> >> +
> >>  	if (driver_sysfs_add(dev)) {
> >>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
> >>  			__func__, dev_name(dev));
> >> @@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
> >>  	goto done;
> >>
> >>  probe_failed:
> >> +	dma_deconfigure(dev);
> >> +dma_failed:
> >>  	if (dev->bus)
> >>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
> >>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> >> @@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
> >>  			dev->bus->remove(dev);
> >>  		else if (drv->remove)
> >>  			drv->remove(dev);
> >> +
> >> +		dma_deconfigure(dev);
> >> +
> >>  		devres_release_all(dev);
> >>  		dev->driver = NULL;
> >>  		dev_set_drvdata(dev, NULL);
> >> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
> >> index d799662..54e87f5 100644
> >> --- a/drivers/base/dma-mapping.c
> >> +++ b/drivers/base/dma-mapping.c
> >> @@ -10,6 +10,7 @@
> >>  #include <linux/dma-mapping.h>
> >>  #include <linux/export.h>
> >>  #include <linux/gfp.h>
> >> +#include <linux/of_device.h>
> >>  #include <linux/slab.h>
> >>  #include <linux/vmalloc.h>
> >>
> >> @@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
> >>  }
> >>  EXPORT_SYMBOL(dmam_free_noncoherent);
> >>
> >> +int dma_configure(struct device *dev)
> >> +{
> >> +	return of_dma_configure(dev, dev->of_node);
> >> +}
> >> +
> >> +void dma_deconfigure(struct device *dev)
> >> +{
> >> +	of_dma_deconfigure(dev);
> >> +}
> >> +
> >>  #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
> >>
> >>  static void dmam_coherent_decl_release(struct device *dev, void *res)
> >> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> >> index 9cb7090..adbd77c 100644
> >> --- a/drivers/of/platform.c
> >> +++ b/drivers/of/platform.c
> >> @@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
> >>
> >>  	dev->dev.bus = &platform_bus_type;
> >>  	dev->dev.platform_data = platform_data;
> >> -	of_dma_configure(&dev->dev, dev->dev.of_node);
> >>  	of_msi_configure(&dev->dev, dev->dev.of_node);
> >>
> >>  	if (of_device_add(dev) != 0) {
> >> -		of_dma_deconfigure(&dev->dev);
> >>  		platform_device_put(dev);
> >>  		goto err_clear_flag;
> >>  	}
> >> @@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
> >>  		dev_set_name(&dev->dev, "%s", bus_id);
> >>  	else
> >>  		of_device_make_bus_id(&dev->dev);
> >> -	of_dma_configure(&dev->dev, dev->dev.of_node);
> >>
> >>  	/* Allow the HW Peripheral ID to be overridden */
> >>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
> >> @@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
> >>  		amba_device_unregister(to_amba_device(dev));
> >>  #endif
> >>
> >> -	of_dma_deconfigure(dev);
> >>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
> >>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
> >>  	return 0;
> >> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> >> index 93f280d..85c9553 100644
> >> --- a/drivers/pci/probe.c
> >> +++ b/drivers/pci/probe.c
> >> @@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
> >>  {
> >>  	struct device *bridge = pci_get_host_bridge_device(dev);
> >>
> >> -	if (IS_ENABLED(CONFIG_OF) &&
> >> -		bridge->parent && bridge->parent->of_node) {
> >> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
> >> -	} else if (has_acpi_companion(bridge)) {
> >> +	if (has_acpi_companion(bridge)) {
> >>  		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
> >>  		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
> >
> >It seems a bit awkward leaving pci_dma_configure here, doing DMA
> >configuration at device add, after we've allegedly moved DMA
> >configuration to driver probe. Lorenzo, do you foresee any issues if
> >this probe-time of_dma_configure() path were to also multiplex
> >acpi_dma_configure() in future, such that everything would be back in
> >the same place eventually?
> >
> >Conversely, is there actually any issue with leaving pci_dma_configure()
> >unchanged, and simply moving the call from pci_device_add() into
> >dma_configure()?
> 
> Ya i removed only the CONFIG_OF part out of this, and was hoping that
> the acpi configure can also be called from the dma_configure during
> probe later. I did not have an ACPI based platform though.  As you
> said, if that looks right to the ACPI world, it can be moved out from
> here. I can try mergin series (after fixing the vfio errors part) with
> the ACPI IO port and see how it behaves.

I do not expect any issue from this change, as long as, as Robin said,
it is done consistently which means that I am not ok with this patch
as it stands (ie you should defer pci_dma_configure() in its entirety,
not just the DT bits, which obviously requires some ACPI testing too).

CC me in please in the next posting since this affects the ACPI probing
path too, we have to coordinate this series with my ACPI IORT SMMU
enablement patches:

https://lkml.org/lkml/2016/10/18/506

Thanks,
Lorenzo

RE: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

[Sricharan]

Hi Lorenzo,

>-----Original Message-----
>From: linux-arm-msm-owner@vger.kernel.org [mailto:linux-arm-msm-owner@vger.kernel.org] On Behalf Of Lorenzo Pieralisi
>Sent: Thursday, October 27, 2016 4:19 PM
>To: Sricharan <sricharan@codeaurora.org>
>Cc: 'Robin Murphy' <robin.murphy@arm.com>; will.deacon@arm.com; joro@8bytes.org; iommu@lists.linux-foundation.org; linux-
>arm-kernel@lists.infradead.org; linux-arm-msm@vger.kernel.org; laurent.pinchart@ideasonboard.com;
>m.szyprowski@samsung.com; tfiga@chromium.org; srinivas.kandagatla@linaro.org
>Subject: Re: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time
>
>On Wed, Oct 26, 2016 at 08:34:59PM +0530, Sricharan wrote:
>> Hi Robin,
>>
>> >On 04/10/16 18:03, Sricharan R wrote:
>> >> Configuring DMA ops at probe time will allow deferring device probe when
>> >> the IOMMU isn't available yet. The dma_configure for the device is now called
>> >> from the generic device_attach callback just before the bus/driver probe
>> >> is called. This way, configuring the dma ops for the device would be called
>> >> at the same place for all bus_types, hence the deferred probing mechanism
>> >> should work for all buses as well.
>> >>
>> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>> >>        |                         |
>> >> pci_bus_add_device     (device_add/driver_register)
>> >>        |                         |
>> >> device_attach           device_initial_probe
>> >>        |                         |
>> >> __device_attach_driver    __device_attach_driver
>> >>        |
>> >> driver_probe_device
>> >>        |
>> >> really_probe
>> >>        |
>> >> dma_configure
>> >>
>> >>  Similarly on the device/driver_unregister path __device_release_driver is
>> >>  called which inturn calls dma_deconfigure.
>> >>
>> >> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >> ---
>> >>  drivers/base/dd.c           | 10 ++++++++++
>> >>  drivers/base/dma-mapping.c  | 11 +++++++++++
>> >>  drivers/of/platform.c       |  4 ----
>> >>  drivers/pci/probe.c         |  5 +----
>> >>  include/linux/dma-mapping.h |  3 +++
>> >>  5 files changed, 25 insertions(+), 8 deletions(-)
>> >>
>> >> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> >> index 16688f5..cfebd48 100644
>> >> --- a/drivers/base/dd.c
>> >> +++ b/drivers/base/dd.c
>> >> @@ -19,6 +19,7 @@
>> >>
>> >>  #include <linux/device.h>
>> >>  #include <linux/delay.h>
>> >> +#include <linux/dma-mapping.h>
>> >>  #include <linux/module.h>
>> >>  #include <linux/kthread.h>
>> >>  #include <linux/wait.h>
>> >> @@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>> >>  	if (ret)
>> >>  		goto pinctrl_bind_failed;
>> >>
>> >> +	ret = dma_configure(dev);
>> >> +	if (ret)
>> >> +		goto dma_failed;
>> >> +
>> >>  	if (driver_sysfs_add(dev)) {
>> >>  		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
>> >>  			__func__, dev_name(dev));
>> >> @@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
>> >>  	goto done;
>> >>
>> >>  probe_failed:
>> >> +	dma_deconfigure(dev);
>> >> +dma_failed:
>> >>  	if (dev->bus)
>> >>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>> >>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>> >> @@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
>> >>  			dev->bus->remove(dev);
>> >>  		else if (drv->remove)
>> >>  			drv->remove(dev);
>> >> +
>> >> +		dma_deconfigure(dev);
>> >> +
>> >>  		devres_release_all(dev);
>> >>  		dev->driver = NULL;
>> >>  		dev_set_drvdata(dev, NULL);
>> >> diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
>> >> index d799662..54e87f5 100644
>> >> --- a/drivers/base/dma-mapping.c
>> >> +++ b/drivers/base/dma-mapping.c
>> >> @@ -10,6 +10,7 @@
>> >>  #include <linux/dma-mapping.h>
>> >>  #include <linux/export.h>
>> >>  #include <linux/gfp.h>
>> >> +#include <linux/of_device.h>
>> >>  #include <linux/slab.h>
>> >>  #include <linux/vmalloc.h>
>> >>
>> >> @@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
>> >>  }
>> >>  EXPORT_SYMBOL(dmam_free_noncoherent);
>> >>
>> >> +int dma_configure(struct device *dev)
>> >> +{
>> >> +	return of_dma_configure(dev, dev->of_node);
>> >> +}
>> >> +
>> >> +void dma_deconfigure(struct device *dev)
>> >> +{
>> >> +	of_dma_deconfigure(dev);
>> >> +}
>> >> +
>> >>  #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
>> >>
>> >>  static void dmam_coherent_decl_release(struct device *dev, void *res)
>> >> diff --git a/drivers/of/platform.c b/drivers/of/platform.c
>> >> index 9cb7090..adbd77c 100644
>> >> --- a/drivers/of/platform.c
>> >> +++ b/drivers/of/platform.c
>> >> @@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
>> >>
>> >>  	dev->dev.bus = &platform_bus_type;
>> >>  	dev->dev.platform_data = platform_data;
>> >> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>> >>  	of_msi_configure(&dev->dev, dev->dev.of_node);
>> >>
>> >>  	if (of_device_add(dev) != 0) {
>> >> -		of_dma_deconfigure(&dev->dev);
>> >>  		platform_device_put(dev);
>> >>  		goto err_clear_flag;
>> >>  	}
>> >> @@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
>> >>  		dev_set_name(&dev->dev, "%s", bus_id);
>> >>  	else
>> >>  		of_device_make_bus_id(&dev->dev);
>> >> -	of_dma_configure(&dev->dev, dev->dev.of_node);
>> >>
>> >>  	/* Allow the HW Peripheral ID to be overridden */
>> >>  	prop = of_get_property(node, "arm,primecell-periphid", NULL);
>> >> @@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
>> >>  		amba_device_unregister(to_amba_device(dev));
>> >>  #endif
>> >>
>> >> -	of_dma_deconfigure(dev);
>> >>  	of_node_clear_flag(dev->of_node, OF_POPULATED);
>> >>  	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
>> >>  	return 0;
>> >> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> >> index 93f280d..85c9553 100644
>> >> --- a/drivers/pci/probe.c
>> >> +++ b/drivers/pci/probe.c
>> >> @@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
>> >>  {
>> >>  	struct device *bridge = pci_get_host_bridge_device(dev);
>> >>
>> >> -	if (IS_ENABLED(CONFIG_OF) &&
>> >> -		bridge->parent && bridge->parent->of_node) {
>> >> -			of_dma_configure(&dev->dev, bridge->parent->of_node);
>> >> -	} else if (has_acpi_companion(bridge)) {
>> >> +	if (has_acpi_companion(bridge)) {
>> >>  		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
>> >>  		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
>> >
>> >It seems a bit awkward leaving pci_dma_configure here, doing DMA
>> >configuration at device add, after we've allegedly moved DMA
>> >configuration to driver probe. Lorenzo, do you foresee any issues if
>> >this probe-time of_dma_configure() path were to also multiplex
>> >acpi_dma_configure() in future, such that everything would be back in
>> >the same place eventually?
>> >
>> >Conversely, is there actually any issue with leaving pci_dma_configure()
>> >unchanged, and simply moving the call from pci_device_add() into
>> >dma_configure()?
>>
>> Ya i removed only the CONFIG_OF part out of this, and was hoping that
>> the acpi configure can also be called from the dma_configure during
>> probe later. I did not have an ACPI based platform though.  As you
>> said, if that looks right to the ACPI world, it can be moved out from
>> here. I can try mergin series (after fixing the vfio errors part) with
>> the ACPI IO port and see how it behaves.
>
>I do not expect any issue from this change, as long as, as Robin said,
>it is done consistently which means that I am not ok with this patch
>as it stands (ie you should defer pci_dma_configure() in its entirety,
>not just the DT bits, which obviously requires some ACPI testing too).
>
>CC me in please in the next posting since this affects the ACPI probing
>path too, we have to coordinate this series with my ACPI IORT SMMU
>enablement patches:
>
>https://lkml.org/lkml/2016/10/18/506

Ok thanks, i will have you in looped for the V4 i am posting
and we can have it tested on ACPI as well.

Regards,
 Sricharan

[PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Sricharan R]

The dma_ops for the device is not getting set to NULL in
arch_tear_down_dma_ops and this causes an issue when the
device's probe gets deferred and retried. So reset the
dma_ops to NULL.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 arch/arm/mm/dma-mapping.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index dde6514..b9191f0 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2295,6 +2295,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)
 
 	__arm_iommu_detach_device(dev);
 	arm_iommu_release_mapping(mapping);
+	set_dma_ops(dev, NULL);
 }
 
 #else
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Robin Murphy]

On 04/10/16 18:03, Sricharan R wrote:
> The dma_ops for the device is not getting set to NULL in
> arch_tear_down_dma_ops and this causes an issue when the
> device's probe gets deferred and retried. So reset the
> dma_ops to NULL.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms,
i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
thus clearing the ops set by the latter, and sending it to Russell as a
fix in its own right.

Robin.

> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> ---
>  arch/arm/mm/dma-mapping.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index dde6514..b9191f0 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2295,6 +2295,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)
>  
>  	__arm_iommu_detach_device(dev);
>  	arm_iommu_release_mapping(mapping);
> +	set_dma_ops(dev, NULL);
>  }
>  
>  #else
> 

RE: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Sricharan]

Hi Robin,

>-----Original Message-----
>From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
>Sent: Wednesday, October 26, 2016 8:37 PM
>To: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>; will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
>kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
>tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
>Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops
>
>On 04/10/16 18:03, Sricharan R wrote:
>> The dma_ops for the device is not getting set to NULL in
>> arch_tear_down_dma_ops and this causes an issue when the
>> device's probe gets deferred and retried. So reset the
>> dma_ops to NULL.
>
>Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>

 Thanks.

>This seems like it could stand independently from the rest of the series
>- might be worth rewording the commit message to more general terms,
>i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>thus clearing the ops set by the latter, and sending it to Russell as a
>fix in its own right.

  Ok, will reword the commit log and push this separately.

Regards,
 Sricharan

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Russell King - ARM Linux]

On Thu, Oct 27, 2016 at 09:07:23AM +0530, Sricharan wrote:
> Hi Robin,
> 
> >-----Original Message-----
> >From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
> >Sent: Wednesday, October 26, 2016 8:37 PM
> >To: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>; will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
> >kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
> >tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
> >Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops
> >
> >On 04/10/16 18:03, Sricharan R wrote:
> >> The dma_ops for the device is not getting set to NULL in
> >> arch_tear_down_dma_ops and this causes an issue when the
> >> device's probe gets deferred and retried. So reset the
> >> dma_ops to NULL.
> >
> >Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> >
> 
>  Thanks.
> 
> >This seems like it could stand independently from the rest of the series
> >- might be worth rewording the commit message to more general terms,
> >i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
> >thus clearing the ops set by the latter, and sending it to Russell as a
> >fix in its own right.
> 
>   Ok, will reword the commit log and push this separately.

So, I've come to apply this patch (since it's finally landed in the patch
system), and I'm not convinced that the commit message is really up to
scratch.

The current commit message looks like this:

"   ARM: 8674/1: dma-mapping: Reset the device's dma_ops

    arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
    dma_ops should be cleared in the teardown path. Otherwise this causes
    problem when the probe of device is retried after being deferred. The
    device's iommu structures are cleared after EPROBEDEFER error, but on
    the next try dma_ops will still be set to old value, which is not right."

It is obviously a fix, but a fix for which patch?  Looking at the
history, we have "arm: dma-mapping: Don't override dma_ops in
arch_setup_dma_ops()" which I would have guessed is the appropriate
one, but this post-dates your patch (it's very recent, v4.12-rc
recent.)

So, I need more description about the problem you were seeing when
you first proposed this patch.

How does leaving the dma_ops in place prior to "arm: dma-mapping:
Don't override dma_ops in arch_setup_dma_ops()" cause problems for
deferred probing?

What patch is your change trying to fix?  In other words, how far
back does this patch need to be backported?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Robin Murphy]

On 23/05/17 17:25, Russell King - ARM Linux wrote:
> On Thu, Oct 27, 2016 at 09:07:23AM +0530, Sricharan wrote:
>> Hi Robin,
>>
>>> -----Original Message-----
>>> From: Robin Murphy [mailto:robin.murphy-5wv7dgnIgG8@public.gmane.org]
>>> Sent: Wednesday, October 26, 2016 8:37 PM
>>> To: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>; will.deacon-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
>>> kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
>>> tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
>>> Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops
>>>
>>> On 04/10/16 18:03, Sricharan R wrote:
>>>> The dma_ops for the device is not getting set to NULL in
>>>> arch_tear_down_dma_ops and this causes an issue when the
>>>> device's probe gets deferred and retried. So reset the
>>>> dma_ops to NULL.
>>>
>>> Reviewed-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>>>
>>
>>  Thanks.
>>
>>> This seems like it could stand independently from the rest of the series
>>> - might be worth rewording the commit message to more general terms,
>>> i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
>>> thus clearing the ops set by the latter, and sending it to Russell as a
>>> fix in its own right.
>>
>>   Ok, will reword the commit log and push this separately.
> 
> So, I've come to apply this patch (since it's finally landed in the patch
> system), and I'm not convinced that the commit message is really up to
> scratch.
> 
> The current commit message looks like this:
> 
> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
> 
>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
>     dma_ops should be cleared in the teardown path. Otherwise this causes
>     problem when the probe of device is retried after being deferred. The
>     device's iommu structures are cleared after EPROBEDEFER error, but on
>     the next try dma_ops will still be set to old value, which is not right."
> 
> It is obviously a fix, but a fix for which patch?  Looking at the
> history, we have "arm: dma-mapping: Don't override dma_ops in
> arch_setup_dma_ops()" which I would have guessed is the appropriate
> one, but this post-dates your patch (it's very recent, v4.12-rc
> recent.)
> 
> So, I need more description about the problem you were seeing when
> you first proposed this patch.
> 
> How does leaving the dma_ops in place prior to "arm: dma-mapping:
> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
> deferred probing?
> 
> What patch is your change trying to fix?  In other words, how far
> back does this patch need to be backported?

In effect, it's fixing a latent inconsistency that's been present since
its introduction in 4bb25789ed28. However, that has clearly not proven
to be an issue in practice since then. With 09515ef5ddad we start
actually calling arch_teardown_dma_ops() in a manner that might leave
things partially initialised if anyone starts removing and reprobing
drivers, but so far that's still from code inspection[1] rather than
anyone hitting it.

Given that the changes which tickle it are fresh in -rc1 I'd say there's
no need to backport this, but at the same time it shouldn't do any
damage if you really want to.

Robin.

[1]:https://www.mail-archive.com/linux-renesas-soc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg14301.html

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Russell King - ARM Linux]

On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
> On 23/05/17 17:25, Russell King - ARM Linux wrote:
> > So, I've come to apply this patch (since it's finally landed in the patch
> > system), and I'm not convinced that the commit message is really up to
> > scratch.
> > 
> > The current commit message looks like this:
> > 
> > "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
> > 
> >     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >     dma_ops should be cleared in the teardown path. Otherwise this causes
> >     problem when the probe of device is retried after being deferred. The
> >     device's iommu structures are cleared after EPROBEDEFER error, but on
> >     the next try dma_ops will still be set to old value, which is not right."
> > 
> > It is obviously a fix, but a fix for which patch?  Looking at the
> > history, we have "arm: dma-mapping: Don't override dma_ops in
> > arch_setup_dma_ops()" which I would have guessed is the appropriate
> > one, but this post-dates your patch (it's very recent, v4.12-rc
> > recent.)
> > 
> > So, I need more description about the problem you were seeing when
> > you first proposed this patch.
> > 
> > How does leaving the dma_ops in place prior to "arm: dma-mapping:
> > Don't override dma_ops in arch_setup_dma_ops()" cause problems for
> > deferred probing?
> > 
> > What patch is your change trying to fix?  In other words, how far
> > back does this patch need to be backported?
> 
> In effect, it's fixing a latent inconsistency that's been present since
> its introduction in 4bb25789ed28. However, that has clearly not proven
> to be an issue in practice since then. With 09515ef5ddad we start
> actually calling arch_teardown_dma_ops() in a manner that might leave
> things partially initialised if anyone starts removing and reprobing
> drivers, but so far that's still from code inspection[1] rather than
> anyone hitting it.
> 
> Given that the changes which tickle it are fresh in -rc1 I'd say there's
> no need to backport this, but at the same time it shouldn't do any
> damage if you really want to.

Well, looking at this, I'm not convinced that much of it is correct.

1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
   the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
   rather than arch_teardown_dma_ops().

   This doesn't strike me as being particularly symmetric.
   arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
   counterpart.

2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
   check, and Xen - Xen wants to override the DMA ops if in the
   initial domain.  It's not clear (at least to me) whether the recent
   patch adding the dma_ops check took account of this or not.

3) random places seem to fiddle with the dma_ops - notice that
   arm_iommu_detach_device() sets the dma_ops to NULL.

   In fact, I think moving __arm_iommu_detach_device() into
   arm_iommu_detach_device(), calling arm_iommu_detach_device(),
   and getting rid of the explicit set_dma_ops(, NULL) in this
   path would be a good first step.

4) I think arch_setup_dma_ops() is over-complex.

So, in summary, this code is a mess today, and that means it's not
obviously correct - which is bad.  This needs sorting.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Laurent Pinchart]

Hi Russell,

On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:
> On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
> > On 23/05/17 17:25, Russell King - ARM Linux wrote:
> >> So, I've come to apply this patch (since it's finally landed in the
> >> patch system), and I'm not convinced that the commit message is really
> >> up to scratch.
> >> 
> >> The current commit message looks like this:
> >> 
> >> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
> >>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>     dma_ops should be cleared in the teardown path. Otherwise this
> >>     causes problem when the probe of device is retried after being
> >>     deferred. The device's iommu structures are cleared after
> >>     EPROBEDEFER error, but on the next try dma_ops will still be set to
> >>     old value, which is not right."
> >> 
> >> It is obviously a fix, but a fix for which patch?  Looking at the
> >> history, we have "arm: dma-mapping: Don't override dma_ops in
> >> arch_setup_dma_ops()" which I would have guessed is the appropriate
> >> one, but this post-dates your patch (it's very recent, v4.12-rc
> >> recent.)
> >> 
> >> So, I need more description about the problem you were seeing when
> >> you first proposed this patch.
> >> 
> >> How does leaving the dma_ops in place prior to "arm: dma-mapping:
> >> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
> >> deferred probing?
> >> 
> >> What patch is your change trying to fix?  In other words, how far
> >> back does this patch need to be backported?
> > 
> > In effect, it's fixing a latent inconsistency that's been present since
> > its introduction in 4bb25789ed28. However, that has clearly not proven
> > to be an issue in practice since then. With 09515ef5ddad we start
> > actually calling arch_teardown_dma_ops() in a manner that might leave
> > things partially initialised if anyone starts removing and reprobing
> > drivers, but so far that's still from code inspection[1] rather than
> > anyone hitting it.
> > 
> > Given that the changes which tickle it are fresh in -rc1 I'd say there's
> > no need to backport this, but at the same time it shouldn't do any
> > damage if you really want to.
> 
> Well, looking at this, I'm not convinced that much of it is correct.
> 
> 1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
>    the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
>    rather than arch_teardown_dma_ops().
> 
>    This doesn't strike me as being particularly symmetric.
>    arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
>    counterpart.
> 
> 2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
>    check, and Xen - Xen wants to override the DMA ops if in the
>    initial domain.  It's not clear (at least to me) whether the recent
>    patch adding the dma_ops check took account of this or not.
> 
> 3) random places seem to fiddle with the dma_ops - notice that
>    arm_iommu_detach_device() sets the dma_ops to NULL.
> 
>    In fact, I think moving __arm_iommu_detach_device() into
>    arm_iommu_detach_device(), calling arm_iommu_detach_device(),
>    and getting rid of the explicit set_dma_ops(, NULL) in this
>    path would be a good first step.
> 
> 4) I think arch_setup_dma_ops() is over-complex.
> 
> So, in summary, this code is a mess today, and that means it's not
> obviously correct - which is bad.  This needs sorting.

We've reached the same conclusion independently, but I'll refrain from 
commenting on whether that's a good or bad thing :-)

I don't think this patch should be applied, as it could break Xen (and other 
platforms/drivers that set the DMA operations manually) by resetting DMA 
operations at device remove() time even if they have been set independently of 
arch_setup_dma_ops().

The IOMMU probe deferral support patch series was merged in v4.12-rc1 and 
breaks IOMMU operations on several platforms. We need a fix for v4.12-rc that 
should be as nonintrusive as possible, as a larger cleanup is likely not -rc 
material. Beside reverting the whole series, the simplest option I came up 
with was [1]. Note that this is not the only fix needed to fix v4.12-rc1 IOMMU 
breakage, there are four more patches in the series that Sricharan plans to 
get merged soon. I don't think there are dependencies between the other four 
drivers/ patches and the arch/arm/ patch, so the latter could be merged 
independently through your tree as soon as it's deemed ready.

[1] https://www.spinics.net/lists/arm-kernel/msg583019.html

-- 
Regards,

Laurent Pinchart

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Russell King - ARM Linux]

On Wed, May 24, 2017 at 12:46:51AM +0300, Laurent Pinchart wrote:
> Hi Russell,
> 
> On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:
> > On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
> > > On 23/05/17 17:25, Russell King - ARM Linux wrote:
> > >> So, I've come to apply this patch (since it's finally landed in the
> > >> patch system), and I'm not convinced that the commit message is really
> > >> up to scratch.
> > >> 
> > >> The current commit message looks like this:
> > >> 
> > >> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
> > >>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> > >>     dma_ops should be cleared in the teardown path. Otherwise this
> > >>     causes problem when the probe of device is retried after being
> > >>     deferred. The device's iommu structures are cleared after
> > >>     EPROBEDEFER error, but on the next try dma_ops will still be set to
> > >>     old value, which is not right."
> > >> 
> > >> It is obviously a fix, but a fix for which patch?  Looking at the
> > >> history, we have "arm: dma-mapping: Don't override dma_ops in
> > >> arch_setup_dma_ops()" which I would have guessed is the appropriate
> > >> one, but this post-dates your patch (it's very recent, v4.12-rc
> > >> recent.)
> > >> 
> > >> So, I need more description about the problem you were seeing when
> > >> you first proposed this patch.
> > >> 
> > >> How does leaving the dma_ops in place prior to "arm: dma-mapping:
> > >> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
> > >> deferred probing?
> > >> 
> > >> What patch is your change trying to fix?  In other words, how far
> > >> back does this patch need to be backported?
> > > 
> > > In effect, it's fixing a latent inconsistency that's been present since
> > > its introduction in 4bb25789ed28. However, that has clearly not proven
> > > to be an issue in practice since then. With 09515ef5ddad we start
> > > actually calling arch_teardown_dma_ops() in a manner that might leave
> > > things partially initialised if anyone starts removing and reprobing
> > > drivers, but so far that's still from code inspection[1] rather than
> > > anyone hitting it.
> > > 
> > > Given that the changes which tickle it are fresh in -rc1 I'd say there's
> > > no need to backport this, but at the same time it shouldn't do any
> > > damage if you really want to.
> > 
> > Well, looking at this, I'm not convinced that much of it is correct.
> > 
> > 1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
> >    the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
> >    rather than arch_teardown_dma_ops().
> > 
> >    This doesn't strike me as being particularly symmetric.
> >    arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
> >    counterpart.
> > 
> > 2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
> >    check, and Xen - Xen wants to override the DMA ops if in the
> >    initial domain.  It's not clear (at least to me) whether the recent
> >    patch adding the dma_ops check took account of this or not.
> > 
> > 3) random places seem to fiddle with the dma_ops - notice that
> >    arm_iommu_detach_device() sets the dma_ops to NULL.
> > 
> >    In fact, I think moving __arm_iommu_detach_device() into
> >    arm_iommu_detach_device(), calling arm_iommu_detach_device(),
> >    and getting rid of the explicit set_dma_ops(, NULL) in this
> >    path would be a good first step.
> > 
> > 4) I think arch_setup_dma_ops() is over-complex.
> > 
> > So, in summary, this code is a mess today, and that means it's not
> > obviously correct - which is bad.  This needs sorting.
> 
> We've reached the same conclusion independently, but I'll refrain from 
> commenting on whether that's a good or bad thing :-)
> 
> I don't think this patch should be applied, as it could break Xen (and other 
> platforms/drivers that set the DMA operations manually) by resetting DMA 
> operations at device remove() time even if they have been set independently of 
> arch_setup_dma_ops().

That will only occur if the dma ops have been overriden once the DMA
operations have been setup via arch_setup_dma_ops.  What saves it from
wholesale NULLing of the DMA operations is the check for a valid
dma_iommu_mapping structure in arm_teardown_iommu_dma_ops().  This only
exists when arm_setup_iommu_dma_ops() has attached a mapping to the
device.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Sricharan R]

Hi Russell/Laurent/Robin,

On 5/24/2017 4:12 AM, Russell King - ARM Linux wrote:
> On Wed, May 24, 2017 at 12:46:51AM +0300, Laurent Pinchart wrote:
>> Hi Russell,
>>
>> On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:
>>> On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
>>>> On 23/05/17 17:25, Russell King - ARM Linux wrote:
>>>>> So, I've come to apply this patch (since it's finally landed in the
>>>>> patch system), and I'm not convinced that the commit message is really
>>>>> up to scratch.
>>>>>
>>>>> The current commit message looks like this:
>>>>>
>>>>> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
>>>>>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
>>>>>     dma_ops should be cleared in the teardown path. Otherwise this
>>>>>     causes problem when the probe of device is retried after being
>>>>>     deferred. The device's iommu structures are cleared after
>>>>>     EPROBEDEFER error, but on the next try dma_ops will still be set to
>>>>>     old value, which is not right."
>>>>>
>>>>> It is obviously a fix, but a fix for which patch?  Looking at the
>>>>> history, we have "arm: dma-mapping: Don't override dma_ops in
>>>>> arch_setup_dma_ops()" which I would have guessed is the appropriate
>>>>> one, but this post-dates your patch (it's very recent, v4.12-rc
>>>>> recent.)
>>>>>
>>>>> So, I need more description about the problem you were seeing when
>>>>> you first proposed this patch.
>>>>>
>>>>> How does leaving the dma_ops in place prior to "arm: dma-mapping:
>>>>> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
>>>>> deferred probing?
>>>>>
>>>>> What patch is your change trying to fix?  In other words, how far
>>>>> back does this patch need to be backported?
>>>>
>>>> In effect, it's fixing a latent inconsistency that's been present since
>>>> its introduction in 4bb25789ed28. However, that has clearly not proven
>>>> to be an issue in practice since then. With 09515ef5ddad we start
>>>> actually calling arch_teardown_dma_ops() in a manner that might leave
>>>> things partially initialised if anyone starts removing and reprobing
>>>> drivers, but so far that's still from code inspection[1] rather than
>>>> anyone hitting it.
>>>>
>>>> Given that the changes which tickle it are fresh in -rc1 I'd say there's
>>>> no need to backport this, but at the same time it shouldn't do any
>>>> damage if you really want to.
>>>
>>> Well, looking at this, I'm not convinced that much of it is correct.
>>>
>>> 1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
>>>    the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
>>>    rather than arch_teardown_dma_ops().
>>>
>>>    This doesn't strike me as being particularly symmetric.
>>>    arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
>>>    counterpart.
>>>
>>> 2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
>>>    check, and Xen - Xen wants to override the DMA ops if in the
>>>    initial domain.  It's not clear (at least to me) whether the recent
>>>    patch adding the dma_ops check took account of this or not.
>>>
>>> 3) random places seem to fiddle with the dma_ops - notice that
>>>    arm_iommu_detach_device() sets the dma_ops to NULL.
>>>
>>>    In fact, I think moving __arm_iommu_detach_device() into
>>>    arm_iommu_detach_device(), calling arm_iommu_detach_device(),
>>>    and getting rid of the explicit set_dma_ops(, NULL) in this
>>>    path would be a good first step.
>>>
>>> 4) I think arch_setup_dma_ops() is over-complex.
>>>
>>> So, in summary, this code is a mess today, and that means it's not
>>> obviously correct - which is bad.  This needs sorting.
>>
>> We've reached the same conclusion independently, but I'll refrain from 
>> commenting on whether that's a good or bad thing :-)
>>
>> I don't think this patch should be applied, as it could break Xen (and other 
>> platforms/drivers that set the DMA operations manually) by resetting DMA 
>> operations at device remove() time even if they have been set independently of 
>> arch_setup_dma_ops().
> 
> That will only occur if the dma ops have been overriden once the DMA
> operations have been setup via arch_setup_dma_ops.  What saves it from
> wholesale NULLing of the DMA operations is the check for a valid
> dma_iommu_mapping structure in arm_teardown_iommu_dma_ops().  This only
> exists when arm_setup_iommu_dma_ops() has attached a mapping to the
> device.
> 

Right, only if the dma ops are set and no dma_iommu_mapping is created for
the device, then arch_teardown_iommu_dma_ops does nothing.

Firstly, this patch for resetting the dma_ops in teardown was required
only when arch_setup_dma_ops has created both the mapping and dma_ops
for the device. Because mappings that were created in arch_setup_dma_ops
are cleared in teardown, so dma ops should also be reset. But this can be 
done by calling arm_iommu_detach_device() from arm_teardown_iommu_dma_ops
to avoid explicitly calling set_dma_ops again, probably this what was
suggested in #3 above ?

Really sorry for the mess, but below cleanups looks required otherwise,

1) set_dma_ops is called for mach-highbank, mach-mevbu during the  
   BUS_NOTIFY_ADD_DEVICE. That should be removed and made to come from
   DT path and arch_setup_dma_ops. dmabounce.c also does set_dma_ops,
   not very sure how to handle that (swiotlb ?), are call
   dmabounce_register_dev during the device's probe instead to have the
   dma_set_ops overriding later in probe ?

2) arm_iommu_attach_device is called from ipmmu-vmsa.c, mtk_iommu_v1.c,
   iommu drivers, from the iommu add_device callback, called
   from BUS_NOTIFY_ADD_DEVICE notifier. This is a problem because,
   with probe-deferral, this can be overridden in arch_setup_dma_ops
   during device probe and cleared in teardown path. But the add_device
   callback notifier is not called again when the device gets reprobed again.
 
   With probe deferral, add_device callback also gets called from
   of_iommu_configure during device probe, so the above drivers should
   be adapted to properly register the iommu_ops to have its add_device
   called from of_iommu_configure path. mtk_iommu_v1.c seems to be fine,
   but ipmmu-vmsa.c should be adapted. otherwise if the devices attached
   to those iommus call arm_iommu_attach_device from its probe path to override
   the default ops set in arch_setup_dma_ops, then all is fine. This seems to
   be the case with exynos_drm_iommu.c, omap3isp/isp.c.

   If the above two are done, the overridding of the default dma_ops and mapping
   should happen after arch_setup_dma_ops is called and also overridden every
   time the device gets reprobed. That should help to get rid of couple of fixes
   that has been added.

3) As Laurent already pointed out earlier, return error codes from some of the
   IOMMU apis needs to standardized.

   Please let me know if its right way of doing it ?

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Laurent Pinchart]

Hello,

On Wednesday 24 May 2017 16:01:45 Sricharan R wrote:
> On 5/24/2017 4:12 AM, Russell King - ARM Linux wrote:
> > On Wed, May 24, 2017 at 12:46:51AM +0300, Laurent Pinchart wrote:
> >> On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:
> >>> On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
> >>>> On 23/05/17 17:25, Russell King - ARM Linux wrote:
> >>>>> So, I've come to apply this patch (since it's finally landed in the
> >>>>> patch system), and I'm not convinced that the commit message is really
> >>>>> up to scratch.
> >>>>> 
> >>>>> The current commit message looks like this:
> >>>>> 
> >>>>> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
> >>>>> 
> >>>>>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
> >>>>>     dma_ops should be cleared in the teardown path. Otherwise this
> >>>>>     causes problem when the probe of device is retried after being
> >>>>>     deferred. The device's iommu structures are cleared after
> >>>>>     EPROBEDEFER error, but on the next try dma_ops will still be set
> >>>>>     to old value, which is not right."
> >>>>> 
> >>>>> It is obviously a fix, but a fix for which patch?  Looking at the
> >>>>> history, we have "arm: dma-mapping: Don't override dma_ops in
> >>>>> arch_setup_dma_ops()" which I would have guessed is the appropriate
> >>>>> one, but this post-dates your patch (it's very recent, v4.12-rc
> >>>>> recent.)
> >>>>> 
> >>>>> So, I need more description about the problem you were seeing when
> >>>>> you first proposed this patch.
> >>>>> 
> >>>>> How does leaving the dma_ops in place prior to "arm: dma-mapping:
> >>>>> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
> >>>>> deferred probing?
> >>>>> 
> >>>>> What patch is your change trying to fix?  In other words, how far
> >>>>> back does this patch need to be backported?
> >>>> 
> >>>> In effect, it's fixing a latent inconsistency that's been present since
> >>>> its introduction in 4bb25789ed28. However, that has clearly not proven
> >>>> to be an issue in practice since then. With 09515ef5ddad we start
> >>>> actually calling arch_teardown_dma_ops() in a manner that might leave
> >>>> things partially initialised if anyone starts removing and reprobing
> >>>> drivers, but so far that's still from code inspection[1] rather than
> >>>> anyone hitting it.
> >>>> 
> >>>> Given that the changes which tickle it are fresh in -rc1 I'd say
> >>>> there's no need to backport this, but at the same time it shouldn't do
> >>>> any damage if you really want to.
> >>> 
> >>> Well, looking at this, I'm not convinced that much of it is correct.
> >>> 
> >>> 1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
> >>>    the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
> >>>    rather than arch_teardown_dma_ops().
> >>>    
> >>>    This doesn't strike me as being particularly symmetric.
> >>>    arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
> >>>    counterpart.
> >>> 
> >>> 2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
> >>>    check, and Xen - Xen wants to override the DMA ops if in the
> >>>    initial domain.  It's not clear (at least to me) whether the recent
> >>>    patch adding the dma_ops check took account of this or not.
> >>> 
> >>> 3) random places seem to fiddle with the dma_ops - notice that
> >>>    arm_iommu_detach_device() sets the dma_ops to NULL.
> >>>    
> >>>    In fact, I think moving __arm_iommu_detach_device() into
> >>>    arm_iommu_detach_device(), calling arm_iommu_detach_device(),
> >>>    and getting rid of the explicit set_dma_ops(, NULL) in this
> >>>    path would be a good first step.
> >>> 
> >>> 4) I think arch_setup_dma_ops() is over-complex.
> >>> 
> >>> So, in summary, this code is a mess today, and that means it's not
> >>> obviously correct - which is bad.  This needs sorting.
> >> 
> >> We've reached the same conclusion independently, but I'll refrain from
> >> commenting on whether that's a good or bad thing :-)
> >> 
> >> I don't think this patch should be applied, as it could break Xen (and
> >> other platforms/drivers that set the DMA operations manually) by
> >> resetting DMA operations at device remove() time even if they have been
> >> set independently of arch_setup_dma_ops().
> > 
> > That will only occur if the dma ops have been overriden once the DMA
> > operations have been setup via arch_setup_dma_ops. What saves it from
> > wholesale NULLing of the DMA operations is the check for a valid
> > dma_iommu_mapping structure in arm_teardown_iommu_dma_ops().  This only
> > exists when arm_setup_iommu_dma_ops() has attached a mapping to the
> > device.

Unfortunately I don't think that's always the case. The dma_iommu_mapping is 
also set by direct callers of arm_iommu_attach_device(), namely

- the Renesas R-Car IOMMU driver (ipmmu-vmsa)
- the Mediatek IOMMU driver (mtk-iommu-v1)
- the Exynos DRM driver
- the OMAP3 ISP driver

All these need to be fixed, but that's not v4.12-rc material. At least in the 
ipmmu-vmsa case, which is the one I noticed the problem with, 
arm_iommu_attach_device() is called before arch_setup_dma_ops(). 
arch_setup_dma_ops() then exits immediately when called due to the

        if (dev->dma_ops)
                return;

check at the beginning of the function. We must ensure that in that case 
arch_teardown_dma_ops() will not remove the mapping or set the DMA ops to 
NULL, and testing to_dma_iommu_mapping(dev) won't help.

> Right, only if the dma ops are set and no dma_iommu_mapping is created for
> the device, then arch_teardown_iommu_dma_ops does nothing.
> 
> Firstly, this patch for resetting the dma_ops in teardown was required
> only when arch_setup_dma_ops has created both the mapping and dma_ops
> for the device. Because mappings that were created in arch_setup_dma_ops
> are cleared in teardown, so dma ops should also be reset. But this can be
> done by calling arm_iommu_detach_device() from arm_teardown_iommu_dma_ops
> to avoid explicitly calling set_dma_ops again, probably this what was
> suggested in #3 above ?
> 
> Really sorry for the mess, but below cleanups looks required otherwise,
> 
> 1) set_dma_ops is called for mach-highbank, mach-mevbu during the
>    BUS_NOTIFY_ADD_DEVICE. That should be removed and made to come from
>    DT path and arch_setup_dma_ops. dmabounce.c also does set_dma_ops,
>    not very sure how to handle that (swiotlb ?), are call
>    dmabounce_register_dev during the device's probe instead to have the
>    dma_set_ops overriding later in probe ?

All this needs to be addressed, but it's definitely not v4.12-rc material.

> 2) arm_iommu_attach_device is called from ipmmu-vmsa.c, mtk_iommu_v1.c,
>    iommu drivers, from the iommu add_device callback, called
>    from BUS_NOTIFY_ADD_DEVICE notifier. This is a problem because,
>    with probe-deferral, this can be overridden in arch_setup_dma_ops
>    during device probe and cleared in teardown path. But the add_device
>    callback notifier is not called again when the device gets reprobed
>    again.
> 
>    With probe deferral, add_device callback also gets called from
>    of_iommu_configure during device probe, so the above drivers should
>    be adapted to properly register the iommu_ops to have its add_device
>    called from of_iommu_configure path. mtk_iommu_v1.c seems to be fine,
>    but ipmmu-vmsa.c should be adapted. otherwise if the devices attached
>    to those iommus call arm_iommu_attach_device from its probe path to
>    override the default ops set in arch_setup_dma_ops, then all is fine.
>    This seems to be the case with exynos_drm_iommu.c, omap3isp/isp.c.

Same here, this needs to be addressed, but not in v4.12-rc. We need a simpler 
fix for v4.12-rc.

>    If the above two are done, the overridding of the default dma_ops and
>    mapping should happen after arch_setup_dma_ops is called and also
>    overridden every time the device gets reprobed. That should help to get
>    rid of couple of fixes that has been added.
> 
> 3) As Laurent already pointed out earlier, return error codes from some of
>    the IOMMU apis needs to standardized.
> 
>    Please let me know if its right way of doing it ?

Again, the patch I propose is the simplest v4.12-rc fix I can think of, short 
of reverting your complete IOMMU probe deferral patch series. Let's focus on 
the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

-- 
Regards,

Laurent Pinchart

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Sricharan R]

Hi Laurent,

On 5/24/2017 4:56 PM, Laurent Pinchart wrote:
> Hello,
> 
> On Wednesday 24 May 2017 16:01:45 Sricharan R wrote:
>> On 5/24/2017 4:12 AM, Russell King - ARM Linux wrote:
>>> On Wed, May 24, 2017 at 12:46:51AM +0300, Laurent Pinchart wrote:
>>>> On Tuesday 23 May 2017 18:53:19 Russell King - ARM Linux wrote:
>>>>> On Tue, May 23, 2017 at 05:55:57PM +0100, Robin Murphy wrote:
>>>>>> On 23/05/17 17:25, Russell King - ARM Linux wrote:
>>>>>>> So, I've come to apply this patch (since it's finally landed in the
>>>>>>> patch system), and I'm not convinced that the commit message is really
>>>>>>> up to scratch.
>>>>>>>
>>>>>>> The current commit message looks like this:
>>>>>>>
>>>>>>> "   ARM: 8674/1: dma-mapping: Reset the device's dma_ops
>>>>>>>
>>>>>>>     arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops(),
>>>>>>>     dma_ops should be cleared in the teardown path. Otherwise this
>>>>>>>     causes problem when the probe of device is retried after being
>>>>>>>     deferred. The device's iommu structures are cleared after
>>>>>>>     EPROBEDEFER error, but on the next try dma_ops will still be set
>>>>>>>     to old value, which is not right."
>>>>>>>
>>>>>>> It is obviously a fix, but a fix for which patch?  Looking at the
>>>>>>> history, we have "arm: dma-mapping: Don't override dma_ops in
>>>>>>> arch_setup_dma_ops()" which I would have guessed is the appropriate
>>>>>>> one, but this post-dates your patch (it's very recent, v4.12-rc
>>>>>>> recent.)
>>>>>>>
>>>>>>> So, I need more description about the problem you were seeing when
>>>>>>> you first proposed this patch.
>>>>>>>
>>>>>>> How does leaving the dma_ops in place prior to "arm: dma-mapping:
>>>>>>> Don't override dma_ops in arch_setup_dma_ops()" cause problems for
>>>>>>> deferred probing?
>>>>>>>
>>>>>>> What patch is your change trying to fix?  In other words, how far
>>>>>>> back does this patch need to be backported?
>>>>>>
>>>>>> In effect, it's fixing a latent inconsistency that's been present since
>>>>>> its introduction in 4bb25789ed28. However, that has clearly not proven
>>>>>> to be an issue in practice since then. With 09515ef5ddad we start
>>>>>> actually calling arch_teardown_dma_ops() in a manner that might leave
>>>>>> things partially initialised if anyone starts removing and reprobing
>>>>>> drivers, but so far that's still from code inspection[1] rather than
>>>>>> anyone hitting it.
>>>>>>
>>>>>> Given that the changes which tickle it are fresh in -rc1 I'd say
>>>>>> there's no need to backport this, but at the same time it shouldn't do
>>>>>> any damage if you really want to.
>>>>>
>>>>> Well, looking at this, I'm not convinced that much of it is correct.
>>>>>
>>>>> 1) set_dma_ops() is in arch_setup_dma_ops() but this patch adds
>>>>>    the unsetting of the DMA ops inside arm_teardown_iommu_dma_ops()
>>>>>    rather than arch_teardown_dma_ops().
>>>>>    
>>>>>    This doesn't strike me as being particularly symmetric.
>>>>>    arm_teardown_iommu_dma_ops() is arm_setup_iommu_dma_ops()'s
>>>>>    counterpart.
>>>>>
>>>>> 2) arch_setup_dma_ops(), the recent patch to add the existing dma_ops
>>>>>    check, and Xen - Xen wants to override the DMA ops if in the
>>>>>    initial domain.  It's not clear (at least to me) whether the recent
>>>>>    patch adding the dma_ops check took account of this or not.
>>>>>
>>>>> 3) random places seem to fiddle with the dma_ops - notice that
>>>>>    arm_iommu_detach_device() sets the dma_ops to NULL.
>>>>>    
>>>>>    In fact, I think moving __arm_iommu_detach_device() into
>>>>>    arm_iommu_detach_device(), calling arm_iommu_detach_device(),
>>>>>    and getting rid of the explicit set_dma_ops(, NULL) in this
>>>>>    path would be a good first step.
>>>>>
>>>>> 4) I think arch_setup_dma_ops() is over-complex.
>>>>>
>>>>> So, in summary, this code is a mess today, and that means it's not
>>>>> obviously correct - which is bad.  This needs sorting.
>>>>
>>>> We've reached the same conclusion independently, but I'll refrain from
>>>> commenting on whether that's a good or bad thing :-)
>>>>
>>>> I don't think this patch should be applied, as it could break Xen (and
>>>> other platforms/drivers that set the DMA operations manually) by
>>>> resetting DMA operations at device remove() time even if they have been
>>>> set independently of arch_setup_dma_ops().
>>>
>>> That will only occur if the dma ops have been overriden once the DMA
>>> operations have been setup via arch_setup_dma_ops. What saves it from
>>> wholesale NULLing of the DMA operations is the check for a valid
>>> dma_iommu_mapping structure in arm_teardown_iommu_dma_ops().  This only
>>> exists when arm_setup_iommu_dma_ops() has attached a mapping to the
>>> device.
> 
> Unfortunately I don't think that's always the case. The dma_iommu_mapping is 
> also set by direct callers of arm_iommu_attach_device(), namely
> 
> - the Renesas R-Car IOMMU driver (ipmmu-vmsa)
> - the Mediatek IOMMU driver (mtk-iommu-v1)
> - the Exynos DRM driver
> - the OMAP3 ISP driver
> 
> All these need to be fixed, but that's not v4.12-rc material. At least in the 
> ipmmu-vmsa case, which is the one I noticed the problem with, 
> arm_iommu_attach_device() is called before arch_setup_dma_ops(). 
> arch_setup_dma_ops() then exits immediately when called due to the
> 
>         if (dev->dma_ops)
>                 return;
> 
> check at the beginning of the function. We must ensure that in that case 
> arch_teardown_dma_ops() will not remove the mapping or set the DMA ops to 
> NULL, and testing to_dma_iommu_mapping(dev) won't help.
> 

Right, agree. Same issue highlighted in #1, #2 below and the fixes posted [1]
for 4.12-rc.

>> Right, only if the dma ops are set and no dma_iommu_mapping is created for
>> the device, then arch_teardown_iommu_dma_ops does nothing.
>>
>> Firstly, this patch for resetting the dma_ops in teardown was required
>> only when arch_setup_dma_ops has created both the mapping and dma_ops
>> for the device. Because mappings that were created in arch_setup_dma_ops
>> are cleared in teardown, so dma ops should also be reset. But this can be
>> done by calling arm_iommu_detach_device() from arm_teardown_iommu_dma_ops
>> to avoid explicitly calling set_dma_ops again, probably this what was
>> suggested in #3 above ?
>>
>> Really sorry for the mess, but below cleanups looks required otherwise,
>>
>> 1) set_dma_ops is called for mach-highbank, mach-mevbu during the
>>    BUS_NOTIFY_ADD_DEVICE. That should be removed and made to come from
>>    DT path and arch_setup_dma_ops. dmabounce.c also does set_dma_ops,
>>    not very sure how to handle that (swiotlb ?), are call
>>    dmabounce_register_dev during the device's probe instead to have the
>>    dma_set_ops overriding later in probe ?
> 
> All this needs to be addressed, but it's definitely not v4.12-rc material.
> 
>> 2) arm_iommu_attach_device is called from ipmmu-vmsa.c, mtk_iommu_v1.c,
>>    iommu drivers, from the iommu add_device callback, called
>>    from BUS_NOTIFY_ADD_DEVICE notifier. This is a problem because,
>>    with probe-deferral, this can be overridden in arch_setup_dma_ops
>>    during device probe and cleared in teardown path. But the add_device
>>    callback notifier is not called again when the device gets reprobed
>>    again.
>>
>>    With probe deferral, add_device callback also gets called from
>>    of_iommu_configure during device probe, so the above drivers should
>>    be adapted to properly register the iommu_ops to have its add_device
>>    called from of_iommu_configure path. mtk_iommu_v1.c seems to be fine,
>>    but ipmmu-vmsa.c should be adapted. otherwise if the devices attached
>>    to those iommus call arm_iommu_attach_device from its probe path to
>>    override the default ops set in arch_setup_dma_ops, then all is fine.
>>    This seems to be the case with exynos_drm_iommu.c, omap3isp/isp.c.
> 
> Same here, this needs to be addressed, but not in v4.12-rc. We need a simpler 
> fix for v4.12-rc.
> 

Right, not for 4.12-rc, but as a better cleanup beyond, so
wanted to discuss the right sequence further. The fixes posted are the ones
for 4.12-rc.

>>    If the above two are done, the overridding of the default dma_ops and
>>    mapping should happen after arch_setup_dma_ops is called and also
>>    overridden every time the device gets reprobed. That should help to get
>>    rid of couple of fixes that has been added.
>>
>> 3) As Laurent already pointed out earlier, return error codes from some of
>>    the IOMMU apis needs to standardized.
>>
>>    Please let me know if its right way of doing it ?
> 
> Again, the patch I propose is the simplest v4.12-rc fix I can think of, short 
> of reverting your complete IOMMU probe deferral patch series. Let's focus on 
> the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.
> 

Agree.

[1] https://lkml.org/lkml/2017/5/23/411


Regards,
 Sricharan
-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Russell King - ARM Linux]

On Wed, May 24, 2017 at 02:26:13PM +0300, Laurent Pinchart wrote:
> Again, the patch I propose is the simplest v4.12-rc fix I can think of, short 
> of reverting your complete IOMMU probe deferral patch series. Let's focus on 
> the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.

Except, I don't think it fixes the problem that Sricharan is trying to
fix, namely that of DMA ops that have been setup, torn down, and are
trying to be re-setup again.  The issue here is that results in a stale
setup, because the dma_ops are left in-place from the first iommu setup,
but the iommu mapping has been disposed of.

Your patch only avoids the problem of tearing down someone else's DMA
ops.  We need a combination of both patches together.

What needs to happen for Sricharan's problem to be resolved is:

1. move all of __arm_iommu_detach_device() into arm_iommu_detach_device().
2. replace the __arm_iommu_detach_device() call in arm_teardown_iommu_dma_ops()
   with arm_iommu_detach_device().

as I don't see any need to have a different order in
arm_teardown_iommu_dma_ops().

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Sricharan R]

Hi Russell,

On 5/25/2017 8:35 PM, Russell King - ARM Linux wrote:
> On Wed, May 24, 2017 at 02:26:13PM +0300, Laurent Pinchart wrote:
>> Again, the patch I propose is the simplest v4.12-rc fix I can think of, short 
>> of reverting your complete IOMMU probe deferral patch series. Let's focus on 
>> the v4.12-rc fix, and then discuss how to move forward in v4.13 and beyond.
> 
> Except, I don't think it fixes the problem that Sricharan is trying to
> fix, namely that of DMA ops that have been setup, torn down, and are
> trying to be re-setup again.  The issue here is that results in a stale
> setup, because the dma_ops are left in-place from the first iommu setup,
> but the iommu mapping has been disposed of.
> 
> Your patch only avoids the problem of tearing down someone else's DMA
> ops.  We need a combination of both patches together.
> 
> What needs to happen for Sricharan's problem to be resolved is:
> 
> 1. move all of __arm_iommu_detach_device() into arm_iommu_detach_device().
> 2. replace the __arm_iommu_detach_device() call in arm_teardown_iommu_dma_ops()
>    with arm_iommu_detach_device().
> 
> as I don't see any need to have a different order in
> arm_teardown_iommu_dma_ops().
> 

Right, both patches are required and i was also thining the same thing about
using arm_iommu_detach_device from arm_teardown_iommu_dma_ops instead. Will
repost with this.

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

[Laurent Pinchart]

Hi Russell,

On Thursday 25 May 2017 16:05:41 Russell King - ARM Linux wrote:
> On Wed, May 24, 2017 at 02:26:13PM +0300, Laurent Pinchart wrote:
> > Again, the patch I propose is the simplest v4.12-rc fix I can think of,
> > short of reverting your complete IOMMU probe deferral patch series. Let's
> > focus on the v4.12-rc fix, and then discuss how to move forward in v4.13
> > and beyond.
>
> Except, I don't think it fixes the problem that Sricharan is trying to
> fix, namely that of DMA ops that have been setup, torn down, and are
> trying to be re-setup again.

You're right, it's two separate problems, and we need to fix them both.

> The issue here is that results in a stale setup, because the dma_ops are
> left in-place from the first iommu setup, but the iommu mapping has been
> disposed of.
> 
> Your patch only avoids the problem of tearing down someone else's DMA
> ops.  We need a combination of both patches together.

Yes exactly. I should read e-mails to the end before starting typing the reply 
:-)

> What needs to happen for Sricharan's problem to be resolved is:
> 
> 1. move all of __arm_iommu_detach_device() into arm_iommu_detach_device().
> 2. replace the __arm_iommu_detach_device() call in
> arm_teardown_iommu_dma_ops() with arm_iommu_detach_device().
> 
> as I don't see any need to have a different order in
> arm_teardown_iommu_dma_ops().

-- 
Regards,

Laurent Pinchart

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Marek Szyprowski]

Hi Sricharan,


On 2016-10-04 19:03, Sricharan R wrote:
> Initial post from Laurent Pinchart[1]. This is
> series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
>
>
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>         |                         |
> pci_bus_add_device     (device_add/driver_register)
>         |                         |
> device_attach           device_initial_probe
>         |                         |
> __device_attach_driver    __device_attach_driver
>         |
> driver_probe_device
>         |
> really_probe
>         |
> dma_configure
>
>   Similarly on the device/driver_unregister path __device_release_driver is
>   called which inturn calls dma_deconfigure.
>
>   If the ACPI bus code follows the same, we can add acpi_dma_configure
>   at the same place as of_dma_configure.
>
>   This series is based on the recently merged Generic DT bindings for
>   PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>
>   This time tested this with platform and pci device for probe deferral
>   and reprobe on arm64 based platform. There is an issue on the cleanup
>   path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>   device is attached to an domain in arch_teardown_dma_ops.
>   But with iommu_groups created from the iommu driver, the device is always
>   attached to a domain/default_domain. So so the WARN has to be removed/handled
>   probably.

Thanks for continuing work on this feature! Your can add my:

Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>

It works fine with Exynos SYSMMU driver, although a patch is needed to fix
infinite loop due to list corruption (same element is added twice if master
device fails with deferred probe):

From: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Date: Mon, 10 Oct 2016 14:22:42 +0200
Subject: [PATCH] iommu/exynos: ensure that sysmmu is added only once to its
  master

Since adding IOMMU deferred probing support, of_xlate() callback might
be called more than once for given master device (for example it happens
when masters device driver fails with EPROBE_DEFER), so ensure that
SYSMMU controller is added to its master device (owner) only once.

Signed-off-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
---
  drivers/iommu/exynos-iommu.c | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 30808e91b775..1525a86eb829 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -1253,7 +1253,7 @@ static int exynos_iommu_of_xlate(struct device *dev,
  {
      struct exynos_iommu_owner *owner = dev->archdata.iommu;
      struct platform_device *sysmmu = of_find_device_by_node(spec->np);
-    struct sysmmu_drvdata *data;
+    struct sysmmu_drvdata *data, *entry;

      if (!sysmmu)
          return -ENODEV;
@@ -1271,6 +1271,10 @@ static int exynos_iommu_of_xlate(struct device *dev,
          dev->archdata.iommu = owner;
      }

+    list_for_each_entry(entry, &owner->controllers, owner_node)
+        if (entry == data)
+            return 0;
+
      list_add_tail(&data->owner_node, &owner->controllers);
      return 0;
  }
-- 
1.9.1


>
>   Previous post of this series [3].
>
>   [V3]
>       * Removed the patch to split dma_masks/dma_ops configuration separately
>         based on review comments that both masks and ops are required only
>         during the device probe time.
>
>       * Reworked the series based on Generic DT bindings series [2].
>
>       * Added call to iommu's remove_device in the cleanup path for arm and arm64.
>
>       * Removed the notifier trick in arm64 to handle early device registration.
>
>       * Added reset of dma_ops in cleanup path for arm based on comments.
>
>       * Fixed the pci_iommu_configure path and tested with PCI device as well.
>   
>       * Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.
>
>       * Fixed few other cosmetic comments.
>    
>   [V2]
>       * Updated the Initial post to call dma_configure/deconfigure from generic code
>   
>       * Added iommu add_device callback from of_iommu_configure path
>
>   [V1]
>       * Initial post
>
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.spinics.net/lists/devicetree/msg142943.html
> [3] https://www.mail-archive.com/iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org/msg13941.html
> [4] https://www.mail-archive.com/iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org/msg13940.html
>
>
>
> Laurent Pinchart (4):
>    arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>    of: dma: Move range size workaround to of_dma_get_range()
>    of: dma: Make of_dma_deconfigure() public
>    iommu: of: Handle IOMMU lookup failure with deferred probing or error
>
> Sricharan R (4):
>    drivers: platform: Configure dma operations at probe time
>    arm: dma-mapping: Reset the device's dma_ops
>    arm/arm64: dma-mapping: Call iommu's remove_device callback during
>      device detach
>    arm64: dma-mapping: Remove the notifier trick to handle early setting
>      of dma_ops
>
>   arch/arm/mm/dma-mapping.c   |  18 ++++++++
>   arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
>   drivers/base/dd.c           |  10 +++++
>   drivers/base/dma-mapping.c  |  11 +++++
>   drivers/iommu/of_iommu.c    |  47 +++++++++++++++++--
>   drivers/of/address.c        |  20 ++++++++-
>   drivers/of/device.c         |  34 +++++++-------
>   drivers/of/platform.c       |   9 ----
>   drivers/pci/probe.c         |   5 +--
>   include/linux/dma-mapping.h |   3 ++
>   include/linux/of_device.h   |   7 ++-
>   11 files changed, 138 insertions(+), 133 deletions(-)
>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

>-----Original Message-----
>From: Sricharan [mailto:sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org]
>Sent: Wednesday, October 12, 2016 11:55 AM
>To: 'Marek Szyprowski' <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>; 'will.deacon-5wv7dgnIgG8@public.gmane.org' <will.deacon-5wv7dgnIgG8@public.gmane.org>; 'robin.murphy-5wv7dgnIgG8@public.gmane.org'
><robin.murphy-5wv7dgnIgG8@public.gmane.org>; 'joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org' <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org>; 'iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org' <iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
>foundation.org>; 'linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org' <linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org>; 'linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'
><linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>; 'laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org' <laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>;
>'tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org' <tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>; 'srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org' <srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
>Subject: RE: [PATCH V3 0/8] IOMMU probe deferral support
>
>Hi Marek,
>
>>Hi Sricharan,
>>
>>
>>On 2016-10-04 19:03, Sricharan R wrote:
>>> Initial post from Laurent Pinchart[1]. This is
>>> series calls the dma ops configuration for the devices
>>> at a generic place so that it works for all busses.
>>> The dma_configure_ops for a device is now called during
>>> the device_attach callback just before the probe of the
>>> bus/driver is called. Similarly dma_deconfigure is called during
>>> device/driver_detach path.
>>>
>>>
>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>         |                         |
>>> pci_bus_add_device     (device_add/driver_register)
>>>         |                         |
>>> device_attach           device_initial_probe
>>>         |                         |
>>> __device_attach_driver    __device_attach_driver
>>>         |
>>> driver_probe_device
>>>         |
>>> really_probe
>>>         |
>>> dma_configure
>>>
>>>   Similarly on the device/driver_unregister path __device_release_driver is
>>>   called which inturn calls dma_deconfigure.
>>>
>>>   If the ACPI bus code follows the same, we can add acpi_dma_configure
>>>   at the same place as of_dma_configure.
>>>
>>>   This series is based on the recently merged Generic DT bindings for
>>>   PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>>
>>>   This time tested this with platform and pci device for probe deferral
>>>   and reprobe on arm64 based platform. There is an issue on the cleanup
>>>   path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>>   device is attached to an domain in arch_teardown_dma_ops.
>>>   But with iommu_groups created from the iommu driver, the device is always
>>>   attached to a domain/default_domain. So so the WARN has to be removed/handled
>>>   probably.
>>
>>Thanks for continuing work on this feature! Your can add my:
>>
>>Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>
Hi Will,Robin,Joerg,
       
       I have tested the probe deferral for platform/pcie bus devices based on latest Generic bindings
        series merged [1], for pci/arm-smmu.
       It will good to know from you on whats the right way to take this forward ?

Regards,
 Sricharan

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Marek,

>Hi Sricharan,
>
>
>On 2016-10-04 19:03, Sricharan R wrote:
>> Initial post from Laurent Pinchart[1]. This is
>> series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>         |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>         |                         |
>> device_attach           device_initial_probe
>>         |                         |
>> __device_attach_driver    __device_attach_driver
>>         |
>> driver_probe_device
>>         |
>> really_probe
>>         |
>> dma_configure
>>
>>   Similarly on the device/driver_unregister path __device_release_driver is
>>   called which inturn calls dma_deconfigure.
>>
>>   If the ACPI bus code follows the same, we can add acpi_dma_configure
>>   at the same place as of_dma_configure.
>>
>>   This series is based on the recently merged Generic DT bindings for
>>   PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>
>>   This time tested this with platform and pci device for probe deferral
>>   and reprobe on arm64 based platform. There is an issue on the cleanup
>>   path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>   device is attached to an domain in arch_teardown_dma_ops.
>>   But with iommu_groups created from the iommu driver, the device is always
>>   attached to a domain/default_domain. So so the WARN has to be removed/handled
>>   probably.
>
>Thanks for continuing work on this feature! Your can add my:
>
>Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>
  Thanks for testing this. So the for the below fix, the remove_device callback
  gets called on the dma_ops cleanup path, so would it be easy to remove the
  data for the device there ?

Regards,
 Sricharan


>It works fine with Exynos SYSMMU driver, although a patch is needed to fix
>infinite loop due to list corruption (same element is added twice if master
>device fails with deferred probe):
>
>From: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>Date: Mon, 10 Oct 2016 14:22:42 +0200
>Subject: [PATCH] iommu/exynos: ensure that sysmmu is added only once to its
>  master
>
>Since adding IOMMU deferred probing support, of_xlate() callback might
>be called more than once for given master device (for example it happens
>when masters device driver fails with EPROBE_DEFER), so ensure that
>SYSMMU controller is added to its master device (owner) only once.
>
>Signed-off-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>---
>  drivers/iommu/exynos-iommu.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
>diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
>index 30808e91b775..1525a86eb829 100644
>--- a/drivers/iommu/exynos-iommu.c
>+++ b/drivers/iommu/exynos-iommu.c
>@@ -1253,7 +1253,7 @@ static int exynos_iommu_of_xlate(struct device *dev,
>  {
>      struct exynos_iommu_owner *owner = dev->archdata.iommu;
>      struct platform_device *sysmmu = of_find_device_by_node(spec->np);
>-    struct sysmmu_drvdata *data;
>+    struct sysmmu_drvdata *data, *entry;
>
>      if (!sysmmu)
>          return -ENODEV;
>@@ -1271,6 +1271,10 @@ static int exynos_iommu_of_xlate(struct device *dev,
>          dev->archdata.iommu = owner;
>      }
>
>+    list_for_each_entry(entry, &owner->controllers, owner_node)
>+        if (entry == data)
>+            return 0;
>+
>      list_add_tail(&data->owner_node, &owner->controllers);
>      return 0;
>  }
>--
>1.9.1
>

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Marek Szyprowski]

Hi Sricharan,


On 2016-10-12 08:24, Sricharan wrote:
> On 2016-10-04 19:03, Sricharan R wrote:
>>> Initial post from Laurent Pinchart[1]. This is
>>> series calls the dma ops configuration for the devices
>>> at a generic place so that it works for all busses.
>>> The dma_configure_ops for a device is now called during
>>> the device_attach callback just before the probe of the
>>> bus/driver is called. Similarly dma_deconfigure is called during
>>> device/driver_detach path.
>>>
>>>
>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>          |                         |
>>> pci_bus_add_device     (device_add/driver_register)
>>>          |                         |
>>> device_attach           device_initial_probe
>>>          |                         |
>>> __device_attach_driver    __device_attach_driver
>>>          |
>>> driver_probe_device
>>>          |
>>> really_probe
>>>          |
>>> dma_configure
>>>
>>>    Similarly on the device/driver_unregister path __device_release_driver is
>>>    called which inturn calls dma_deconfigure.
>>>
>>>    If the ACPI bus code follows the same, we can add acpi_dma_configure
>>>    at the same place as of_dma_configure.
>>>
>>>    This series is based on the recently merged Generic DT bindings for
>>>    PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy@arm.com [2]
>>>
>>>    This time tested this with platform and pci device for probe deferral
>>>    and reprobe on arm64 based platform. There is an issue on the cleanup
>>>    path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>>    device is attached to an domain in arch_teardown_dma_ops.
>>>    But with iommu_groups created from the iommu driver, the device is always
>>>    attached to a domain/default_domain. So so the WARN has to be removed/handled
>>>    probably.
>> Thanks for continuing work on this feature! Your can add my:
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>
>    Thanks for testing this. So the for the below fix, the remove_device callback
>    gets called on the dma_ops cleanup path, so would it be easy to remove the
>    data for the device there ?

I assumed that IOMMU driver cannot be removed reliably, so all 
structures that it
creates are permanent. I didn't use device_add()/device_remove() 
callbacks, because
in current implementation device_add() is called too late (after 
dma-mapping glue
triggers device_attach_iommu()).

Maybe once your patchset is merged, I will move creation and management 
of the all
IOMMU related structures to device_add/remove callbacks.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Marek,

>>>> Initial post from Laurent Pinchart[1]. This is
>>>> series calls the dma ops configuration for the devices
>>>> at a generic place so that it works for all busses.
>>>> The dma_configure_ops for a device is now called during
>>>> the device_attach callback just before the probe of the
>>>> bus/driver is called. Similarly dma_deconfigure is called during
>>>> device/driver_detach path.
>>>>
>>>>
>>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>>          |                         |
>>>> pci_bus_add_device     (device_add/driver_register)
>>>>          |                         |
>>>> device_attach           device_initial_probe
>>>>          |                         |
>>>> __device_attach_driver    __device_attach_driver
>>>>          |
>>>> driver_probe_device
>>>>          |
>>>> really_probe
>>>>          |
>>>> dma_configure
>>>>
>>>>    Similarly on the device/driver_unregister path __device_release_driver is
>>>>    called which inturn calls dma_deconfigure.
>>>>
>>>>    If the ACPI bus code follows the same, we can add acpi_dma_configure
>>>>    at the same place as of_dma_configure.
>>>>
>>>>    This series is based on the recently merged Generic DT bindings for
>>>>    PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>>>
>>>>    This time tested this with platform and pci device for probe deferral
>>>>    and reprobe on arm64 based platform. There is an issue on the cleanup
>>>>    path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>>>    device is attached to an domain in arch_teardown_dma_ops.
>>>>    But with iommu_groups created from the iommu driver, the device is always
>>>>    attached to a domain/default_domain. So so the WARN has to be removed/handled
>>>>    probably.
>>> Thanks for continuing work on this feature! Your can add my:
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>
>>    Thanks for testing this. So the for the below fix, the remove_device callback
>>    gets called on the dma_ops cleanup path, so would it be easy to remove the
>>    data for the device there ?
>
>I assumed that IOMMU driver cannot be removed reliably, so all
>structures that it
>creates are permanent. I didn't use device_add()/device_remove()
>callbacks, because
>in current implementation device_add() is called too late (after
>dma-mapping glue
>triggers device_attach_iommu()).
>
>Maybe once your patchset is merged, I will move creation and management
>of the all
>IOMMU related structures to device_add/remove callbacks.
>
    ok understand it.

Regards,
Sricharan

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Resending, missed out on the link last time.

>-----Original Message-----
>From: linux-arm-msm-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-arm-msm-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Marek Szyprowski
>Sent: Monday, October 10, 2016 6:07 PM
>To: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>; will.deacon-5wv7dgnIgG8@public.gmane.org; robin.murphy-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
>foundation.org; linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org;
>tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
>Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
>
>Hi Sricharan,
>
>
>On 2016-10-04 19:03, Sricharan R wrote:
>> Initial post from Laurent Pinchart[1]. This is
>> series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>         |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>         |                         |
>> device_attach           device_initial_probe
>>         |                         |
>> __device_attach_driver    __device_attach_driver
>>         |
>> driver_probe_device
>>         |
>> really_probe
>>         |
>> dma_configure
>>
>>   Similarly on the device/driver_unregister path __device_release_driver
>> is
>>   called which inturn calls dma_deconfigure.
>>
>>   If the ACPI bus code follows the same, we can add acpi_dma_configure
>>   at the same place as of_dma_configure.
>>
>>   This series is based on the recently merged Generic DT bindings for
>>   PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>
>>   This time tested this with platform and pci device for probe deferral
>>   and reprobe on arm64 based platform. There is an issue on the cleanup
>>   path for arm64 though, where there is WARN_ON if the dma_ops is reset
>> while
>>   device is attached to an domain in arch_teardown_dma_ops.
>>   But with iommu_groups created from the iommu driver, the device is
>> always
>>   attached to a domain/default_domain. So so the WARN has to be
>> removed/handled
>>   probably.
>
>Thanks for continuing work on this feature! Your can add my:
>
>Tested-by: Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>

Hi Will,Robin,Joerg,
       
       I have tested the probe deferral for platform/pcie bus devices based on latest Generic DT bindings
        series merged [1], for pci/arm-smmu.
       It will be good to know from you on whats the right way to take this forward ?

[1] http://www.spinics.net/lists/devicetree/msg142943.html

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Archit Taneja]

On 10/04/2016 10:33 PM, Sricharan R wrote:
> Initial post from Laurent Pinchart[1]. This is
> series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
>
>
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
>
>  Similarly on the device/driver_unregister path __device_release_driver is
>  called which inturn calls dma_deconfigure.
>
>  If the ACPI bus code follows the same, we can add acpi_dma_configure
>  at the same place as of_dma_configure.
>
>  This series is based on the recently merged Generic DT bindings for
>  PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>
>  This time tested this with platform and pci device for probe deferral
>  and reprobe on arm64 based platform. There is an issue on the cleanup
>  path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>  device is attached to an domain in arch_teardown_dma_ops.
>  But with iommu_groups created from the iommu driver, the device is always
>  attached to a domain/default_domain. So so the WARN has to be removed/handled
>  probably.
>

Tested-by: Archit Taneja <architt-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>

>  Previous post of this series [3].
>
>  [V3]
>      * Removed the patch to split dma_masks/dma_ops configuration separately
>        based on review comments that both masks and ops are required only
>        during the device probe time.
>
>      * Reworked the series based on Generic DT bindings series [2].
>
>      * Added call to iommu's remove_device in the cleanup path for arm and arm64.
>
>      * Removed the notifier trick in arm64 to handle early device registration.
>
>      * Added reset of dma_ops in cleanup path for arm based on comments.
>
>      * Fixed the pci_iommu_configure path and tested with PCI device as well.
>
>      * Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.
>
>      * Fixed few other cosmetic comments.
>
>  [V2]
>      * Updated the Initial post to call dma_configure/deconfigure from generic code
>
>      * Added iommu add_device callback from of_iommu_configure path
>
>  [V1]
>      * Initial post
>
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
> [2] http://www.spinics.net/lists/devicetree/msg142943.html
> [3] https://www.mail-archive.com/iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org/msg13941.html
> [4] https://www.mail-archive.com/iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org/msg13940.html
>
>
>
> Laurent Pinchart (4):
>   arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
>   of: dma: Move range size workaround to of_dma_get_range()
>   of: dma: Make of_dma_deconfigure() public
>   iommu: of: Handle IOMMU lookup failure with deferred probing or error
>
> Sricharan R (4):
>   drivers: platform: Configure dma operations at probe time
>   arm: dma-mapping: Reset the device's dma_ops
>   arm/arm64: dma-mapping: Call iommu's remove_device callback during
>     device detach
>   arm64: dma-mapping: Remove the notifier trick to handle early setting
>     of dma_ops
>
>  arch/arm/mm/dma-mapping.c   |  18 ++++++++
>  arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
>  drivers/base/dd.c           |  10 +++++
>  drivers/base/dma-mapping.c  |  11 +++++
>  drivers/iommu/of_iommu.c    |  47 +++++++++++++++++--
>  drivers/of/address.c        |  20 ++++++++-
>  drivers/of/device.c         |  34 +++++++-------
>  drivers/of/platform.c       |   9 ----
>  drivers/pci/probe.c         |   5 +--
>  include/linux/dma-mapping.h |   3 ++
>  include/linux/of_device.h   |   7 ++-
>  11 files changed, 138 insertions(+), 133 deletions(-)
>

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH V3 2/8] of: dma: Move range size workaround to of_dma_get_range()

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ EXPORT_SYMBOL(of_io_request_and_map);
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index fd5cfad..d9898d9 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V3 3/8] of: dma: Make of_dma_deconfigure() public

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index d9898d9..1c843e2 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index f39ccd5..9cb7090 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -153,11 +153,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index cc7dd687..d20a31a 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,
@@ -100,6 +101,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

[PATCH V3 5/8] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

The current iommu framework handles pci and non-pci devices separately,
so taken care of both the paths in this patch. The iommu's add_device
callback is invoked after the master's configuration data is added in
xlate. This is needed because the iommu core calls add_device callback
during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
that call has to be removed.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 drivers/iommu/of_iommu.c  | 47 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/of/device.c       |  7 ++++++-
 include/linux/of_device.h |  6 ++++--
 3 files changed, 53 insertions(+), 7 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 5b82862..1a5e28b 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -23,6 +23,7 @@
 #include <linux/of.h>
 #include <linux/of_iommu.h>
 #include <linux/of_pci.h>
+#include <linux/pci.h>
 #include <linux/slab.h>
 
 static const struct of_device_id __iommu_of_table_sentinel
@@ -167,12 +168,29 @@ static const struct iommu_ops
 		return NULL;
 
 	ops = of_iommu_get_ops(iommu_spec.np);
+
+	if (!ops) {
+		const struct of_device_id *oid;
+
+		oid = of_match_node(&__iommu_of_table, iommu_spec.np);
+		ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+		return ops;
+	}
+
 	if (!ops || !ops->of_xlate ||
 	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
 	    ops->of_xlate(&pdev->dev, &iommu_spec))
 		ops = NULL;
 
+	if (ops && ops->add_device) {
+		ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;
+
+		if (!ops)
+			dev_err(&pdev->dev, "Failed to setup iommu ops\n");
+	}
+
 	of_node_put(iommu_spec.np);
+
 	return ops;
 }
 
@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
+	struct device *bridge;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
+		if (bridge && bridge->parent && bridge->parent->of_node)
+			return of_pci_iommu_configure(to_pci_dev(dev),
+						      bridge->parent->of_node);
+	}
 
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
@@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 		np = iommu_spec.np;
 		ops = of_iommu_get_ops(np);
 
+		if (!ops) {
+			const struct of_device_id *oid;
+
+			oid = of_match_node(&__iommu_of_table, np);
+			ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+			goto err_put_node;
+		}
+
 		if (!ops || !ops->of_xlate ||
 		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
 		    ops->of_xlate(dev, &iommu_spec))
@@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 		idx++;
 	}
 
+	if (ops && ops->add_device) {
+		ops = (ops->add_device(dev) == 0) ? ops : NULL;
+
+		if (!ops)
+			dev_err(dev, "Failed to setup iommu_ops\n");
+	}
+
 	return ops;
 
 err_put_node:
 	of_node_put(np);
-	return NULL;
+	return ops;
 }
 
 static int __init of_iommu_init(void)
@@ -222,7 +261,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 1c843e2..c8e74d7 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index d20a31a..2bdb872 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
@@ -100,7 +100,9 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return NULL;
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 5/8] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Robin Murphy]

On 04/10/16 18:03, Sricharan R wrote:
> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
> 
> Failures to look up an IOMMU when parsing the DT iommus property need to
> be handled separately from the .of_xlate() failures to support deferred
> probing.
> 
> The lack of a registered IOMMU can be caused by the lack of a driver for
> the IOMMU, the IOMMU device probe not having been performed yet, having
> been deferred, or having failed.
> 
> The first case occurs when the device tree describes the bus master and
> IOMMU topology correctly but no device driver exists for the IOMMU yet
> or the device driver has not been compiled in. Return NULL, the caller
> will configure the device without an IOMMU.
> 
> The second and third cases are handled by deferring the probe of the bus
> master device which will eventually get reprobed after the IOMMU.
> 
> The last case is currently handled by deferring the probe of the bus
> master device as well. A mechanism to either configure the bus master
> device without an IOMMU or to fail the bus master device probe depending
> on whether the IOMMU is optional or mandatory would be a good
> enhancement.
> 
> The current iommu framework handles pci and non-pci devices separately,
> so taken care of both the paths in this patch. The iommu's add_device
> callback is invoked after the master's configuration data is added in
> xlate. This is needed because the iommu core calls add_device callback
> during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
> that call has to be removed.

Laurent's signoff seems to have gone missing here.

> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/of_iommu.c  | 47 +++++++++++++++++++++++++++++++++++++++++++----
>  drivers/of/device.c       |  7 ++++++-
>  include/linux/of_device.h |  6 ++++--
>  3 files changed, 53 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
> index 5b82862..1a5e28b 100644
> --- a/drivers/iommu/of_iommu.c
> +++ b/drivers/iommu/of_iommu.c
> @@ -23,6 +23,7 @@
>  #include <linux/of.h>
>  #include <linux/of_iommu.h>
>  #include <linux/of_pci.h>
> +#include <linux/pci.h>
>  #include <linux/slab.h>
>  
>  static const struct of_device_id __iommu_of_table_sentinel
> @@ -167,12 +168,29 @@ static const struct iommu_ops
>  		return NULL;
>  
>  	ops = of_iommu_get_ops(iommu_spec.np);
> +
> +	if (!ops) {
> +		const struct of_device_id *oid;
> +
> +		oid = of_match_node(&__iommu_of_table, iommu_spec.np);
> +		ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;

It would seem even simpler and more convenient to roll this into the end
of of_iommu_get_ops():

	if (!ops && of_match_node(&__iommu_of_table, iommu_spec.np))
		ops = ERR_PTR(-EPROBE_DEFER);

then just fix up the existing !ops checks to !IS_ERR(ops) appropriately.

> +		return ops;
> +	}
> +
>  	if (!ops || !ops->of_xlate ||
>  	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
>  	    ops->of_xlate(&pdev->dev, &iommu_spec))
>  		ops = NULL;
>  
> +	if (ops && ops->add_device) {
> +		ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;

This is a really obtuse way of writing

	if (ops->add_device(...))
		ops = NULL;

However, given that we're now returning an ERR_PTR, it would be worth
capturing the return value of add_device and propagating the error back
up - if the IOMMU driver has refused one of its masters for some reason,
it probably isn't safe to allow that device to do DMA either way, so we
ought to prevent it probing at all.

> +
> +		if (!ops)
> +			dev_err(&pdev->dev, "Failed to setup iommu ops\n");

Given the above, I think this should be more along the lines of "Device
rejected by IOMMU: %d" with the actual error code as well. It's one of
those "if you ever see it, you're going to have to debug it" cases, so
the clearer the better.

> +	}
> +
>  	of_node_put(iommu_spec.np);
> +
>  	return ops;
>  }
>  
> @@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>  	struct device_node *np;
>  	const struct iommu_ops *ops = NULL;
>  	int idx = 0;
> +	struct device *bridge;
> +
> +	if (dev_is_pci(dev)) {
> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>  
> -	if (dev_is_pci(dev))
> -		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
> +		if (bridge && bridge->parent && bridge->parent->of_node)
> +			return of_pci_iommu_configure(to_pci_dev(dev),
> +						      bridge->parent->of_node);

		else fall through to treating it as a platform device?

...that's not right. Anyway, this is PCI-specific stuff so should be in
the PCI-specific helper function.

> +	}
>  
>  	/*
>  	 * We don't currently walk up the tree looking for a parent IOMMU.
> @@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>  		np = iommu_spec.np;
>  		ops = of_iommu_get_ops(np);
>  
> +		if (!ops) {
> +			const struct of_device_id *oid;
> +
> +			oid = of_match_node(&__iommu_of_table, np);
> +			ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
> +			goto err_put_node;
> +		}

Same comment as above. Especially since moving it to of_iommu_get_ops()
would obviate the duplication.

> +
>  		if (!ops || !ops->of_xlate ||
>  		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
>  		    ops->of_xlate(dev, &iommu_spec))
> @@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>  		idx++;
>  	}
>  
> +	if (ops && ops->add_device) {
> +		ops = (ops->add_device(dev) == 0) ? ops : NULL;
> +
> +		if (!ops)
> +			dev_err(dev, "Failed to setup iommu_ops\n");
> +	}
> +

It would be nice to avoid duplicating this as well.

Robin.

>  	return ops;
>  
>  err_put_node:
>  	of_node_put(np);
> -	return NULL;
> +	return ops;
>  }
>  
>  static int __init of_iommu_init(void)
> @@ -222,7 +261,7 @@ static int __init of_iommu_init(void)
>  	for_each_matching_node_and_match(np, matches, &match) {
>  		const of_iommu_init_fn init_fn = match->data;
>  
> -		if (init_fn(np))
> +		if (init_fn && init_fn(np))
>  			pr_err("Failed to initialise IOMMU %s\n",
>  				of_node_full_name(np));
>  	}
> diff --git a/drivers/of/device.c b/drivers/of/device.c
> index 1c843e2..c8e74d7 100644
> --- a/drivers/of/device.c
> +++ b/drivers/of/device.c
> @@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
>   * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
>   * to fix up DMA configuration.
>   */
> -void of_dma_configure(struct device *dev, struct device_node *np)
> +int of_dma_configure(struct device *dev, struct device_node *np)
>  {
>  	u64 dma_addr, paddr, size;
>  	int ret;
> @@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
>  		coherent ? " " : " not ");
>  
>  	iommu = of_iommu_configure(dev, np);
> +	if (IS_ERR(iommu))
> +		return PTR_ERR(iommu);
> +
>  	dev_dbg(dev, "device is%sbehind an iommu\n",
>  		iommu ? " " : " not ");
>  
>  	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
> +
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(of_dma_configure);
>  
> diff --git a/include/linux/of_device.h b/include/linux/of_device.h
> index d20a31a..2bdb872 100644
> --- a/include/linux/of_device.h
> +++ b/include/linux/of_device.h
> @@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  	return of_node_get(cpu_dev->of_node);
>  }
>  
> -void of_dma_configure(struct device *dev, struct device_node *np);
> +int of_dma_configure(struct device *dev, struct device_node *np);
>  void of_dma_deconfigure(struct device *dev);
>  #else /* CONFIG_OF */
>  
> @@ -100,7 +100,9 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
>  	return NULL;
>  }
>  static inline void of_dma_configure(struct device *dev, struct device_node *np)
> -{}
> +{
> +	return 0;
> +}
>  static inline void of_dma_deconfigure(struct device *dev)
>  {}
>  #endif /* CONFIG_OF */
> 

RE: [PATCH V3 5/8] iommu: of: Handle IOMMU lookup failure with deferred probing or error

[Sricharan]

Hi Robin,

>> From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
>>
>> Failures to look up an IOMMU when parsing the DT iommus property need to
>> be handled separately from the .of_xlate() failures to support deferred
>> probing.
>>
>> The lack of a registered IOMMU can be caused by the lack of a driver for
>> the IOMMU, the IOMMU device probe not having been performed yet, having
>> been deferred, or having failed.
>>
>> The first case occurs when the device tree describes the bus master and
>> IOMMU topology correctly but no device driver exists for the IOMMU yet
>> or the device driver has not been compiled in. Return NULL, the caller
>> will configure the device without an IOMMU.
>>
>> The second and third cases are handled by deferring the probe of the bus
>> master device which will eventually get reprobed after the IOMMU.
>>
>> The last case is currently handled by deferring the probe of the bus
>> master device as well. A mechanism to either configure the bus master
>> device without an IOMMU or to fail the bus master device probe depending
>> on whether the IOMMU is optional or mandatory would be a good
>> enhancement.
>>
>> The current iommu framework handles pci and non-pci devices separately,
>> so taken care of both the paths in this patch. The iommu's add_device
>> callback is invoked after the master's configuration data is added in
>> xlate. This is needed because the iommu core calls add_device callback
>> during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
>> that call has to be removed.
>
>Laurent's signoff seems to have gone missing here.

Ah, preserved his authorship, but missed this. Will add it back.

>
>> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>> ---
>>  drivers/iommu/of_iommu.c  | 47 +++++++++++++++++++++++++++++++++++++++++++----
>>  drivers/of/device.c       |  7 ++++++-
>>  include/linux/of_device.h |  6 ++++--
>>  3 files changed, 53 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
>> index 5b82862..1a5e28b 100644
>> --- a/drivers/iommu/of_iommu.c
>> +++ b/drivers/iommu/of_iommu.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/of.h>
>>  #include <linux/of_iommu.h>
>>  #include <linux/of_pci.h>
>> +#include <linux/pci.h>
>>  #include <linux/slab.h>
>>
>>  static const struct of_device_id __iommu_of_table_sentinel
>> @@ -167,12 +168,29 @@ static const struct iommu_ops
>>  		return NULL;
>>
>>  	ops = of_iommu_get_ops(iommu_spec.np);
>> +
>> +	if (!ops) {
>> +		const struct of_device_id *oid;
>> +
>> +		oid = of_match_node(&__iommu_of_table, iommu_spec.np);
>> +		ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
>
>It would seem even simpler and more convenient to roll this into the end
>of of_iommu_get_ops():

ok, understand. Will move this there.

>
>	if (!ops && of_match_node(&__iommu_of_table, iommu_spec.np))
>		ops = ERR_PTR(-EPROBE_DEFER);
>
>then just fix up the existing !ops checks to !IS_ERR(ops) appropriately.

ok.

>
>> +		return ops;
>> +	}
>> +
>>  	if (!ops || !ops->of_xlate ||
>>  	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
>>  	    ops->of_xlate(&pdev->dev, &iommu_spec))
>>  		ops = NULL;
>>
>> +	if (ops && ops->add_device) {
>> +		ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;
>
>This is a really obtuse way of writing
>
>	if (ops->add_device(...))
>		ops = NULL;

ok, will change it this way.

>
>However, given that we're now returning an ERR_PTR, it would be worth
>capturing the return value of add_device and propagating the error back
>up - if the IOMMU driver has refused one of its masters for some reason,
>it probably isn't safe to allow that device to do DMA either way, so we
>ought to prevent it probing at all.
>

ok, will return the err value instead of NULL here.

>> +
>> +		if (!ops)
>> +			dev_err(&pdev->dev, "Failed to setup iommu ops\n");
>
>Given the above, I think this should be more along the lines of "Device
>rejected by IOMMU: %d" with the actual error code as well. It's one of
>those "if you ever see it, you're going to have to debug it" cases, so
>the clearer the better.
>

ok, will reword the print.

>> +	}
>> +
>>  	of_node_put(iommu_spec.np);
>> +
>>  	return ops;
>>  }
>>
>> @@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>  	struct device_node *np;
>>  	const struct iommu_ops *ops = NULL;
>>  	int idx = 0;
>> +	struct device *bridge;
>> +
>> +	if (dev_is_pci(dev)) {
>> +		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>
>> -	if (dev_is_pci(dev))
>> -		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
>> +		if (bridge && bridge->parent && bridge->parent->of_node)
>> +			return of_pci_iommu_configure(to_pci_dev(dev),
>> +						      bridge->parent->of_node);
>
>		else fall through to treating it as a platform device?
>

ha, surely wrong. Will correct this and move it to the of_pci_iommu_configure helper.

>...that's not right. Anyway, this is PCI-specific stuff so should be in
>the PCI-specific helper function.
>
>> +	}
>>
>>  	/*
>>  	 * We don't currently walk up the tree looking for a parent IOMMU.
>> @@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>  		np = iommu_spec.np;
>>  		ops = of_iommu_get_ops(np);
>>
>> +		if (!ops) {
>> +			const struct of_device_id *oid;
>> +
>> +			oid = of_match_node(&__iommu_of_table, np);
>> +			ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
>> +			goto err_put_node;
>> +		}
>
>Same comment as above. Especially since moving it to of_iommu_get_ops()
>would obviate the duplication.

ok.

>
>> +
>>  		if (!ops || !ops->of_xlate ||
>>  		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
>>  		    ops->of_xlate(dev, &iommu_spec))
>> @@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
>>  		idx++;
>>  	}
>>
>> +	if (ops && ops->add_device) {
>> +		ops = (ops->add_device(dev) == 0) ? ops : NULL;
>> +
>> +		if (!ops)
>> +			dev_err(dev, "Failed to setup iommu_ops\n");
>> +	}
>> +
>
>It would be nice to avoid duplicating this as well.

ok, sure, will correct.

Regards,
 Sricharan

[PATCH V3 7/8] arm/arm64: dma-mapping: Call iommu's remove_device callback during device detach

[Sricharan R]

dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
which is called when the device gets detached from the driver.
When the device was added, iommu's add_device callback was used to
add the device in to its iommu_group and setup the device to be ready
to use its iommu. Similarly, call remove_device callback to remove the
device from the group and reset any other device's iommu configurations.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 arch/arm/mm/dma-mapping.c   | 8 ++++++++
 arch/arm64/mm/dma-mapping.c | 7 +++++++
 2 files changed, 15 insertions(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index b9191f0..cbe22de 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
 static void arm_teardown_iommu_dma_ops(struct device *dev)
 {
 	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+	const struct iommu_ops *ops;
 
 	if (!mapping)
 		return;
 
 	__arm_iommu_detach_device(dev);
+
+	if (dev->iommu_fwspec) {
+		ops = dev->iommu_fwspec->ops;
+		if (ops->remove_device)
+			ops->remove_device(dev);
+	}
+
 	arm_iommu_release_mapping(mapping);
 	set_dma_ops(dev, NULL);
 }
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 610d8e5..faf4b92 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -938,6 +938,13 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 void arch_teardown_dma_ops(struct device *dev)
 {
 	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	const struct iommu_ops *ops;
+
+	if (dev->iommu_fwspec) {
+		ops = dev->iommu_fwspec->ops;
+		if (ops->remove_device)
+			ops->remove_device(dev);
+	}
 
 	if (WARN_ON(domain))
 		iommu_detach_device(domain, dev);
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH V3 7/8] arm/arm64: dma-mapping: Call iommu's remove_device callback during device detach

[Robin Murphy]

On 04/10/16 18:03, Sricharan R wrote:
> dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
> which is called when the device gets detached from the driver.
> When the device was added, iommu's add_device callback was used to
> add the device in to its iommu_group and setup the device to be ready
> to use its iommu. Similarly, call remove_device callback to remove the
> device from the group and reset any other device's iommu configurations.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  arch/arm/mm/dma-mapping.c   | 8 ++++++++
>  arch/arm64/mm/dma-mapping.c | 7 +++++++
>  2 files changed, 15 insertions(+)
> 
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index b9191f0..cbe22de 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  static void arm_teardown_iommu_dma_ops(struct device *dev)
>  {
>  	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
> +	const struct iommu_ops *ops;
>  
>  	if (!mapping)
>  		return;
>  
>  	__arm_iommu_detach_device(dev);
> +
> +	if (dev->iommu_fwspec) {
> +		ops = dev->iommu_fwspec->ops;
> +		if (ops->remove_device)
> +			ops->remove_device(dev);
> +	}
> +

Yuck. It's a little unfortunate that we have to do this at all, but I
see why. Still, it should be done in common code, not duplicated across
arch code, not least for symmetry with where the matching add_device
happened (although I think of_dma_deconfigure() would suffice, I'm not
sure we really need to add an of_iommu_deconfigure() just for this).

It's also broken for IOMMU drivers which rely on the
of_iommu_configure() mechanism but have not yet been converted to use
iommu_fwspec (Exynos, MSM, etc.)

Robin.

>  	arm_iommu_release_mapping(mapping);
>  	set_dma_ops(dev, NULL);
>  }
> diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
> index 610d8e5..faf4b92 100644
> --- a/arch/arm64/mm/dma-mapping.c
> +++ b/arch/arm64/mm/dma-mapping.c
> @@ -938,6 +938,13 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  void arch_teardown_dma_ops(struct device *dev)
>  {
>  	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> +	const struct iommu_ops *ops;
> +
> +	if (dev->iommu_fwspec) {
> +		ops = dev->iommu_fwspec->ops;
> +		if (ops->remove_device)
> +			ops->remove_device(dev);
> +	}
>  
>  	if (WARN_ON(domain))
>  		iommu_detach_device(domain, dev);
> 

RE: [PATCH V3 7/8] arm/arm64: dma-mapping: Call iommu's remove_device callback during device detach

[Sricharan]

Hi Robin,

>> dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
>> which is called when the device gets detached from the driver.
>> When the device was added, iommu's add_device callback was used to
>> add the device in to its iommu_group and setup the device to be ready
>> to use its iommu. Similarly, call remove_device callback to remove the
>> device from the group and reset any other device's iommu configurations.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  arch/arm/mm/dma-mapping.c   | 8 ++++++++
>>  arch/arm64/mm/dma-mapping.c | 7 +++++++
>>  2 files changed, 15 insertions(+)
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index b9191f0..cbe22de 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
>>  static void arm_teardown_iommu_dma_ops(struct device *dev)
>>  {
>>  	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
>> +	const struct iommu_ops *ops;
>>
>>  	if (!mapping)
>>  		return;
>>
>>  	__arm_iommu_detach_device(dev);
>> +
>> +	if (dev->iommu_fwspec) {
>> +		ops = dev->iommu_fwspec->ops;
>> +		if (ops->remove_device)
>> +			ops->remove_device(dev);
>> +	}
>> +
>
>Yuck. It's a little unfortunate that we have to do this at all, but I
>see why. Still, it should be done in common code, not duplicated across
>arch code, not least for symmetry with where the matching add_device
>happened (although I think of_dma_deconfigure() would suffice, I'm not
>sure we really need to add an of_iommu_deconfigure() just for this).
>

 So one way is its already called via a the BUS_NOTIFY_REMOVED_DEVICE
 notifier in iommu_bus_notifier. I put it here, one for symmetry and
 another thinking that the remove_device callback should be done
 before the dma_ops is set to NULL. If thats not required then the existing
 iommu_bus_notifier itself can do the cleanup. The corresponding
 add_device in that notifier is dummy now, i will add a patch to remove
 that. If not the whole thing, we can add of_iommu_deconfigure as well.

>It's also broken for IOMMU drivers which rely on the
>of_iommu_configure() mechanism but have not yet been converted to use
>iommu_fwspec (Exynos, MSM, etc.)
>

 I did this, because fwspec was the right way to get ops in the future,
 but yes its getting broken for those which are not converted,
 will have to use dev->bus->iommu_ops in those cases. Will address this
 while changing the above.

Regards,
 Sricharan

 

[PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after the
device's iommu is probed, the notifier trick required to handle the early
setup of dma_ops before the iommu group gets created is not required.
So removing the notifier's here.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
---
 arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
 1 file changed, 3 insertions(+), 97 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index faf4b92..eb593af 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
 static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 			   u64 dma_base, u64 size)
 {
@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 	return true;
 }
 
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
 static int __init __iommu_dma_init(void)
 {
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
+	return iommu_dma_init();
 }
 arch_initcall(__iommu_dma_init);
 
@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 
 	if (!ops)
 		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
+
 	group = iommu_group_get(dev);
+
 	if (group) {
 		do_iommu_attach(dev, ops, dma_base, size);
 		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
 	}
 }
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

RE: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan]

Hi,

>-----Original Message-----
>From: linux-arm-kernel [mailto:linux-arm-kernel-bounces-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org] On Behalf Of Sricharan R
>Sent: Tuesday, October 04, 2016 10:34 PM
>To: will.deacon-5wv7dgnIgG8@public.gmane.org; robin.murphy-5wv7dgnIgG8@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; linux-arm-
>kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org;
>tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org
>Cc: sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
>Subject: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops
>
>With arch_setup_dma_ops now being called late during device's probe after the
>device's iommu is probed, the notifier trick required to handle the early
>setup of dma_ops before the iommu group gets created is not required.
>So removing the notifier's here.
>
>Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>---
> arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
> 1 file changed, 3 insertions(+), 97 deletions(-)
>
>diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
>index faf4b92..eb593af 100644
>--- a/arch/arm64/mm/dma-mapping.c
>+++ b/arch/arm64/mm/dma-mapping.c
>@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
> 	.mapping_error = iommu_dma_mapping_error,
> };
>
>-/*
>- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
>- * everything it needs to - the device is only partially created and the
>- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
>- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
>- * to move the arch_setup_dma_ops() call later, all the notifier bits below
>- * become unnecessary, and will go away.
>- */
>-struct iommu_dma_notifier_data {
>-	struct list_head list;
>-	struct device *dev;
>-	const struct iommu_ops *ops;
>-	u64 dma_base;
>-	u64 size;
>-};
>-static LIST_HEAD(iommu_dma_masters);
>-static DEFINE_MUTEX(iommu_dma_notifier_lock);
>-
> static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
> 			   u64 dma_base, u64 size)
> {
>@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
> 	return true;
> }
>
>-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>-			      u64 dma_base, u64 size)
>-{
>-	struct iommu_dma_notifier_data *iommudata;
>-
>-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
>-	if (!iommudata)
>-		return;
>-
>-	iommudata->dev = dev;
>-	iommudata->ops = ops;
>-	iommudata->dma_base = dma_base;
>-	iommudata->size = size;
>-
>-	mutex_lock(&iommu_dma_notifier_lock);
>-	list_add(&iommudata->list, &iommu_dma_masters);
>-	mutex_unlock(&iommu_dma_notifier_lock);
>-}
>-
>-static int __iommu_attach_notifier(struct notifier_block *nb,
>-				   unsigned long action, void *data)
>-{
>-	struct iommu_dma_notifier_data *master, *tmp;
>-
>-	if (action != BUS_NOTIFY_BIND_DRIVER)
>-		return 0;
>-
>-	mutex_lock(&iommu_dma_notifier_lock);
>-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
>-		if (data == master->dev && do_iommu_attach(master->dev,
>-				master->ops, master->dma_base, master->size)) {
>-			list_del(&master->list);
>-			kfree(master);
>-			break;
>-		}
>-	}
>-	mutex_unlock(&iommu_dma_notifier_lock);
>-	return 0;
>-}
>-
>-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
>-{
>-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
>-	int ret;
>-
>-	if (!nb)
>-		return -ENOMEM;
>-
>-	nb->notifier_call = __iommu_attach_notifier;
>-
>-	ret = bus_register_notifier(bus, nb);
>-	if (ret) {
>-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
>-			bus->name);
>-		kfree(nb);
>-	}
>-	return ret;
>-}
>-
> static int __init __iommu_dma_init(void)
> {
>-	int ret;
>-
>-	ret = iommu_dma_init();
>-	if (!ret)
>-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
>-	if (!ret)
>-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
>-#ifdef CONFIG_PCI
>-	if (!ret)
>-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
>-#endif
>-	return ret;
>+	return iommu_dma_init();
> }
> arch_initcall(__iommu_dma_init);
>
>@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>
> 	if (!ops)
> 		return;
>-	/*
>-	 * TODO: As a concession to the future, we're ready to handle being
>-	 * called both early and late (i.e. after bus_add_device). Once all
>-	 * the platform bus code is reworked to call us late and the notifier
>-	 * junk above goes away, move the body of do_iommu_attach here.
>-	 */
>+
> 	group = iommu_group_get(dev);
>+
> 	if (group) {
> 		do_iommu_attach(dev, ops, dma_base, size);
> 		iommu_group_put(group);
>-	} else {
>-		queue_iommu_attach(dev, ops, dma_base, size);
> 	}
> }
>
I should have has this as well for being removed,

From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
Date: Fri, 7 Oct 2016 19:20:21 +0530
Subject: [PATCH] iommu:/arm-smmu: Avoid early iommu device registration

of_platform_device_create was called early in the init
to have the smmu probed before the master. But now with
the probe deferral, this is not needed.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/iommu/arm-smmu.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index c841eb7..083489e4 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2066,9 +2066,6 @@ static int __init arm_smmu_of_init(struct device_node *np)
 	if (ret)
 		return ret;
 
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
 	return 0;
 }
 IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);
-- 
1.8.2.1

Regards,
 Sricharan

Re: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Robin Murphy]

On 04/10/16 18:03, Sricharan R wrote:
> With arch_setup_dma_ops now being called late during device's probe after the
> device's iommu is probed, the notifier trick required to handle the early
> setup of dma_ops before the iommu group gets created is not required.
> So removing the notifier's here.

Hooray!

> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
>  1 file changed, 3 insertions(+), 97 deletions(-)
> 
> diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
> index faf4b92..eb593af 100644
> --- a/arch/arm64/mm/dma-mapping.c
> +++ b/arch/arm64/mm/dma-mapping.c
> @@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
>  	.mapping_error = iommu_dma_mapping_error,
>  };
>  
> -/*
> - * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
> - * everything it needs to - the device is only partially created and the
> - * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
> - * need this delayed attachment dance. Once IOMMU probe ordering is sorted
> - * to move the arch_setup_dma_ops() call later, all the notifier bits below
> - * become unnecessary, and will go away.
> - */
> -struct iommu_dma_notifier_data {
> -	struct list_head list;
> -	struct device *dev;
> -	const struct iommu_ops *ops;
> -	u64 dma_base;
> -	u64 size;
> -};
> -static LIST_HEAD(iommu_dma_masters);
> -static DEFINE_MUTEX(iommu_dma_notifier_lock);
> -
>  static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>  			   u64 dma_base, u64 size)

This should go as well...

>  {
> @@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>  	return true;
>  }
>  
> -static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
> -			      u64 dma_base, u64 size)
> -{
> -	struct iommu_dma_notifier_data *iommudata;
> -
> -	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
> -	if (!iommudata)
> -		return;
> -
> -	iommudata->dev = dev;
> -	iommudata->ops = ops;
> -	iommudata->dma_base = dma_base;
> -	iommudata->size = size;
> -
> -	mutex_lock(&iommu_dma_notifier_lock);
> -	list_add(&iommudata->list, &iommu_dma_masters);
> -	mutex_unlock(&iommu_dma_notifier_lock);
> -}
> -
> -static int __iommu_attach_notifier(struct notifier_block *nb,
> -				   unsigned long action, void *data)
> -{
> -	struct iommu_dma_notifier_data *master, *tmp;
> -
> -	if (action != BUS_NOTIFY_BIND_DRIVER)
> -		return 0;
> -
> -	mutex_lock(&iommu_dma_notifier_lock);
> -	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
> -		if (data == master->dev && do_iommu_attach(master->dev,
> -				master->ops, master->dma_base, master->size)) {
> -			list_del(&master->list);
> -			kfree(master);
> -			break;
> -		}
> -	}
> -	mutex_unlock(&iommu_dma_notifier_lock);
> -	return 0;
> -}
> -
> -static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
> -{
> -	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
> -	int ret;
> -
> -	if (!nb)
> -		return -ENOMEM;
> -
> -	nb->notifier_call = __iommu_attach_notifier;
> -
> -	ret = bus_register_notifier(bus, nb);
> -	if (ret) {
> -		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
> -			bus->name);
> -		kfree(nb);
> -	}
> -	return ret;
> -}
> -
>  static int __init __iommu_dma_init(void)
>  {
> -	int ret;
> -
> -	ret = iommu_dma_init();
> -	if (!ret)
> -		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
> -	if (!ret)
> -		ret = register_iommu_dma_ops_notifier(&amba_bustype);
> -#ifdef CONFIG_PCI
> -	if (!ret)
> -		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
> -#endif
> -	return ret;
> +	return iommu_dma_init();
>  }
>  arch_initcall(__iommu_dma_init);
>  
> @@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  
>  	if (!ops)
>  		return;
> -	/*
> -	 * TODO: As a concession to the future, we're ready to handle being
> -	 * called both early and late (i.e. after bus_add_device). Once all
> -	 * the platform bus code is reworked to call us late and the notifier
> -	 * junk above goes away, move the body of do_iommu_attach here.

...per this commment. It has no need to be a separate function once this
is the only call site (plus it has a misleadingly inaccurate name anyway).

> -	 */
> +
>  	group = iommu_group_get(dev);
> +
>  	if (group) {
>  		do_iommu_attach(dev, ops, dma_base, size);
>  		iommu_group_put(group);
> -	} else {
> -		queue_iommu_attach(dev, ops, dma_base, size);
>  	}
>  }

I overlooked it in the last patch as the relevant context was in the
cover letter, so I'll just add the comment here; the domain check and
WARN_ON() in arch_teardown_dma_ops() is mostly a left-over from the
'fake default domain' bodge that was already gone from the final version
of the arm64 dma_ops series. It's based on the assumption that
arch_teardown_dma_ops() is called after the device is removed from the
bus - I'm not convinced that assumption was ever actually true, it now
doesn't necessarily do anything anyway (since detach_device is
deprecated). I'll spin a cleanup patch for that myself...

Robin.

RE: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops

[Sricharan]

Hi Robin,

>On 04/10/16 18:03, Sricharan R wrote:
>> With arch_setup_dma_ops now being called late during device's probe after the
>> device's iommu is probed, the notifier trick required to handle the early
>> setup of dma_ops before the iommu group gets created is not required.
>> So removing the notifier's here.
>
>Hooray!
>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
>>  1 file changed, 3 insertions(+), 97 deletions(-)
>>
>> diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
>> index faf4b92..eb593af 100644
>> --- a/arch/arm64/mm/dma-mapping.c
>> +++ b/arch/arm64/mm/dma-mapping.c
>> @@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
>>  	.mapping_error = iommu_dma_mapping_error,
>>  };
>>
>> -/*
>> - * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
>> - * everything it needs to - the device is only partially created and the
>> - * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
>> - * need this delayed attachment dance. Once IOMMU probe ordering is sorted
>> - * to move the arch_setup_dma_ops() call later, all the notifier bits below
>> - * become unnecessary, and will go away.
>> - */
>> -struct iommu_dma_notifier_data {
>> -	struct list_head list;
>> -	struct device *dev;
>> -	const struct iommu_ops *ops;
>> -	u64 dma_base;
>> -	u64 size;
>> -};
>> -static LIST_HEAD(iommu_dma_masters);
>> -static DEFINE_MUTEX(iommu_dma_notifier_lock);
>> -
>>  static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>>  			   u64 dma_base, u64 size)
>
>This should go as well...

 ok.
>
>>  {
>> @@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>>  	return true;
>>  }
>>
>> -static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
>> -			      u64 dma_base, u64 size)
>> -{
>> -	struct iommu_dma_notifier_data *iommudata;
>> -
>> -	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
>> -	if (!iommudata)
>> -		return;
>> -
>> -	iommudata->dev = dev;
>> -	iommudata->ops = ops;
>> -	iommudata->dma_base = dma_base;
>> -	iommudata->size = size;
>> -
>> -	mutex_lock(&iommu_dma_notifier_lock);
>> -	list_add(&iommudata->list, &iommu_dma_masters);
>> -	mutex_unlock(&iommu_dma_notifier_lock);
>> -}
>> -
>> -static int __iommu_attach_notifier(struct notifier_block *nb,
>> -				   unsigned long action, void *data)
>> -{
>> -	struct iommu_dma_notifier_data *master, *tmp;
>> -
>> -	if (action != BUS_NOTIFY_BIND_DRIVER)
>> -		return 0;
>> -
>> -	mutex_lock(&iommu_dma_notifier_lock);
>> -	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
>> -		if (data == master->dev && do_iommu_attach(master->dev,
>> -				master->ops, master->dma_base, master->size)) {
>> -			list_del(&master->list);
>> -			kfree(master);
>> -			break;
>> -		}
>> -	}
>> -	mutex_unlock(&iommu_dma_notifier_lock);
>> -	return 0;
>> -}
>> -
>> -static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
>> -{
>> -	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
>> -	int ret;
>> -
>> -	if (!nb)
>> -		return -ENOMEM;
>> -
>> -	nb->notifier_call = __iommu_attach_notifier;
>> -
>> -	ret = bus_register_notifier(bus, nb);
>> -	if (ret) {
>> -		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
>> -			bus->name);
>> -		kfree(nb);
>> -	}
>> -	return ret;
>> -}
>> -
>>  static int __init __iommu_dma_init(void)
>>  {
>> -	int ret;
>> -
>> -	ret = iommu_dma_init();
>> -	if (!ret)
>> -		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
>> -	if (!ret)
>> -		ret = register_iommu_dma_ops_notifier(&amba_bustype);
>> -#ifdef CONFIG_PCI
>> -	if (!ret)
>> -		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
>> -#endif
>> -	return ret;
>> +	return iommu_dma_init();
>>  }
>>  arch_initcall(__iommu_dma_init);
>>
>> @@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>>
>>  	if (!ops)
>>  		return;
>> -	/*
>> -	 * TODO: As a concession to the future, we're ready to handle being
>> -	 * called both early and late (i.e. after bus_add_device). Once all
>> -	 * the platform bus code is reworked to call us late and the notifier
>> -	 * junk above goes away, move the body of do_iommu_attach here.
>
>...per this commment. It has no need to be a separate function once this
>is the only call site (plus it has a misleadingly inaccurate name anyway).

 ya, missed to remove that function as well. Will do it in the next.

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Robin Murphy]

Hi Sricharan,

On 04/10/16 18:03, Sricharan R wrote:
> Initial post from Laurent Pinchart[1]. This is
> series calls the dma ops configuration for the devices
> at a generic place so that it works for all busses.
> The dma_configure_ops for a device is now called during
> the device_attach callback just before the probe of the
> bus/driver is called. Similarly dma_deconfigure is called during
> device/driver_detach path.
> 
> 
> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>        |                         |
> pci_bus_add_device     (device_add/driver_register)
>        |                         |
> device_attach           device_initial_probe
>        |                         |
> __device_attach_driver    __device_attach_driver
>        |
> driver_probe_device
>        |
> really_probe
>        |
> dma_configure
> 
>  Similarly on the device/driver_unregister path __device_release_driver is
>  called which inturn calls dma_deconfigure.
> 
>  If the ACPI bus code follows the same, we can add acpi_dma_configure
>  at the same place as of_dma_configure.
> 
>  This series is based on the recently merged Generic DT bindings for
>  PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy@arm.com [2]
> 
>  This time tested this with platform and pci device for probe deferral
>  and reprobe on arm64 based platform. There is an issue on the cleanup
>  path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>  device is attached to an domain in arch_teardown_dma_ops.
>  But with iommu_groups created from the iommu driver, the device is always
>  attached to a domain/default_domain. So so the WARN has to be removed/handled
>  probably.

I've finally got the chance to take a proper look at this series (sorry
for the delay). First up, with these patches on top of 4.9-rc2, my
little script for unbinding some PCI devices and rebinding them to VFIO
now goes horribly, horribly wrong.

Robin.

[   39.901592] iommu: Removing device 0000:08:00.0 from group 0
[   39.907383] ------------[ cut here ]------------
[   39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[   39.921266] Modules linked in:
[   39.924290]
[   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[   39.931967] Hardware name: ARM Juno development board (r1) (DT)
[   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[   39.960834] sp : ffffffc974d63ca0
[   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[   40.011492] x11: 0000000000000018 x10: 000000000000000d
[   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[   40.043078]
[   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[   40.049117] Call trace:
[   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[   40.057914] 3ac0:                                   ffffffc9763f20a0
0000008000000000
[   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[   40.228075] ------------[ cut here ]------------
[   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[   40.243181] Modules linked in:
[   40.246201]
[   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.255076] Hardware name: ARM Juno development board (r1) (DT)
[   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.266787] PC is at kobject_get+0x64/0x88
[   40.270840] LR is at iommu_bus_notifier+0x40/0x110
[   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[   40.282894] sp : ffffffc974d63c00
[   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[   40.328255] x13: 0000000000000004 x12: 0000000000000030
[   40.333515] x11: 0000000000000030 x10: 0101010101010101
[   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[   40.365074]
[   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[   40.371107] Call trace:
[   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[   40.379895] 3a20:                                   ffffffc9763ebc80
0000008000000000
[   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   40.541324] ata1.00: disabled
[   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.558871] sd 0:0:0:0: [sda] Stopping disk
[   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
hostbyte=0x04 driverbyte=0x00
[   40.586990] Unable to handle kernel paging request at virtual address
0002003e
[   40.594702] pgd = ffffffc974c80000
[   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[   40.607694]
[   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[   40.614684] Modules linked in:
[   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
4.9.0-rc2+ #1249
[   40.625118] Hardware name: ARM Juno development board (r1) (DT)
[   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[   40.636841] PC is at kobject_get+0x14/0x88
[   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[   40.653387] sp : ffffffc974d63c60
[   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[   40.677720] x21: 000000000000000d x20: ffffff8008dab568
[   40.682984] x19: 0000000000020002 x18: 0000000000000000
[   40.688246] x17: 0000000000000007 x16: 0000000000000001
[   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[   40.735613]
[   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[   40.973986] Call trace:
[   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[   40.982780] 3a80:                                   0000000000020002
0000008000000000
[   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[   41.148667] ---[ end trace 35c1e743d6e6c037 ]---
Segmentation fault
/ #

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>On 04/10/16 18:03, Sricharan R wrote:
>> Initial post from Laurent Pinchart[1]. This is
>> series calls the dma ops configuration for the devices
>> at a generic place so that it works for all busses.
>> The dma_configure_ops for a device is now called during
>> the device_attach callback just before the probe of the
>> bus/driver is called. Similarly dma_deconfigure is called during
>> device/driver_detach path.
>>
>>
>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>        |                         |
>> pci_bus_add_device     (device_add/driver_register)
>>        |                         |
>> device_attach           device_initial_probe
>>        |                         |
>> __device_attach_driver    __device_attach_driver
>>        |
>> driver_probe_device
>>        |
>> really_probe
>>        |
>> dma_configure
>>
>>  Similarly on the device/driver_unregister path __device_release_driver is
>>  called which inturn calls dma_deconfigure.
>>
>>  If the ACPI bus code follows the same, we can add acpi_dma_configure
>>  at the same place as of_dma_configure.
>>
>>  This series is based on the recently merged Generic DT bindings for
>>  PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>
>>  This time tested this with platform and pci device for probe deferral
>>  and reprobe on arm64 based platform. There is an issue on the cleanup
>>  path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>  device is attached to an domain in arch_teardown_dma_ops.
>>  But with iommu_groups created from the iommu driver, the device is always
>>  attached to a domain/default_domain. So so the WARN has to be removed/handled
>>  probably.
>
>I've finally got the chance to take a proper look at this series (sorry
>for the delay). First up, with these patches on top of 4.9-rc2, my
>little script for unbinding some PCI devices and rebinding them to VFIO
>now goes horribly, horribly wrong.
>
>Robin.
>

   Thanks for looking in to this.
    I was trying to reproduce the below with a command like this in my setup.

echo 0002\:00\:00.0 >  /sys/bus/pci/devices/0002\:00\:000.0/driver/unbind0.0/driver/unbind
echo 0x17cb 0x0104 > /sys/bus/pci/drivers/vfio-pci/new_id

But for me the unbind and reconfiguring/adding the iommus_ops to vfio-pci did
succeed, although the WARN_ON in arch_teardown_dma_ops was there, that
could be suppresed.  The vfio_pci_probe was not going through because
 the pci hdr_type was not PCI_HEADER_TYPE_NORMAL.
 But anyways iommu unbind/rebind seemed to be going through.

If i can get what your script is doing, i can try that and see what happens.

Regards,
  Sricharan



>[   39.901592] iommu: Removing device 0000:08:00.0 from group 0
>[   39.907383] ------------[ cut here ]------------
>[   39.911969] WARNING: CPU: 0 PID: 174 at
>arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
>[   39.921266] Modules linked in:
>[   39.924290]
>[   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
>[   39.931967] Hardware name: ARM Juno development board (r1) (DT)
>[   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>[   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
>[   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
>[   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
>pstate: 60000145
>[   39.960834] sp : ffffffc974d63ca0
>[   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
>[   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
>[   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
>[   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
>[   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
>[   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
>[   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
>[   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
>[   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
>[   40.011492] x11: 0000000000000018 x10: 000000000000000d
>[   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
>[   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
>[   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
>[   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
>[   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
>[   40.043078]
>[   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
>[   40.049117] Call trace:
>[   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
>[   40.057914] 3ac0:                                   ffffffc9763f20a0
>0000008000000000
>[   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
>ffffffc975ea9800
>[   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
>ffffffc974d60000
>[   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
>ffffffc974d60000
>[   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
>ffffffc974d60000
>[   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
>0000000000000000
>[   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
>ffffff80084c4208
>[   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
>0000000040000000
>[   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
>ffffffffff000000
>[   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
>0000007f99c18018
>[   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
>[   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>[   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>[   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>[   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>[   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
>[   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>[   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>[   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>[   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>[   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>[   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>[   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>[   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
>[   40.228075] ------------[ cut here ]------------
>[   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
>kobject_get+0x64/0x88
>[   40.243181] Modules linked in:
>[   40.246201]
>[   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
>4.9.0-rc2+ #1249
>[   40.255076] Hardware name: ARM Juno development board (r1) (DT)
>[   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>[   40.266787] PC is at kobject_get+0x64/0x88
>[   40.270840] LR is at iommu_bus_notifier+0x40/0x110
>[   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
>pstate: 80000145
>[   40.282894] sp : ffffffc974d63c00
>[   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
>[   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
>[   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
>[   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
>[   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
>[   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
>[   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
>[   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
>[   40.328255] x13: 0000000000000004 x12: 0000000000000030
>[   40.333515] x11: 0000000000000030 x10: 0101010101010101
>[   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
>[   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
>[   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
>[   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
>[   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
>[   40.365074]
>[   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
>[   40.371107] Call trace:
>[   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
>[   40.379895] 3a20:                                   ffffffc9763ebc80
>0000008000000000
>[   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
>ffffff8008237d94
>[   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
>0000000000005b81
>[   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
>ffffffc976811c80
>[   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
>ffffff8008237000
>[   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
>0000000000000001
>[   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
>ffffffc976abf400
>[   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
>feff716475687163
>[   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
>0000000000000004
>[   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
>0000007f8d0ae3c0
>[   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
>[   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
>[   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
>[   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
>[   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
>[   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
>[   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>[   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
>[   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>[   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>[   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>[   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>[   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>[   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>[   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>[   40.541324] ata1.00: disabled
>[   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
>[   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
>hostbyte=0x04 driverbyte=0x00
>[   40.558871] sd 0:0:0:0: [sda] Stopping disk
>[   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
>hostbyte=0x04 driverbyte=0x00
>[   40.586990] Unable to handle kernel paging request at virtual address
>0002003e
>[   40.594702] pgd = ffffffc974c80000
>[   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
>*pud=00000009f5102003
>, *pmd=0000000000000000[   40.607694]
>[   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
>[   40.614684] Modules linked in:
>[   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
>4.9.0-rc2+ #1249
>[   40.625118] Hardware name: ARM Juno development board (r1) (DT)
>[   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>[   40.636841] PC is at kobject_get+0x14/0x88
>[   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
>[   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
>pstate: 60000145
>[   40.653387] sp : ffffffc974d63c60
>[   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
>[   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
>[   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
>[   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
>[   40.677720] x21: 000000000000000d x20: ffffff8008dab568
>[   40.682984] x19: 0000000000020002 x18: 0000000000000000
>[   40.688246] x17: 0000000000000007 x16: 0000000000000001
>[   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
>[   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
>[   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
>[   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
>[   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
>[   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
>[   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
>[   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
>[   40.735613]
>[   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
>[   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
>[   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
>ffffff80085377a4
>[   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
>ffffff8008dab568
>[   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
>ffffff8008dab568
>[   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
>ffffff800853ba10
>[   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
>ffffffc9763f10a0
>[   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
>ffffffc9763f10a0
>[   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
>ffffffc974c8f400
>[   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
>ffffff800823ab18
>[   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
>ffffff80081c507c
>[   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
>ffffffc974d63eb8
>[   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
>ffffff80080bf6ac
>[   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
>0000000000000000
>[   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
>ffffff80081c5c0c
>[   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
>ffffffc974d63eb8
>[   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
>ffffffc974c8f100
>[   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
>ffffffc974c8f100
>[   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
>0000000000000000
>[   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
>0000000000000001
>[   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
>0000000000000000
>[   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
>0000000000000000
>[   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
>39322f392f2f2f2f
>[   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
>0101010101010101
>[   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
>0000007f8d136588
>[   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
>00000000004ee000
>[   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
>0000000000000001
>[   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
>0000000000000000
>[   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
>0000007fd6507130
>[   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
>0000000000000040
>[   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
>4000000010000000
>[   40.973986] Call trace:
>[   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
>[   40.982780] 3a80:                                   0000000000020002
>0000008000000000
>[   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
>ffffff80085377a4
>[   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
>ffffffc975e3ac38
>[   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
>ffffff8008da62c0
>[   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
>ffffff8008540a08
>[   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
>0000000000000000
>[   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
>0000000000000000
>[   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
>0000000040000000
>[   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
>ffffffc974cd01cd
>[   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
>0000000000000007
>[   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
>[   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
>[   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
>[   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>[   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>[   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>[   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>[   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
>[   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>[   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>[   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>[   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>[   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>[   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>[   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>[   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
>[   41.148667] ---[ end trace 35c1e743d6e6c037 ]---
>Segmentation fault
>/ #
>
>
>_______________________________________________
>linux-arm-kernel mailing list
>linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
>http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Robin Murphy]

On 26/10/16 15:44, Sricharan wrote:
> Hi Robin,
> 
>> On 04/10/16 18:03, Sricharan R wrote:
>>> Initial post from Laurent Pinchart[1]. This is
>>> series calls the dma ops configuration for the devices
>>> at a generic place so that it works for all busses.
>>> The dma_configure_ops for a device is now called during
>>> the device_attach callback just before the probe of the
>>> bus/driver is called. Similarly dma_deconfigure is called during
>>> device/driver_detach path.
>>>
>>>
>>> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
>>>        |                         |
>>> pci_bus_add_device     (device_add/driver_register)
>>>        |                         |
>>> device_attach           device_initial_probe
>>>        |                         |
>>> __device_attach_driver    __device_attach_driver
>>>        |
>>> driver_probe_device
>>>        |
>>> really_probe
>>>        |
>>> dma_configure
>>>
>>>  Similarly on the device/driver_unregister path __device_release_driver is
>>>  called which inturn calls dma_deconfigure.
>>>
>>>  If the ACPI bus code follows the same, we can add acpi_dma_configure
>>>  at the same place as of_dma_configure.
>>>
>>>  This series is based on the recently merged Generic DT bindings for
>>>  PCI IOMMUs and ARM SMMU from Robin Murphy robin.murphy-5wv7dgnIgG8@public.gmane.org [2]
>>>
>>>  This time tested this with platform and pci device for probe deferral
>>>  and reprobe on arm64 based platform. There is an issue on the cleanup
>>>  path for arm64 though, where there is WARN_ON if the dma_ops is reset while
>>>  device is attached to an domain in arch_teardown_dma_ops.
>>>  But with iommu_groups created from the iommu driver, the device is always
>>>  attached to a domain/default_domain. So so the WARN has to be removed/handled
>>>  probably.
>>
>> I've finally got the chance to take a proper look at this series (sorry
>> for the delay). First up, with these patches on top of 4.9-rc2, my
>> little script for unbinding some PCI devices and rebinding them to VFIO
>> now goes horribly, horribly wrong.
>>
>> Robin.
>>
> 
>    Thanks for looking in to this.
>     I was trying to reproduce the below with a command like this in my setup.
> 
> echo 0002\:00\:00.0 >  /sys/bus/pci/devices/0002\:00\:000.0/driver/unbind0.0/driver/unbind
> echo 0x17cb 0x0104 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> But for me the unbind and reconfiguring/adding the iommus_ops to vfio-pci did
> succeed, although the WARN_ON in arch_teardown_dma_ops was there, that

Oh, yes, I hacked that out already to cut the noise down.

> could be suppresed.  The vfio_pci_probe was not going through because
>  the pci hdr_type was not PCI_HEADER_TYPE_NORMAL.
>  But anyways iommu unbind/rebind seemed to be going through.
> 
> If i can get what your script is doing, i can try that and see what happens.

---8<---
#!/bin/sh

#Juno Sky2, SATA
DEVICES='0000:08:00.0 0000:03:00.0'

for DEV in $DEVICES
do
	BUSDEV=/sys/bus/pci/devices/$DEV
	GROUP=$(basename $(readlink $BUSDEV/iommu_group))
	DRV=$(readlink -f $BUSDEV/driver)
	read VID < $BUSDEV/vendor
	read DID < $BUSDEV/device

	echo $DEV > $BUSDEV/driver/unbind
	echo $VID $DID > /sys/bus/pci/drivers/vfio-pci/new_id
done
# it would then goes on to launch kvmtool and rebind the original
# drivers afterwards, but that doesn't matter here
--->8---

The segfault doesn't always happen, but the kref warnings and the
vfio-pci driver failing to probe certainly do.

> 
> Regards,
>   Sricharan
> 
> 
> 
>> [   39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

Robin.

>> [   39.907383] ------------[ cut here ]------------
>> [   39.911969] WARNING: CPU: 0 PID: 174 at
>> arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
>> [   39.921266] Modules linked in:
>> [   39.924290]
>> [   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
>> [   39.931967] Hardware name: ARM Juno development board (r1) (DT)
>> [   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>> [   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
>> [   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
>> [   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
>> pstate: 60000145
>> [   39.960834] sp : ffffffc974d63ca0
>> [   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
>> [   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
>> [   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
>> [   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
>> [   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
>> [   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
>> [   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
>> [   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
>> [   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
>> [   40.011492] x11: 0000000000000018 x10: 000000000000000d
>> [   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
>> [   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
>> [   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
>> [   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
>> [   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
>> [   40.043078]
>> [   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
>> [   40.049117] Call trace:
>> [   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
>> [   40.057914] 3ac0:                                   ffffffc9763f20a0
>> 0000008000000000
>> [   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
>> ffffffc975ea9800
>> [   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
>> ffffffc974d60000
>> [   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
>> ffffffc974d60000
>> [   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
>> ffffffc974d60000
>> [   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
>> 0000000000000000
>> [   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
>> ffffff80084c4208
>> [   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
>> 0000000040000000
>> [   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
>> ffffffffff000000
>> [   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
>> 0000007f99c18018
>> [   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
>> [   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>> [   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>> [   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>> [   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>> [   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
>> [   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>> [   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>> [   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>> [   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>> [   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>> [   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>> [   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>> [   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
>> [   40.228075] ------------[ cut here ]------------
>> [   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
>> kobject_get+0x64/0x88
>> [   40.243181] Modules linked in:
>> [   40.246201]
>> [   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
>> 4.9.0-rc2+ #1249
>> [   40.255076] Hardware name: ARM Juno development board (r1) (DT)
>> [   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>> [   40.266787] PC is at kobject_get+0x64/0x88
>> [   40.270840] LR is at iommu_bus_notifier+0x40/0x110
>> [   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
>> pstate: 80000145
>> [   40.282894] sp : ffffffc974d63c00
>> [   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
>> [   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
>> [   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
>> [   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
>> [   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
>> [   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
>> [   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
>> [   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
>> [   40.328255] x13: 0000000000000004 x12: 0000000000000030
>> [   40.333515] x11: 0000000000000030 x10: 0101010101010101
>> [   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
>> [   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
>> [   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
>> [   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
>> [   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
>> [   40.365074]
>> [   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
>> [   40.371107] Call trace:
>> [   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
>> [   40.379895] 3a20:                                   ffffffc9763ebc80
>> 0000008000000000
>> [   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
>> ffffff8008237d94
>> [   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
>> 0000000000005b81
>> [   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
>> ffffffc976811c80
>> [   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
>> ffffff8008237000
>> [   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
>> 0000000000000001
>> [   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
>> ffffffc976abf400
>> [   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
>> feff716475687163
>> [   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
>> 0000000000000004
>> [   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
>> 0000007f8d0ae3c0
>> [   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
>> [   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
>> [   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
>> [   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
>> [   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
>> [   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
>> [   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>> [   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
>> [   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>> [   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>> [   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>> [   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>> [   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>> [   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>> [   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>> [   40.541324] ata1.00: disabled
>> [   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
>> [   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
>> hostbyte=0x04 driverbyte=0x00
>> [   40.558871] sd 0:0:0:0: [sda] Stopping disk
>> [   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
>> hostbyte=0x04 driverbyte=0x00
>> [   40.586990] Unable to handle kernel paging request at virtual address
>> 0002003e
>> [   40.594702] pgd = ffffffc974c80000
>> [   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
>> *pud=00000009f5102003
>> , *pmd=0000000000000000[   40.607694]
>> [   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
>> [   40.614684] Modules linked in:
>> [   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
>> 4.9.0-rc2+ #1249
>> [   40.625118] Hardware name: ARM Juno development board (r1) (DT)
>> [   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>> [   40.636841] PC is at kobject_get+0x14/0x88
>> [   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
>> [   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
>> pstate: 60000145
>> [   40.653387] sp : ffffffc974d63c60
>> [   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
>> [   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
>> [   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
>> [   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
>> [   40.677720] x21: 000000000000000d x20: ffffff8008dab568
>> [   40.682984] x19: 0000000000020002 x18: 0000000000000000
>> [   40.688246] x17: 0000000000000007 x16: 0000000000000001
>> [   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
>> [   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
>> [   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
>> [   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
>> [   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
>> [   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
>> [   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
>> [   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
>> [   40.735613]
>> [   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
>> [   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
>> [   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
>> ffffff80085377a4
>> [   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
>> ffffff8008dab568
>> [   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
>> ffffff8008dab568
>> [   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
>> ffffff800853ba10
>> [   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
>> ffffffc9763f10a0
>> [   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
>> ffffffc9763f10a0
>> [   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
>> ffffffc974c8f400
>> [   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
>> ffffff800823ab18
>> [   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
>> ffffff80081c507c
>> [   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
>> ffffffc974d63eb8
>> [   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
>> ffffff80080bf6ac
>> [   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
>> 0000000000000000
>> [   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
>> ffffff80081c5c0c
>> [   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
>> ffffffc974d63eb8
>> [   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
>> ffffffc974c8f100
>> [   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
>> ffffffc974c8f100
>> [   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
>> 0000000000000000
>> [   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
>> 0000000000000001
>> [   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
>> 0000000000000000
>> [   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
>> 0000000000000000
>> [   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
>> 39322f392f2f2f2f
>> [   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
>> 0101010101010101
>> [   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
>> 0000007f8d136588
>> [   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
>> 00000000004ee000
>> [   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
>> 0000000000000001
>> [   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
>> 0000000000000000
>> [   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
>> 0000007fd6507130
>> [   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
>> 0000000000000040
>> [   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
>> 4000000010000000
>> [   40.973986] Call trace:
>> [   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
>> [   40.982780] 3a80:                                   0000000000020002
>> 0000008000000000
>> [   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
>> ffffff80085377a4
>> [   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
>> ffffffc975e3ac38
>> [   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
>> ffffff8008da62c0
>> [   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
>> ffffff8008540a08
>> [   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
>> 0000000000000000
>> [   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
>> 0000000000000000
>> [   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
>> 0000000040000000
>> [   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
>> ffffffc974cd01cd
>> [   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
>> 0000000000000007
>> [   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
>> [   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
>> [   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
>> [   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>> [   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>> [   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>> [   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>> [   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
>> [   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>> [   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>> [   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>> [   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>> [   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>> [   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>> [   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>> [   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
>> [   41.148667] ---[ end trace 35c1e743d6e6c037 ]---
>> Segmentation fault
>> / #
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>>
>>
>>
>>> [   39.901592] iommu: Removing device 0000:08:00.0 from group 0
>
>Yikes, on second look, that definitely shouldn't be happening.
>Everything below is probably the resulting fallout.

[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops

I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.


Regards,
  Sricharan

    
>
>Robin.
>
>>> [   39.907383] ------------[ cut here ]------------
>>> [   39.911969] WARNING: CPU: 0 PID: 174 at
>>> arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
>>> [   39.921266] Modules linked in:
>>> [   39.924290]
>>> [   39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
>>> [   39.931967] Hardware name: ARM Juno development board (r1) (DT)
>>> [   39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>>> [   39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
>>> [   39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
>>> [   39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
>>> pstate: 60000145
>>> [   39.960834] sp : ffffffc974d63ca0
>>> [   39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
>>> [   39.969377] x27: ffffff80088a2000 x26: 0000000000000040
>>> [   39.974642] x25: 0000000000000123 x24: ffffffc976a48918
>>> [   39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
>>> [   39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
>>> [   39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
>>> [   39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
>>> [   40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
>>> [   40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
>>> [   40.011492] x11: 0000000000000018 x10: 000000000000000d
>>> [   40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
>>> [   40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
>>> [   40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
>>> [   40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
>>> [   40.037815] x1 : 0000000000000000 x0 : 0000000000000080
>>> [   40.043078]
>>> [   40.044549] ---[ end trace 35c1e743d6e6c035 ]---
>>> [   40.049117] Call trace:
>>> [   40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
>>> [   40.057914] 3ac0:                                   ffffffc9763f20a0
>>> 0000008000000000
>>> [   40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
>>> ffffffc975ea9800
>>> [   40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
>>> ffffffc974d60000
>>> [   40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
>>> ffffffc974d60000
>>> [   40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
>>> ffffffc974d60000
>>> [   40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
>>> 0000000000000000
>>> [   40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
>>> ffffff80084c4208
>>> [   40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
>>> 0000000040000000
>>> [   40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
>>> ffffffffff000000
>>> [   40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
>>> 0000007f99c18018
>>> [   40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
>>> [   40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>>> [   40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>>> [   40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>>> [   40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>>> [   40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
>>> [   40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>>> [   40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>>> [   40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>>> [   40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>>> [   40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>>> [   40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>>> [   40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>> [   40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
>>> [   40.228075] ------------[ cut here ]------------
>>> [   40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
>>> kobject_get+0x64/0x88
>>> [   40.243181] Modules linked in:
>>> [   40.246201]
>>> [   40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G        W
>>> 4.9.0-rc2+ #1249
>>> [   40.255076] Hardware name: ARM Juno development board (r1) (DT)
>>> [   40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>>> [   40.266787] PC is at kobject_get+0x64/0x88
>>> [   40.270840] LR is at iommu_bus_notifier+0x40/0x110
>>> [   40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
>>> pstate: 80000145
>>> [   40.282894] sp : ffffffc974d63c00
>>> [   40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
>>> [   40.291431] x27: ffffff80088a2000 x26: 0000000000000040
>>> [   40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
>>> [   40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
>>> [   40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
>>> [   40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
>>> [   40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
>>> [   40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
>>> [   40.328255] x13: 0000000000000004 x12: 0000000000000030
>>> [   40.333515] x11: 0000000000000030 x10: 0101010101010101
>>> [   40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
>>> [   40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
>>> [   40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
>>> [   40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
>>> [   40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
>>> [   40.365074]
>>> [   40.366542] ---[ end trace 35c1e743d6e6c036 ]---
>>> [   40.371107] Call trace:
>>> [   40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
>>> [   40.379895] 3a20:                                   ffffffc9763ebc80
>>> 0000008000000000
>>> [   40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
>>> ffffff8008237d94
>>> [   40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
>>> 0000000000005b81
>>> [   40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
>>> ffffffc976811c80
>>> [   40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
>>> ffffff8008237000
>>> [   40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
>>> 0000000000000001
>>> [   40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
>>> ffffffc976abf400
>>> [   40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
>>> feff716475687163
>>> [   40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
>>> 0000000000000004
>>> [   40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
>>> 0000007f8d0ae3c0
>>> [   40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
>>> [   40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
>>> [   40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
>>> [   40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
>>> [   40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
>>> [   40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
>>> [   40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>>> [   40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
>>> [   40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>>> [   40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>>> [   40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>>> [   40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>>> [   40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>>> [   40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>>> [   40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>>> [   40.541324] ata1.00: disabled
>>> [   40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
>>> [   40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
>>> hostbyte=0x04 driverbyte=0x00
>>> [   40.558871] sd 0:0:0:0: [sda] Stopping disk
>>> [   40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
>>> hostbyte=0x04 driverbyte=0x00
>>> [   40.586990] Unable to handle kernel paging request at virtual address
>>> 0002003e
>>> [   40.594702] pgd = ffffffc974c80000
>>> [   40.598165] [0002003e] *pgd=00000009f5102003[   40.602241] ,
>>> *pud=00000009f5102003
>>> , *pmd=0000000000000000[   40.607694]
>>> [   40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
>>> [   40.614684] Modules linked in:
>>> [   40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G        W
>>> 4.9.0-rc2+ #1249
>>> [   40.625118] Hardware name: ARM Juno development board (r1) (DT)
>>> [   40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
>>> [   40.636841] PC is at kobject_get+0x14/0x88
>>> [   40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
>>> [   40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
>>> pstate: 60000145
>>> [   40.653387] sp : ffffffc974d63c60
>>> [   40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
>>> [   40.661928] x27: ffffff80088a2000 x26: 0000000000000040
>>> [   40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
>>> [   40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
>>> [   40.677720] x21: 000000000000000d x20: ffffff8008dab568
>>> [   40.682984] x19: 0000000000020002 x18: 0000000000000000
>>> [   40.688246] x17: 0000000000000007 x16: 0000000000000001
>>> [   40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
>>> [   40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
>>> [   40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
>>> [   40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
>>> [   40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
>>> [   40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
>>> [   40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
>>> [   40.730351] x1 : 0000000000000000 x0 : 0000000000020002
>>> [   40.735613]
>>> [   40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
>>> [   40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
>>> [   40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
>>> ffffff80085377a4
>>> [   40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
>>> ffffff8008dab568
>>> [   40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
>>> ffffff8008dab568
>>> [   40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
>>> ffffff800853ba10
>>> [   40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
>>> ffffffc9763f10a0
>>> [   40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
>>> ffffffc9763f10a0
>>> [   40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
>>> ffffffc974c8f400
>>> [   40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
>>> ffffff800823ab18
>>> [   40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
>>> ffffff80081c507c
>>> [   40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
>>> ffffffc974d63eb8
>>> [   40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
>>> ffffff80080bf6ac
>>> [   40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
>>> 0000000000000000
>>> [   40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
>>> ffffff80081c5c0c
>>> [   40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
>>> ffffffc974d63eb8
>>> [   40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
>>> ffffffc974c8f100
>>> [   40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
>>> ffffffc974c8f100
>>> [   40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
>>> 0000000000000000
>>> [   40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
>>> 0000000000000001
>>> [   40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
>>> 0000000000000000
>>> [   40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
>>> 0000000000000000
>>> [   40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
>>> 39322f392f2f2f2f
>>> [   40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
>>> 0101010101010101
>>> [   40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
>>> 0000007f8d136588
>>> [   40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
>>> 00000000004ee000
>>> [   40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
>>> 0000000000000001
>>> [   40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
>>> 0000000000000000
>>> [   40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
>>> 0000007fd6507130
>>> [   40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
>>> 0000000000000040
>>> [   40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
>>> 4000000010000000
>>> [   40.973986] Call trace:
>>> [   40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
>>> [   40.982780] 3a80:                                   0000000000020002
>>> 0000008000000000
>>> [   40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
>>> ffffff80085377a4
>>> [   40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
>>> ffffffc975e3ac38
>>> [   41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
>>> ffffff8008da62c0
>>> [   41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
>>> ffffff8008540a08
>>> [   41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
>>> 0000000000000000
>>> [   41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
>>> 0000000000000000
>>> [   41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
>>> 0000000040000000
>>> [   41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
>>> ffffffc974cd01cd
>>> [   41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
>>> 0000000000000007
>>> [   41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
>>> [   41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
>>> [   41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
>>> [   41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
>>> [   41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
>>> [   41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
>>> [   41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
>>> [   41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
>>> [   41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
>>> [   41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
>>> [   41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
>>> [   41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
>>> [   41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
>>> [   41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
>>> [   41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
>>> [   41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
>>> [   41.148667] ---[ end trace 35c1e743d6e6c037 ]---
>>> Segmentation fault
>>> / #
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>
>
>_______________________________________________
>linux-arm-kernel mailing list
>linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
>http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>>>> [   39.901592] iommu: Removing device 0000:08:00.0 from group 0
>>
>>Yikes, on second look, that definitely shouldn't be happening.
>>Everything below is probably the resulting fallout.
>
>[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>
>I think the above print which says "failed to setup iommu_ops"
>because the call ops->add_device failed in of_pci_iommu_configure
>is the reason for the failure, in my case i simply do not get this even with
>your scripts. ops->add_device succeeds in the rebind as well. So still
>checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.


[1] git://linux-arm.org/linux-rm iommu/misc
>
>
>Regards,
>  Sricharan
>

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>>>Yikes, on second look, that definitely shouldn't be happening.
>>>Everything below is probably the resulting fallout.
>>
>>[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>
>>I think the above print which says "failed to setup iommu_ops"
>>because the call ops->add_device failed in of_pci_iommu_configure
>>is the reason for the failure, in my case i simply do not get this even with
>>your scripts. ops->add_device succeeds in the rebind as well. So still
>>checking what could be happening in your case.
>
>I was looking at your code base from [1].The ops->add_device
>callback from of_pci_iommu_configure on the rebind is the
>one which is causing the failure. But not able to spot out
>from code which point is causing the failure. It would be very helpful
>if i can know which is the return value from the add_device callback
>or point inside add_device callback which fails in your setup.
>
>
>[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?

From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting

iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
---
 drivers/iommu/arm-smmu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
 		group = smmu->s2crs[idx].group;
 	}
 
-	if (group)
+	if (group) {
+		iommu_group_get_by_id(iommu_group_id(group));
 		return group;
+	}
 
 	if (dev_is_pci(dev))
 		group = pci_device_group(dev);
-- 
1.8.2.1

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Will Deacon]

On Fri, Nov 04, 2016 at 08:46:06PM +0530, Sricharan wrote:
> >>>Yikes, on second look, that definitely shouldn't be happening.
> >>>Everything below is probably the resulting fallout.
> >>
> >>[   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
> >>
> >>I think the above print which says "failed to setup iommu_ops"
> >>because the call ops->add_device failed in of_pci_iommu_configure
> >>is the reason for the failure, in my case i simply do not get this even with
> >>your scripts. ops->add_device succeeds in the rebind as well. So still
> >>checking what could be happening in your case.
> >
> >I was looking at your code base from [1].The ops->add_device
> >callback from of_pci_iommu_configure on the rebind is the
> >one which is causing the failure. But not able to spot out
> >from code which point is causing the failure. It would be very helpful
> >if i can know which is the return value from the add_device callback
> >or point inside add_device callback which fails in your setup.
> >
> >
> >[1] git://linux-arm.org/linux-rm iommu/misc

So this also applies to mainline.

> With little more try, i saw an issue where i had an failure
> similar to what you reported. The issue happens when multiple
> devices fall in to same group due to matching sids. I ended up
> doing a fix like below and it would be nice to verify if it is the same
> that we are seeing in your setup and if the fix makes a difference ?
> 
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Fri, 4 Nov 2016 20:28:49 +0530
> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
> 
> iommu_group_get_for_dev which gets called in the add_device
> callback, increases the reference count of the iommu_group,
> so we do an iommu_group_put after that. iommu_group_get_for_dev
> inturn calls device_group callback and in the case of arm-smmu
> we call generic_device_group/pci_device_group which takes
> care of increasing the group's reference. But when we return
> an already existing group(when multiple devices have same group)
> the reference is not incremented, resulting in issues when the
> remove_device callback for the devices is invoked.
> Fixing the same here.
> 
> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/arm-smmu.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 71ce4b6..a1d0b3c 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
>  		group = smmu->s2crs[idx].group;
>  	}
>  
> -	if (group)
> +	if (group) {
> +		iommu_group_get_by_id(iommu_group_id(group));
>  		return group;
> +	}

This is foul :(

I think we should either add a function for incrementing the refcount on a
group, or we should get a handle on the existing aliasing device and get
the group from that. As written, this patch is far too subtle.

Joerg -- do you have any preference?

Will

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Robin Murphy]

Hi Sricharan,

On 04/11/16 15:16, Sricharan wrote:
> Hi Robin,
> 
>>>> Yikes, on second look, that definitely shouldn't be happening.
>>>> Everything below is probably the resulting fallout.
>>>
>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>>
>>> I think the above print which says "failed to setup iommu_ops"
>>> because the call ops->add_device failed in of_pci_iommu_configure
>>> is the reason for the failure, in my case i simply do not get this even with
>>> your scripts. ops->add_device succeeds in the rebind as well. So still
>>> checking what could be happening in your case.
>>
>> I was looking at your code base from [1].The ops->add_device
>> callback from of_pci_iommu_configure on the rebind is the
>> one which is causing the failure. But not able to spot out
>>from code which point is causing the failure. It would be very helpful
>> if i can know which is the return value from the add_device callback
>> or point inside add_device callback which fails in your setup.
>>
>>
>> [1] git://linux-arm.org/linux-rm iommu/misc
> 
> With little more try, i saw an issue where i had an failure
> similar to what you reported. The issue happens when multiple
> devices fall in to same group due to matching sids. I ended up
> doing a fix like below and it would be nice to verify if it is the same
> that we are seeing in your setup and if the fix makes a difference ?
> 
> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> Date: Fri, 4 Nov 2016 20:28:49 +0530
> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
> 
> iommu_group_get_for_dev which gets called in the add_device
> callback, increases the reference count of the iommu_group,
> so we do an iommu_group_put after that. iommu_group_get_for_dev
> inturn calls device_group callback and in the case of arm-smmu
> we call generic_device_group/pci_device_group which takes
> care of increasing the group's reference. But when we return
> an already existing group(when multiple devices have same group)
> the reference is not incremented, resulting in issues when the
> remove_device callback for the devices is invoked.
> Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.

On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

> Signed-off-by: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> ---
>  drivers/iommu/arm-smmu.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 71ce4b6..a1d0b3c 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
>  		group = smmu->s2crs[idx].group;
>  	}
>  
> -	if (group)
> +	if (group) {
> +		iommu_group_get_by_id(iommu_group_id(group));
>  		return group;

This might as well just be inline, i.e.:

		return iommu_group_get_by_id(iommu_group_id(group));

It's a shame we have to go all round the houses when we have the group
right there, but this is probably the most expedient fix. I guess we can
extend the API with some sort of iommu_group_get(group) overload in
future if we really want to.

Robin.

> +	}
>  
>  	if (dev_is_pci(dev))
>  		group = pci_device_group(dev);
> 

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>On 04/11/16 15:16, Sricharan wrote:
>> Hi Robin,
>>
>>>>> Yikes, on second look, that definitely shouldn't be happening.
>>>>> Everything below is probably the resulting fallout.
>>>>
>>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>>>
>>>> I think the above print which says "failed to setup iommu_ops"
>>>> because the call ops->add_device failed in of_pci_iommu_configure
>>>> is the reason for the failure, in my case i simply do not get this even with
>>>> your scripts. ops->add_device succeeds in the rebind as well. So still
>>>> checking what could be happening in your case.
>>>
>>> I was looking at your code base from [1].The ops->add_device
>>> callback from of_pci_iommu_configure on the rebind is the
>>> one which is causing the failure. But not able to spot out
>>>from code which point is causing the failure. It would be very helpful
>>> if i can know which is the return value from the add_device callback
>>> or point inside add_device callback which fails in your setup.
>>>
>>>
>>> [1] git://linux-arm.org/linux-rm iommu/misc
>>
>> With little more try, i saw an issue where i had an failure
>> similar to what you reported. The issue happens when multiple
>> devices fall in to same group due to matching sids. I ended up
>> doing a fix like below and it would be nice to verify if it is the same
>> that we are seeing in your setup and if the fix makes a difference ?
>>
>> From: Sricharan R <sricharan@codeaurora.org>
>> Date: Fri, 4 Nov 2016 20:28:49 +0530
>> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
>>
>> iommu_group_get_for_dev which gets called in the add_device
>> callback, increases the reference count of the iommu_group,
>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>> inturn calls device_group callback and in the case of arm-smmu
>> we call generic_device_group/pci_device_group which takes
>> care of increasing the group's reference. But when we return
>> an already existing group(when multiple devices have same group)
>> the reference is not incremented, resulting in issues when the
>> remove_device callback for the devices is invoked.
>> Fixing the same here.
>
>Bah, yes, this does look like my fault - after flip-flopping between
>about 3 different ways to keep refcounts for the S2CR entries, none of
>which would quite work, I ripped it all out but apparently still got
>things wrong, oh well. Thanks for figuring it out.
 >
>On the probe-deferral angle, whilst it's useful to have uncovered this
>bug, I don't think we should actually be calling remove_device() from
>DMA teardown. I think it's preferable from a user perspective if group
>numbering remains stable, rather than changing depending on the order in
>which they unbind/rebind VFIO drivers. I'm really keen to try and get
>this in shape for 4.10, so I've taken the liberty of hacking up my own
>branch (iommu/defer) based on v3 - would you mind taking a look at the
>two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> ---
>>  drivers/iommu/arm-smmu.c | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 71ce4b6..a1d0b3c 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
>>  		group = smmu->s2crs[idx].group;
>>  	}
>>
>> -	if (group)
>> +	if (group) {
>> +		iommu_group_get_by_id(iommu_group_id(group));
>>  		return group;
>
>This might as well just be inline, i.e.:
>
>		return iommu_group_get_by_id(iommu_group_id(group));
>
>It's a shame we have to go all round the houses when we have the group
>right there, but this is probably the most expedient fix. I guess we can
>extend the API with some sort of iommu_group_get(group) overload in
>future if we really want to.
>

ok, i can send this fix separately then. Otherwise, Will was suggesting on the
other thread that there should probably be a separate API to increment
the group refcount or get the group from the existing aliasing device.
As per me adding the api, looks like another option or post the above ?

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Will Deacon]

On Wed, Nov 09, 2016 at 11:54:20AM +0530, Sricharan wrote:
> >> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> >> index 71ce4b6..a1d0b3c 100644
> >> --- a/drivers/iommu/arm-smmu.c
> >> +++ b/drivers/iommu/arm-smmu.c
> >> @@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
> >>  		group = smmu->s2crs[idx].group;
> >>  	}
> >>
> >> -	if (group)
> >> +	if (group) {
> >> +		iommu_group_get_by_id(iommu_group_id(group));
> >>  		return group;
> >
> >This might as well just be inline, i.e.:
> >
> >		return iommu_group_get_by_id(iommu_group_id(group));
> >
> >It's a shame we have to go all round the houses when we have the group
> >right there, but this is probably the most expedient fix. I guess we can
> >extend the API with some sort of iommu_group_get(group) overload in
> >future if we really want to.
> >
> 
> ok, i can send this fix separately then. Otherwise, Will was suggesting on the
> other thread that there should probably be a separate API to increment
> the group refcount or get the group from the existing aliasing device.
> As per me adding the api, looks like another option or post the above ?

I think adding a new function to the API is the way to go -- having code
like what you propose above littered around the drivers is hard to read and
pretty error-prone, since it looks like a NOP to people who aren't already
thinking about ref counts.

Will

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>Hi Robin,
>
>>On 04/11/16 15:16, Sricharan wrote:
>>> Hi Robin,
>>>
>>>>>> Yikes, on second look, that definitely shouldn't be happening.
>>>>>> Everything below is probably the resulting fallout.
>>>>>
>>>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>>>>
>>>>> I think the above print which says "failed to setup iommu_ops"
>>>>> because the call ops->add_device failed in of_pci_iommu_configure
>>>>> is the reason for the failure, in my case i simply do not get this even with
>>>>> your scripts. ops->add_device succeeds in the rebind as well. So still
>>>>> checking what could be happening in your case.
>>>>
>>>> I was looking at your code base from [1].The ops->add_device
>>>> callback from of_pci_iommu_configure on the rebind is the
>>>> one which is causing the failure. But not able to spot out
>>>>from code which point is causing the failure. It would be very helpful
>>>> if i can know which is the return value from the add_device callback
>>>> or point inside add_device callback which fails in your setup.
>>>>
>>>>
>>>> [1] git://linux-arm.org/linux-rm iommu/misc
>>>
>>> With little more try, i saw an issue where i had an failure
>>> similar to what you reported. The issue happens when multiple
>>> devices fall in to same group due to matching sids. I ended up
>>> doing a fix like below and it would be nice to verify if it is the same
>>> that we are seeing in your setup and if the fix makes a difference ?
>>>
>>> From: Sricharan R <sricharan@codeaurora.org>
>>> Date: Fri, 4 Nov 2016 20:28:49 +0530
>>> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
>>>
>>> iommu_group_get_for_dev which gets called in the add_device
>>> callback, increases the reference count of the iommu_group,
>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>> inturn calls device_group callback and in the case of arm-smmu
>>> we call generic_device_group/pci_device_group which takes
>>> care of increasing the group's reference. But when we return
>>> an already existing group(when multiple devices have same group)
>>> the reference is not incremented, resulting in issues when the
>>> remove_device callback for the devices is invoked.
>>> Fixing the same here.
>>
>>Bah, yes, this does look like my fault - after flip-flopping between
>>about 3 different ways to keep refcounts for the S2CR entries, none of
>>which would quite work, I ripped it all out but apparently still got
>>things wrong, oh well. Thanks for figuring it out.
> >
>>On the probe-deferral angle, whilst it's useful to have uncovered this
>>bug, I don't think we should actually be calling remove_device() from
>>DMA teardown. I think it's preferable from a user perspective if group
>>numbering remains stable, rather than changing depending on the order in
>>which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>this in shape for 4.10, so I've taken the liberty of hacking up my own
>>branch (iommu/defer) based on v3 - would you mind taking a look at the
>>two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>to your later patches - that was an experiment which didn't really work out)
>
>Ok, will take a look at this now and respond more on this.
>
Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases.  The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Regards,
 Sricharan

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>Hi Robin,
>
>>Hi Robin,
>>
>>>On 04/11/16 15:16, Sricharan wrote:
>>>> Hi Robin,
>>>>
>>>>>>> Yikes, on second look, that definitely shouldn't be happening.
>>>>>>> Everything below is probably the resulting fallout.
>>>>>>
>>>>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>>>>>
>>>>>> I think the above print which says "failed to setup iommu_ops"
>>>>>> because the call ops->add_device failed in of_pci_iommu_configure
>>>>>> is the reason for the failure, in my case i simply do not get this even with
>>>>>> your scripts. ops->add_device succeeds in the rebind as well. So still
>>>>>> checking what could be happening in your case.
>>>>>
>>>>> I was looking at your code base from [1].The ops->add_device
>>>>> callback from of_pci_iommu_configure on the rebind is the
>>>>> one which is causing the failure. But not able to spot out
>>>>>from code which point is causing the failure. It would be very helpful
>>>>> if i can know which is the return value from the add_device callback
>>>>> or point inside add_device callback which fails in your setup.
>>>>>
>>>>>
>>>>> [1] git://linux-arm.org/linux-rm iommu/misc
>>>>
>>>> With little more try, i saw an issue where i had an failure
>>>> similar to what you reported. The issue happens when multiple
>>>> devices fall in to same group due to matching sids. I ended up
>>>> doing a fix like below and it would be nice to verify if it is the same
>>>> that we are seeing in your setup and if the fix makes a difference ?
>>>>
>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>> Date: Fri, 4 Nov 2016 20:28:49 +0530
>>>> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
>>>>
>>>> iommu_group_get_for_dev which gets called in the add_device
>>>> callback, increases the reference count of the iommu_group,
>>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>>> inturn calls device_group callback and in the case of arm-smmu
>>>> we call generic_device_group/pci_device_group which takes
>>>> care of increasing the group's reference. But when we return
>>>> an already existing group(when multiple devices have same group)
>>>> the reference is not incremented, resulting in issues when the
>>>> remove_device callback for the devices is invoked.
>>>> Fixing the same here.
>>>
>>>Bah, yes, this does look like my fault - after flip-flopping between
>>>about 3 different ways to keep refcounts for the S2CR entries, none of
>>>which would quite work, I ripped it all out but apparently still got
>>>things wrong, oh well. Thanks for figuring it out.
>> >
>>>On the probe-deferral angle, whilst it's useful to have uncovered this
>>>bug, I don't think we should actually be calling remove_device() from
>>>DMA teardown. I think it's preferable from a user perspective if group
>>>numbering remains stable, rather than changing depending on the order in
>>>which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>>this in shape for 4.10, so I've taken the liberty of hacking up my own
>>>branch (iommu/defer) based on v3 - would you mind taking a look at the
>>>two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>>to your later patches - that was an experiment which didn't really work out)
>>
>>Ok, will take a look at this now and respond more on this.
>>
>Sorry for the delayed response on this. I was OOO for the last few days.
>So i tested this branch and it worked fine. I tested it with a pci device
>for both normal and deferred probe cases.  The of/iommu patches
>are the cleanup/preparation patches and it looks fine. One thing is without
>calling the remove_device callback, the resources like (smes for exmaple)
>and the group association of the device all remain allocated. That does not
>feel correct, given that the associated device does not exist. So to
>understand that, what happens with VFIO in this case which makes the
>group renumbering/rebinding a problem ?
>

Would it be ok if i post a V4 based on your branch above ?

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Robin Murphy]

On 20/11/16 15:11, Sricharan wrote:
> Hi Robin,
> 
>> Hi Robin,
>>
>>> Hi Robin,
>>>
>>>> On 04/11/16 15:16, Sricharan wrote:
>>>>> Hi Robin,
>>>>>
>>>>>>>> Yikes, on second look, that definitely shouldn't be happening.
>>>>>>>> Everything below is probably the resulting fallout.
>>>>>>>
>>>>>>> [   40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
>>>>>>>
>>>>>>> I think the above print which says "failed to setup iommu_ops"
>>>>>>> because the call ops->add_device failed in of_pci_iommu_configure
>>>>>>> is the reason for the failure, in my case i simply do not get this even with
>>>>>>> your scripts. ops->add_device succeeds in the rebind as well. So still
>>>>>>> checking what could be happening in your case.
>>>>>>
>>>>>> I was looking at your code base from [1].The ops->add_device
>>>>>> callback from of_pci_iommu_configure on the rebind is the
>>>>>> one which is causing the failure. But not able to spot out
>>>>> >from code which point is causing the failure. It would be very helpful
>>>>>> if i can know which is the return value from the add_device callback
>>>>>> or point inside add_device callback which fails in your setup.
>>>>>>
>>>>>>
>>>>>> [1] git://linux-arm.org/linux-rm iommu/misc
>>>>>
>>>>> With little more try, i saw an issue where i had an failure
>>>>> similar to what you reported. The issue happens when multiple
>>>>> devices fall in to same group due to matching sids. I ended up
>>>>> doing a fix like below and it would be nice to verify if it is the same
>>>>> that we are seeing in your setup and if the fix makes a difference ?
>>>>>
>>>>> From: Sricharan R <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>>>>> Date: Fri, 4 Nov 2016 20:28:49 +0530
>>>>> Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
>>>>>
>>>>> iommu_group_get_for_dev which gets called in the add_device
>>>>> callback, increases the reference count of the iommu_group,
>>>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>>>> inturn calls device_group callback and in the case of arm-smmu
>>>>> we call generic_device_group/pci_device_group which takes
>>>>> care of increasing the group's reference. But when we return
>>>>> an already existing group(when multiple devices have same group)
>>>>> the reference is not incremented, resulting in issues when the
>>>>> remove_device callback for the devices is invoked.
>>>>> Fixing the same here.
>>>>
>>>> Bah, yes, this does look like my fault - after flip-flopping between
>>>> about 3 different ways to keep refcounts for the S2CR entries, none of
>>>> which would quite work, I ripped it all out but apparently still got
>>>> things wrong, oh well. Thanks for figuring it out.
>>>>
>>>> On the probe-deferral angle, whilst it's useful to have uncovered this
>>>> bug, I don't think we should actually be calling remove_device() from
>>>> DMA teardown. I think it's preferable from a user perspective if group
>>>> numbering remains stable, rather than changing depending on the order in
>>>> which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>>> this in shape for 4.10, so I've taken the liberty of hacking up my own
>>>> branch (iommu/defer) based on v3 - would you mind taking a look at the
>>>> two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>>> to your later patches - that was an experiment which didn't really work out)
>>>
>>> Ok, will take a look at this now and respond more on this.
>>>
>> Sorry for the delayed response on this. I was OOO for the last few days.
>> So i tested this branch and it worked fine. I tested it with a pci device
>> for both normal and deferred probe cases.  The of/iommu patches
>> are the cleanup/preparation patches and it looks fine. One thing is without
>> calling the remove_device callback, the resources like (smes for exmaple)
>> and the group association of the device all remain allocated. That does not
>> feel correct, given that the associated device does not exist. So to
>> understand that, what happens with VFIO in this case which makes the
>> group renumbering/rebinding a problem ?
>>
> 
> Would it be ok if i post a V4 based on your branch above ?

Sure, as long as none of the hacks slip through :) - I've just pushed
out a mild rework based on Lorenzo's v9, which I hope shouldn't break
anything for you.

Having thought a bit more about the add/remove thing, I'm inclined to
agree that the group numbering itself may not be that big an issue in
practice - sure, it could break my little script, but it looks like QEMU
and such work with the device ID rather than the group number directly,
so might not even notice. However, the fact remains that the callbacks
are intended to handle a device being added to/removed from its bus, and
will continue to do so on other platforms, so I don't like the idea of
introducing needlessly different behaviour. If you unbind a driver, the
stream IDs and everything don't stop existing at the hardware level; the
struct device to which the in-kernel data belongs still exists and
doesn't stop being associated with its bus. There's no good reason for
freeing SMEs that we'll only reallocate again (inadequately-specced
hardware with not enough SMRs/contexts is not a *good* reason), and
there are also some strong arguments against letting any stream IDs the
kernel knows about go back to bypass after a driver has been bound - by
keeping groups around as expected that's something we can implement
quite easily without having to completely lock down bypass for stream
IDs the kernel *doesn't* know about.

Robin.

> 
> Regards,
>  Sricharan
> 

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

<snip..>

>
>>>>>> iommu_group_get_for_dev which gets called in the add_device
>>>>>> callback, increases the reference count of the iommu_group,
>>>>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>>>>> inturn calls device_group callback and in the case of arm-smmu
>>>>>> we call generic_device_group/pci_device_group which takes
>>>>>> care of increasing the group's reference. But when we return
>>>>>> an already existing group(when multiple devices have same group)
>>>>>> the reference is not incremented, resulting in issues when the
>>>>>> remove_device callback for the devices is invoked.
>>>>>> Fixing the same here.
>>>>>
>>>>> Bah, yes, this does look like my fault - after flip-flopping between
>>>>> about 3 different ways to keep refcounts for the S2CR entries, none of
>>>>> which would quite work, I ripped it all out but apparently still got
>>>>> things wrong, oh well. Thanks for figuring it out.
>>>>>
>>>>> On the probe-deferral angle, whilst it's useful to have uncovered this
>>>>> bug, I don't think we should actually be calling remove_device() from
>>>>> DMA teardown. I think it's preferable from a user perspective if group
>>>>> numbering remains stable, rather than changing depending on the order in
>>>>> which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>>>> this in shape for 4.10, so I've taken the liberty of hacking up my own
>>>>> branch (iommu/defer) based on v3 - would you mind taking a look at the
>>>>> two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>>>> to your later patches - that was an experiment which didn't really work out)
>>>>
>>>> Ok, will take a look at this now and respond more on this.
>>>>
>>> Sorry for the delayed response on this. I was OOO for the last few days.
>>> So i tested this branch and it worked fine. I tested it with a pci device
>>> for both normal and deferred probe cases.  The of/iommu patches
>>> are the cleanup/preparation patches and it looks fine. One thing is without
>>> calling the remove_device callback, the resources like (smes for exmaple)
>>> and the group association of the device all remain allocated. That does not
>>> feel correct, given that the associated device does not exist. So to
>>> understand that, what happens with VFIO in this case which makes the
>>> group renumbering/rebinding a problem ?
>>>
>>
>> Would it be ok if i post a V4 based on your branch above ?
>
>Sure, as long as none of the hacks slip through :) - I've just pushed
>out a mild rework based on Lorenzo's v9, which I hope shouldn't break
>anything for you.
>

Ok sure, i will test and just the post out the stuff from your branch then
mostly by tomorrow.

>Having thought a bit more about the add/remove thing, I'm inclined to
>agree that the group numbering itself may not be that big an issue in
>practice - sure, it could break my little script, but it looks like QEMU
>and such work with the device ID rather than the group number directly,
>so might not even notice. However, the fact remains that the callbacks
>are intended to handle a device being added to/removed from its bus, and
>will continue to do so on other platforms, so I don't like the idea of
>introducing needlessly different behaviour. If you unbind a driver, the
>stream IDs and everything don't stop existing at the hardware level; the
>struct device to which the in-kernel data belongs still exists and
>doesn't stop being associated with its bus. There's no good reason for
>freeing SMEs that we'll only reallocate again (inadequately-specced
>hardware with not enough SMRs/contexts is not a *good* reason), and

ok, so SMRs/contexts was the reason i was adding the remove_dev
callback, but if thats not good enough then there was no other
intention.

>there are also some strong arguments against letting any stream IDs the
>kernel knows about go back to bypass after a driver has been bound - by

ok, but not sure why is this so ?

>keeping groups around as expected that's something we can implement
>quite easily without having to completely lock down bypass for stream
>IDs the kernel *doesn't* know about.
>

So do you mean in this case to keep the unbound device's group/context bank
to bypass rather than resetting the streamids ?

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Robin Murphy]

On 24/11/16 16:10, Sricharan wrote:
> Hi Robin,
> 
> <snip..>
> 
>>
>>>>>>> iommu_group_get_for_dev which gets called in the add_device
>>>>>>> callback, increases the reference count of the iommu_group,
>>>>>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>>>>>> inturn calls device_group callback and in the case of arm-smmu
>>>>>>> we call generic_device_group/pci_device_group which takes
>>>>>>> care of increasing the group's reference. But when we return
>>>>>>> an already existing group(when multiple devices have same group)
>>>>>>> the reference is not incremented, resulting in issues when the
>>>>>>> remove_device callback for the devices is invoked.
>>>>>>> Fixing the same here.
>>>>>>
>>>>>> Bah, yes, this does look like my fault - after flip-flopping between
>>>>>> about 3 different ways to keep refcounts for the S2CR entries, none of
>>>>>> which would quite work, I ripped it all out but apparently still got
>>>>>> things wrong, oh well. Thanks for figuring it out.
>>>>>>
>>>>>> On the probe-deferral angle, whilst it's useful to have uncovered this
>>>>>> bug, I don't think we should actually be calling remove_device() from
>>>>>> DMA teardown. I think it's preferable from a user perspective if group
>>>>>> numbering remains stable, rather than changing depending on the order in
>>>>>> which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>>>>> this in shape for 4.10, so I've taken the liberty of hacking up my own
>>>>>> branch (iommu/defer) based on v3 - would you mind taking a look at the
>>>>>> two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>>>>> to your later patches - that was an experiment which didn't really work out)
>>>>>
>>>>> Ok, will take a look at this now and respond more on this.
>>>>>
>>>> Sorry for the delayed response on this. I was OOO for the last few days.
>>>> So i tested this branch and it worked fine. I tested it with a pci device
>>>> for both normal and deferred probe cases.  The of/iommu patches
>>>> are the cleanup/preparation patches and it looks fine. One thing is without
>>>> calling the remove_device callback, the resources like (smes for exmaple)
>>>> and the group association of the device all remain allocated. That does not
>>>> feel correct, given that the associated device does not exist. So to
>>>> understand that, what happens with VFIO in this case which makes the
>>>> group renumbering/rebinding a problem ?
>>>>
>>>
>>> Would it be ok if i post a V4 based on your branch above ?
>>
>> Sure, as long as none of the hacks slip through :) - I've just pushed
>> out a mild rework based on Lorenzo's v9, which I hope shouldn't break
>> anything for you.
>>
> 
> Ok sure, i will test and just the post out the stuff from your branch then
> mostly by tomorrow.

Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

>> Having thought a bit more about the add/remove thing, I'm inclined to
>> agree that the group numbering itself may not be that big an issue in
>> practice - sure, it could break my little script, but it looks like QEMU
>> and such work with the device ID rather than the group number directly,
>> so might not even notice. However, the fact remains that the callbacks
>> are intended to handle a device being added to/removed from its bus, and
>> will continue to do so on other platforms, so I don't like the idea of
>> introducing needlessly different behaviour. If you unbind a driver, the
>> stream IDs and everything don't stop existing at the hardware level; the
>> struct device to which the in-kernel data belongs still exists and
>> doesn't stop being associated with its bus. There's no good reason for
>> freeing SMEs that we'll only reallocate again (inadequately-specced
>> hardware with not enough SMRs/contexts is not a *good* reason), and
> 
> ok, so SMRs/contexts was the reason i was adding the remove_dev
> callback, but if thats not good enough then there was no other
> intention.
> 
>> there are also some strong arguments against letting any stream IDs the
>> kernel knows about go back to bypass after a driver has been bound - by
> 
> ok, but not sure why is this so ?

Any device the kernel is in control of, having bound a driver to it,
definitely should not be doing DMA after that driver is unbound...

>> keeping groups around as expected that's something we can implement
>> quite easily without having to completely lock down bypass for stream
>> IDs the kernel *doesn't* know about.
>>
> 
> So do you mean in this case to keep the unbound device's group/context bank
> to bypass rather than resetting the streamids ?

...which we can easily enforce by keeping the device attached to its
default domain, in which nothing should be mapped by that point (we
could even have a group notifier switch its S2CRs to faulting entries
for extreme paranoia). Freeing the SMRs means those stream IDs would
instead fall back to the default "unmatched" behaviour, which in general
is going to be bypass, and thus allow DMA attacks.

It's harder to disable unmatched bypass in general, because we may have
devices which physically master through the SMMU but want low latency
more than they want translation (at the moment the best we can do is
leave the kernel unaware of those stream IDs), or there could be unknown
devices under control of the firmware or other agents which we would
disrupt by hitting a system-wide switch.

Robin.

> 
> Regards,
>  Sricharan
> 

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Robin,

>> <snip..>
>>
>>>
>>>>>>>> iommu_group_get_for_dev which gets called in the add_device
>>>>>>>> callback, increases the reference count of the iommu_group,
>>>>>>>> so we do an iommu_group_put after that. iommu_group_get_for_dev
>>>>>>>> inturn calls device_group callback and in the case of arm-smmu
>>>>>>>> we call generic_device_group/pci_device_group which takes
>>>>>>>> care of increasing the group's reference. But when we return
>>>>>>>> an already existing group(when multiple devices have same group)
>>>>>>>> the reference is not incremented, resulting in issues when the
>>>>>>>> remove_device callback for the devices is invoked.
>>>>>>>> Fixing the same here.
>>>>>>>
>>>>>>> Bah, yes, this does look like my fault - after flip-flopping between
>>>>>>> about 3 different ways to keep refcounts for the S2CR entries, none of
>>>>>>> which would quite work, I ripped it all out but apparently still got
>>>>>>> things wrong, oh well. Thanks for figuring it out.
>>>>>>>
>>>>>>> On the probe-deferral angle, whilst it's useful to have uncovered this
>>>>>>> bug, I don't think we should actually be calling remove_device() from
>>>>>>> DMA teardown. I think it's preferable from a user perspective if group
>>>>>>> numbering remains stable, rather than changing depending on the order in
>>>>>>> which they unbind/rebind VFIO drivers. I'm really keen to try and get
>>>>>>> this in shape for 4.10, so I've taken the liberty of hacking up my own
>>>>>>> branch (iommu/defer) based on v3 - would you mind taking a look at the
>>>>>>> two "iommu/of:" commits to see what you think? (Ignore the PCI changes
>>>>>>> to your later patches - that was an experiment which didn't really work out)
>>>>>>
>>>>>> Ok, will take a look at this now and respond more on this.
>>>>>>
>>>>> Sorry for the delayed response on this. I was OOO for the last few days.
>>>>> So i tested this branch and it worked fine. I tested it with a pci device
>>>>> for both normal and deferred probe cases.  The of/iommu patches
>>>>> are the cleanup/preparation patches and it looks fine. One thing is without
>>>>> calling the remove_device callback, the resources like (smes for exmaple)
>>>>> and the group association of the device all remain allocated. That does not
>>>>> feel correct, given that the associated device does not exist. So to
>>>>> understand that, what happens with VFIO in this case which makes the
>>>>> group renumbering/rebinding a problem ?
>>>>>
>>>>
>>>> Would it be ok if i post a V4 based on your branch above ?
>>>
>>> Sure, as long as none of the hacks slip through :) - I've just pushed
>>> out a mild rework based on Lorenzo's v9, which I hope shouldn't break
>>> anything for you.
>>>
>>
>> Ok sure, i will test and just the post out the stuff from your branch then
>> mostly by tomorrow.
>
>Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
>now, so it's probably worth pulling the rest of that in (beyond the one
>patch I picked) to make sure the of_dma_configure/acpi_dma_configure
>paths don't inadvertently diverge.
>

I rebased and was testing your branch with Lorenzo's series. One thing
i am still trying to get right is the acpi_dma_configure call. With your
series dma_configure calls pci_dma/of_dma configure, so i am just adding
acpi_dma_configure call there for non-pci ACPI devices as well. I see that
acpi_dma_configure right now is called from acpi_bind_one and 
iort_add_smmu_platform_device, both go through the really_probe function
path, so moving acpi_dma_configure from the above the two functions
to dma_configure. I remember we discussed this on another thread, so
hopefully it is correct. I do not have an platform to test the ACPI though.
I will take some testing help on V4 for this.

>>> Having thought a bit more about the add/remove thing, I'm inclined to
>>> agree that the group numbering itself may not be that big an issue in
>>> practice - sure, it could break my little script, but it looks like QEMU
>>> and such work with the device ID rather than the group number directly,
>>> so might not even notice. However, the fact remains that the callbacks
>>> are intended to handle a device being added to/removed from its bus, and
>>> will continue to do so on other platforms, so I don't like the idea of
>>> introducing needlessly different behaviour. If you unbind a driver, the
>>> stream IDs and everything don't stop existing at the hardware level; the
>>> struct device to which the in-kernel data belongs still exists and
>>> doesn't stop being associated with its bus. There's no good reason for
>>> freeing SMEs that we'll only reallocate again (inadequately-specced
>>> hardware with not enough SMRs/contexts is not a *good* reason), and
>>
>> ok, so SMRs/contexts was the reason i was adding the remove_dev
>> callback, but if thats not good enough then there was no other
>> intention.
>>
>>> there are also some strong arguments against letting any stream IDs the
>>> kernel knows about go back to bypass after a driver has been bound - by
>>
>> ok, but not sure why is this so ?
>
>Any device the kernel is in control of, having bound a driver to it,
>definitely should not be doing DMA after that driver is unbound...
>

ok.

>>> keeping groups around as expected that's something we can implement
>>> quite easily without having to completely lock down bypass for stream
>>> IDs the kernel *doesn't* know about.
>>>
>>
>> So do you mean in this case to keep the unbound device's group/context bank
>> to bypass rather than resetting the streamids ?
>
>...which we can easily enforce by keeping the device attached to its
>default domain, in which nothing should be mapped by that point (we
>could even have a group notifier switch its S2CRs to faulting entries
>for extreme paranoia). Freeing the SMRs means those stream IDs would
>instead fall back to the default "unmatched" behaviour, which in general
>is going to be bypass, and thus allow DMA attacks.
>
>It's harder to disable unmatched bypass in general, because we may have
>devices which physically master through the SMMU but want low latency
>more than they want translation (at the moment the best we can do is
>leave the kernel unaware of those stream IDs), or there could be unknown
>devices under control of the firmware or other agents which we would
>disrupt by hitting a system-wide switch.
>

ok thanks, understand the point now. Agree that putting the previously bound
devices to fault is right than putting it to bypass. So the notifier to switch the
S2CRs to faulting entries can be added as a separate patch on top of this.

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Lorenzo Pieralisi]

On Mon, Nov 28, 2016 at 11:12:08PM +0530, Sricharan wrote:

[...]

> >Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
> >now, so it's probably worth pulling the rest of that in (beyond the one
> >patch I picked) to make sure the of_dma_configure/acpi_dma_configure
> >paths don't inadvertently diverge.
> >
> 
> I rebased and was testing your branch with Lorenzo's series. One thing
> i am still trying to get right is the acpi_dma_configure call. With your
> series dma_configure calls pci_dma/of_dma configure, so i am just adding
> acpi_dma_configure call there for non-pci ACPI devices as well. I see that
> acpi_dma_configure right now is called from acpi_bind_one and 
> iort_add_smmu_platform_device, both go through the really_probe function
> path, so moving acpi_dma_configure from the above the two functions
> to dma_configure. I remember we discussed this on another thread, so
> hopefully it is correct. I do not have an platform to test the ACPI though.
> I will take some testing help on V4 for this.

I am happy to test it for you please just send me a pointer at your v4
code.

Thank you !
Lorenzo

RE: [PATCH V3 0/8] IOMMU probe deferral support

[Sricharan]

Hi Lorenzo,

>-----Original Message-----
>From: linux-arm-kernel [mailto:linux-arm-kernel-bounces-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org] On Behalf Of Lorenzo Pieralisi
>Sent: Monday, November 28, 2016 11:44 PM
>To: Sricharan <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
>Cc: linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org; tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
>foundation.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; 'Robin Murphy' <robin.murphy-5wv7dgnIgG8@public.gmane.org>;
>linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org
>Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
>
>On Mon, Nov 28, 2016 at 11:12:08PM +0530, Sricharan wrote:
>
>[...]
>
>> >Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
>> >now, so it's probably worth pulling the rest of that in (beyond the one
>> >patch I picked) to make sure the of_dma_configure/acpi_dma_configure
>> >paths don't inadvertently diverge.
>> >
>>
>> I rebased and was testing your branch with Lorenzo's series. One thing
>> i am still trying to get right is the acpi_dma_configure call. With your
>> series dma_configure calls pci_dma/of_dma configure, so i am just adding
>> acpi_dma_configure call there for non-pci ACPI devices as well. I see that
>> acpi_dma_configure right now is called from acpi_bind_one and
>> iort_add_smmu_platform_device, both go through the really_probe function
>> path, so moving acpi_dma_configure from the above the two functions
>> to dma_configure. I remember we discussed this on another thread, so
>> hopefully it is correct. I do not have an platform to test the ACPI though.
>> I will take some testing help on V4 for this.
>
>I am happy to test it for you please just send me a pointer at your v4
>code.
>
 I posted the v4 and CCed you there. So i am little skeptical about the acpi
changes that i have posted. I was checking for a function equivalent 
in acpi as of_match_node in DT, to figure out if the iommu_spec.np that
the master device is pointing to is there in the iommu_of_table and based
on that we can decide if to defer the probe. I was seeing iort_scan_node
was its equivalent. But if that is not correct, then last patch has to be reworked.
Anyways will be good to know your feedback on this.

Regards,
 Sricharan

Re: [PATCH V3 0/8] IOMMU probe deferral support

[Lorenzo Pieralisi]

Hi Sricharan,

On Wed, Nov 30, 2016 at 06:04:13AM +0530, Sricharan wrote:
> Hi Lorenzo,
> 
> >-----Original Message-----
> >From: linux-arm-kernel [mailto:linux-arm-kernel-bounces-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org] On Behalf Of Lorenzo Pieralisi
> >Sent: Monday, November 28, 2016 11:44 PM
> >To: Sricharan <sricharan-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org>
> >Cc: linux-arm-msm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org; will.deacon-5wv7dgnIgG8@public.gmane.org; tfiga-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> >foundation.org; srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org; laurent.pinchart-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org; 'Robin Murphy' <robin.murphy-5wv7dgnIgG8@public.gmane.org>;
> >linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org; m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org
> >Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
> >
> >On Mon, Nov 28, 2016 at 11:12:08PM +0530, Sricharan wrote:
> >
> >[...]
> >
> >> >Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
> >> >now, so it's probably worth pulling the rest of that in (beyond the one
> >> >patch I picked) to make sure the of_dma_configure/acpi_dma_configure
> >> >paths don't inadvertently diverge.
> >> >
> >>
> >> I rebased and was testing your branch with Lorenzo's series. One thing
> >> i am still trying to get right is the acpi_dma_configure call. With your
> >> series dma_configure calls pci_dma/of_dma configure, so i am just adding
> >> acpi_dma_configure call there for non-pci ACPI devices as well. I see that
> >> acpi_dma_configure right now is called from acpi_bind_one and
> >> iort_add_smmu_platform_device, both go through the really_probe function
> >> path, so moving acpi_dma_configure from the above the two functions
> >> to dma_configure. I remember we discussed this on another thread, so
> >> hopefully it is correct. I do not have an platform to test the ACPI though.
> >> I will take some testing help on V4 for this.
> >
> >I am happy to test it for you please just send me a pointer at your v4
> >code.
> >
>  I posted the v4 and CCed you there. So i am little skeptical about the acpi
> changes that i have posted. I was checking for a function equivalent 
> in acpi as of_match_node in DT, to figure out if the iommu_spec.np that
> the master device is pointing to is there in the iommu_of_table and based
> on that we can decide if to defer the probe. I was seeing iort_scan_node
> was its equivalent. But if that is not correct, then last patch has to be reworked.
> Anyways will be good to know your feedback on this.

Sure I will test it asap, thanks for putting it together.

Thanks,
Lorenzo