[PATCH v12 0/4] iommu/arm-smmu: Add runtime pm/sleep support

[PATCH v12 0/4] iommu/arm-smmu: Add runtime pm/sleep support

[Vivek Gautam]

This series provides the support for turning on the arm-smmu's
clocks/power domains using runtime pm. This is done using the
recently introduced device links patches, which lets the smmu's
runtime to follow the master's runtime pm, so the smmu remains
powered only when the masters use it.
As not all implementations support clock/power gating, we are checking
for a valid 'smmu->dev's pm_domain' to conditionally enable the runtime
power management for such smmu implementations that can support it.
With addition of a new device link flag DL_FLAG_AUTOREMOVE_SUPPLIER [11]
the device links created between arm-smmu and its clients will be
automatically purged when arm-smmu driver unbinds from its device.

This series also adds support for Qcom's arm-smmu-v2 variant that
has different clocks and power requirements.

Took some reference from the exynos runtime patches [1].

With conditional runtime pm now, we avoid touching dev->power.lock
in fastpaths for smmu implementations that don't need to do anything
useful with pm_runtime.
This lets us to use the much-argued pm_runtime_get_sync/put_sync()
calls in map/unmap callbacks so that the clients do not have to
worry about handling any of the arm-smmu's power.

Previous version of this patch series is @ [5].

Note: This series is now based on the device link changes [11] for
adding new flag - DL_FLAG_AUTOREMOVE_SUPPLIER.
Rafael will pull in the device link changes and these patches have
to be pulled in based on Rafael's branch. As Rafael said -
"I would prefer to apply them myself to be honest and put them on an
public git branch for you to pull from."

[v12]
   * Use new device link's flag introduced in [11] -
     DL_FLAG_AUTOREMOVE_SUPPLIER. With this devices links are automatically
     purged when arm-smmu driver unbinds.
   * Using pm_runtime_force_suspend() instead of pm_runtime_disable() to
     avoid following warning from arm_smmu_device_remove()

     [295711.537507] ------------[ cut here ]------------
     [295711.544226] Unpreparing enabled smmu_mdp_ahb_clk
     [295711.549099] WARNING: CPU: 0 PID: 1 at ../drivers/clk/clk.c:697
                     clk_core_unprepare+0xd8/0xe0
     ...
     [295711.674073] Call trace:
     [295711.679454]  clk_core_unprepare+0xd8/0xe0
     [295711.682059]  clk_unprepare+0x28/0x40
     [295711.685964]  clk_bulk_unprepare+0x28/0x40
     [295711.689701]  arm_smmu_device_remove+0x88/0xd8
     [295711.693692]  arm_smmu_device_shutdown+0xc/0x18
     [295711.698120]  platform_drv_shutdown+0x20/0x30

[v11]
   * Some more cleanups for device link. We don't need an explicit
     delete for device link from the driver, but just set the flag
     DL_FLAG_AUTOREMOVE.
     device_link_add() API description says -
     "If the DL_FLAG_AUTOREMOVE is set, the link will be removed
     automatically when the consumer device driver unbinds."
   * Addressed the comments for 'smmu' in arm_smmu_map/unmap().
   * Dropped the patch [10] that introduced device_link_del_dev() API. 

[v10]
   * Introduce device_link_del_dev() API to delete the link between
     given consumer and supplier devices. The users of device link
     do not need to store link pointer to delete the link later.
     They can straightaway use this API by passing consumer and
     supplier devices.
   * Made corresponding changes to arm-smmu driver patch handling the
     device links.
   * Dropped the patch [9] that was adding device_link_find() API to
     device core layer. device_link_del_dev() serves the purpose to
     directly delete the link between two given devices.

[v9]
   * Removed 'rpm_supported' flag, instead checking on pm_domain
     to enable runtime pm.
   * Creating device link only when the runtime pm is enabled, as we
     don't need a device link besides managing the power dependency
     between supplier and consumer devices.
   * Introducing a patch to add device_link_find() API that finds
     and existing link between supplier and consumer devices.
     Also, made necessary change to device_link_add() to use this API.
   * arm_smmu_remove_device() now uses this device_link_find() to find
     the device link between smmu device and the master device, and then
     delete this link.
   * Dropped the destroy_domain_context() fix [8] as it was rather,
     introducing catastrophically bad problem by destroying
     'good dev's domain context.
   * Added 'Reviwed-by' tag for Tomasz's review.

[v8]
   * Major change -
     - Added a flag 'rpm_supported' which each platform that supports
       runtime pm, can enable, and we enable runtime_pm over arm-smmu
       only when this flag is set.
     - Adding the conditional pm_runtime_get/put() calls to .map, .unmap
       and .attach_dev ops.
     - Dropped the patch [6] that exported pm_runtim_get/put_suupliers(),
       and also dropped the user driver patch [7] for these APIs.

   * Clock code further cleanup
     - doing only clk_bulk_enable() and clk_bulk_disable() in runtime pm
       callbacks. We shouldn't be taking a slow path (clk_prepare/unprepare())
       from these runtime pm callbacks. Thereby, moved clk_bulk_prepare() to
       arm_smmu_device_probe(), and clk_bulk_unprepare() to
       arm_smmu_device_remove().
     - clk data filling to a common method arm_smmu_fill_clk_data() that
       fills the clock ids and number of clocks.

   * Addressed other nits and comments
     - device_link_add() error path fixed.
     - Fix for checking negative error value from pm_runtime_get_sync().
     - Documentation redo.

   * Added another patch fixing the error path in arm_smmu_attach_dev()
     to destroy allocated domain context.

[v7]
   * Addressed review comments given by Robin Murphy -
     - Added device_link_del() in .remove_device path.
     - Error path cleanup in arm_smmu_add_device().
     - Added pm_runtime_get/put_sync() in .remove path, and replaced
        pm_runtime_force_suspend() with pm_runtime_disable().
     - clk_names cleanup in arm_smmu_init_clks()
   * Added 'Reviewed-by' given by Rob H.

[V6]
   * Added Ack given by Rafael to first patch in the series.
   * Addressed Rob Herring's comment for adding soc specific compatible
     string as well besides 'qcom,smmu-v2'.

[V5]
   * Dropped runtime pm calls from "arm_smmu_unmap" op as discussed over
     the list [3] for the last patch series.
   * Added a patch to export pm_runtime_get/put_suppliers() APIs to the
     series as agreed with Rafael [4].
   * Added the related patch for msm drm iommu layer to use
     pm_runtime_get/put_suppliers() APIs in msm_mmu_funcs.
   * Dropped arm-mmu500 clock patch since that would break existing
     platforms.
   * Changed compatible 'qcom,msm8996-smmu-v2' to 'qcom,smmu-v2' to reflect
     the IP version rather than the platform on which it is used.
     The same IP is used across multiple platforms including msm8996,
     and sdm845 etc.
   * Using clock bulk APIs to handle the clocks available to the IP as
     suggested by Stephen Boyd.
   * The first patch in v4 version of the patch-series:
     ("iommu/arm-smmu: Fix the error path in arm_smmu_add_device") has
     already made it to mainline.

[V4]
   * Reworked the clock handling part. We now take clock names as data
     in the driver for supported compatible versions, and loop over them
     to get, enable, and disable the clocks.
   * Using qcom,msm8996 based compatibles for bindings instead of a generic
     qcom compatible.
   * Refactor MMU500 patch to just add the necessary clock names data and
     corresponding bindings.
   * Added the pm_runtime_get/put() calls in .unmap iommu op (fix added by
     Stanimir on top of previous patch version.
   * Added a patch to fix error path in arm_smmu_add_device()
   * Removed patch 3/5 of V3 patch series that added qcom,smmu-v2 bindings.

[V3]
   * Reworked the patches to keep the clocks init/enabling function
     separately for each compatible.

   * Added clocks bindings for MMU40x/500.

   * Added a new compatible for qcom,smmu-v2 implementation and
     the clock bindings for the same.

   * Rebased on top of 4.11-rc1

[V2]
   * Split the patches little differently.

   * Addressed comments.

   * Removed the patch #4 [2] from previous post
     for arm-smmu context save restore. Planning to
     post this separately after reworking/addressing Robin's
     feedback.

   * Reversed the sequence to disable clocks than enabling.
     This was required for those cases where the
     clocks are populated in a dependent order from DT.

[1] https://lkml.org/lkml/2016/10/20/70
[2] https://patchwork.kernel.org/patch/9389717/
[3] https://patchwork.kernel.org/patch/10204925/
[4] https://patchwork.kernel.org/patch/10102445/
[5] https://lkml.org/lkml/2018/3/22/191
[6] https://patchwork.kernel.org/patch/10204945/
[7] https://patchwork.kernel.org/patch/10204925/
[8] https://patchwork.kernel.org/patch/10254105/
[9] https://patchwork.kernel.org/patch/10277975/
[10] https://patchwork.kernel.org/patch/10281613/
[11] https://patchwork.kernel.org/patch/10491481/

Sricharan R (3):
  iommu/arm-smmu: Add pm_runtime/sleep ops
  iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device
  iommu/arm-smmu: Add the device_link between masters and smmu

Vivek Gautam (1):
  iommu/arm-smmu: Add support for qcom,smmu-v2 variant

 .../devicetree/bindings/iommu/arm,smmu.txt         |  42 +++++
 drivers/iommu/arm-smmu.c                           | 178 +++++++++++++++++++--
 2 files changed, 210 insertions(+), 10 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

[PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Vivek Gautam]

From: Sricharan R <sricharan@codeaurora.org>

The smmu needs to be functional only when the respective
master's using it are active. The device_link feature
helps to track such functional dependencies, so that the
iommu gets powered when the master device enables itself
using pm_runtime. So by adapting the smmu driver for
runtime pm, above said dependency can be addressed.

This patch adds the pm runtime/sleep callbacks to the
driver and also the functions to parse the smmu clocks
from DT and enable them in resume/suspend.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
Signed-off-by: Archit Taneja <architt@codeaurora.org>
[vivek: Clock rework to request bulk of clocks]
Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
---

 - No change since v11.

 drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 58 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index f7a96bcf94a6..a01d0dde21dd 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -48,6 +48,7 @@
 #include <linux/of_iommu.h>
 #include <linux/pci.h>
 #include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 
@@ -205,6 +206,8 @@ struct arm_smmu_device {
 	u32				num_global_irqs;
 	u32				num_context_irqs;
 	unsigned int			*irqs;
+	struct clk_bulk_data		*clks;
+	int				num_clks;
 
 	u32				cavium_id_base; /* Specific to Cavium */
 
@@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
 struct arm_smmu_match_data {
 	enum arm_smmu_arch_version version;
 	enum arm_smmu_implementation model;
+	const char * const *clks;
+	int num_clks;
 };
 
 #define ARM_SMMU_MATCH_DATA(name, ver, imp)	\
-static struct arm_smmu_match_data name = { .version = ver, .model = imp }
+static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
 
 ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
 ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
@@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
 };
 MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
 
+static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
+				   const char * const *clks)
+{
+	int i;
+
+	if (smmu->num_clks < 1)
+		return;
+
+	smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
+				  sizeof(*smmu->clks), GFP_KERNEL);
+	if (!smmu->clks)
+		return;
+
+	for (i = 0; i < smmu->num_clks; i++)
+		smmu->clks[i].id = clks[i];
+}
+
 #ifdef CONFIG_ACPI
 static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
 {
@@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	data = of_device_get_match_data(dev);
 	smmu->version = data->version;
 	smmu->model = data->model;
+	smmu->num_clks = data->num_clks;
+
+	arm_smmu_fill_clk_data(smmu, data->clks);
 
 	parse_driver_options(smmu);
 
@@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		smmu->irqs[i] = irq;
 	}
 
+	err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
+	if (err)
+		return err;
+
+	err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
+	if (err)
+		return err;
+
 	err = arm_smmu_device_cfg_probe(smmu);
 	if (err)
 		return err;
@@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 
 	/* Turn the thing off */
 	writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+
+	clk_bulk_unprepare(smmu->num_clks, smmu->clks);
+
 	return 0;
 }
 
@@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
 	return 0;
 }
 
-static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
+static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
+{
+	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+
+	return clk_bulk_enable(smmu->num_clks, smmu->clks);
+}
+
+static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
+{
+	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+
+	clk_bulk_disable(smmu->num_clks, smmu->clks);
+
+	return 0;
+}
+
+static const struct dev_pm_ops arm_smmu_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
+	SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend,
+			   arm_smmu_runtime_resume, NULL)
+};
 
 static struct platform_driver arm_smmu_driver = {
 	.driver	= {
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
> From: Sricharan R <sricharan@codeaurora.org>
> 
> The smmu needs to be functional only when the respective
> master's using it are active. The device_link feature
> helps to track such functional dependencies, so that the
> iommu gets powered when the master device enables itself
> using pm_runtime. So by adapting the smmu driver for
> runtime pm, above said dependency can be addressed.
> 
> This patch adds the pm runtime/sleep callbacks to the
> driver and also the functions to parse the smmu clocks
> from DT and enable them in resume/suspend.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> Signed-off-by: Archit Taneja <architt@codeaurora.org>
> [vivek: Clock rework to request bulk of clocks]
> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
> ---
> 
>  - No change since v11.
> 
>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 58 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index f7a96bcf94a6..a01d0dde21dd 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -48,6 +48,7 @@
>  #include <linux/of_iommu.h>
>  #include <linux/pci.h>
>  #include <linux/platform_device.h>
> +#include <linux/pm_runtime.h>
>  #include <linux/slab.h>
>  #include <linux/spinlock.h>
>  
> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>  	u32				num_global_irqs;
>  	u32				num_context_irqs;
>  	unsigned int			*irqs;
> +	struct clk_bulk_data		*clks;
> +	int				num_clks;
>  
>  	u32				cavium_id_base; /* Specific to Cavium */
>  
> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>  struct arm_smmu_match_data {
>  	enum arm_smmu_arch_version version;
>  	enum arm_smmu_implementation model;
> +	const char * const *clks;
> +	int num_clks;
>  };
>  
>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)	\
> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>  
>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>  };
>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>  
> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
> +				   const char * const *clks)
> +{
> +	int i;
> +
> +	if (smmu->num_clks < 1)
> +		return;
> +
> +	smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
> +				  sizeof(*smmu->clks), GFP_KERNEL);
> +	if (!smmu->clks)
> +		return;
> +
> +	for (i = 0; i < smmu->num_clks; i++)
> +		smmu->clks[i].id = clks[i];
> +}
> +
>  #ifdef CONFIG_ACPI
>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>  {
> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>  	data = of_device_get_match_data(dev);
>  	smmu->version = data->version;
>  	smmu->model = data->model;
> +	smmu->num_clks = data->num_clks;
> +
> +	arm_smmu_fill_clk_data(smmu, data->clks);
>  
>  	parse_driver_options(smmu);
>  
> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>  		smmu->irqs[i] = irq;
>  	}
>  
> +	err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
> +	if (err)
> +		return err;
> +
> +	err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
> +	if (err)
> +		return err;
> +
>  	err = arm_smmu_device_cfg_probe(smmu);
>  	if (err)
>  		return err;
> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>  
>  	/* Turn the thing off */
>  	writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> +
> +	clk_bulk_unprepare(smmu->num_clks, smmu->clks);
> +
>  	return 0;
>  }
>  
> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>  	return 0;
>  }
>  
> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
> +{
> +	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> +
> +	return clk_bulk_enable(smmu->num_clks, smmu->clks);
> +}
> +
> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
> +{
> +	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> +
> +	clk_bulk_disable(smmu->num_clks, smmu->clks);
> +
> +	return 0;
> +}
> +
> +static const struct dev_pm_ops arm_smmu_pm_ops = {
> +	SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)

This is suspicious.

If you need a runtime suspend method, why do you think that it is not necessary
to suspend the device during system-wide transitions?

> +	SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend,
> +			   arm_smmu_runtime_resume, NULL)
> +};
>  
>  static struct platform_driver arm_smmu_driver = {
>  	.driver	= {
> 

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Vivek Gautam]

Hi Rafael,


On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>> From: Sricharan R <sricharan@codeaurora.org>
>>
>> The smmu needs to be functional only when the respective
>> master's using it are active. The device_link feature
>> helps to track such functional dependencies, so that the
>> iommu gets powered when the master device enables itself
>> using pm_runtime. So by adapting the smmu driver for
>> runtime pm, above said dependency can be addressed.
>>
>> This patch adds the pm runtime/sleep callbacks to the
>> driver and also the functions to parse the smmu clocks
>> from DT and enable them in resume/suspend.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>> [vivek: Clock rework to request bulk of clocks]
>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>> ---
>>
>>  - No change since v11.
>>
>>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>  1 file changed, 58 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index f7a96bcf94a6..a01d0dde21dd 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -48,6 +48,7 @@
>>  #include <linux/of_iommu.h>
>>  #include <linux/pci.h>
>>  #include <linux/platform_device.h>
>> +#include <linux/pm_runtime.h>
>>  #include <linux/slab.h>
>>  #include <linux/spinlock.h>
>>
>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>       u32                             num_global_irqs;
>>       u32                             num_context_irqs;
>>       unsigned int                    *irqs;
>> +     struct clk_bulk_data            *clks;
>> +     int                             num_clks;
>>
>>       u32                             cavium_id_base; /* Specific to Cavium */
>>
>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>  struct arm_smmu_match_data {
>>       enum arm_smmu_arch_version version;
>>       enum arm_smmu_implementation model;
>> +     const char * const *clks;
>> +     int num_clks;
>>  };
>>
>>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>
>>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>  };
>>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>
>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>> +                                const char * const *clks)
>> +{
>> +     int i;
>> +
>> +     if (smmu->num_clks < 1)
>> +             return;
>> +
>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>> +     if (!smmu->clks)
>> +             return;
>> +
>> +     for (i = 0; i < smmu->num_clks; i++)
>> +             smmu->clks[i].id = clks[i];
>> +}
>> +
>>  #ifdef CONFIG_ACPI
>>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>  {
>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>       data = of_device_get_match_data(dev);
>>       smmu->version = data->version;
>>       smmu->model = data->model;
>> +     smmu->num_clks = data->num_clks;
>> +
>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>
>>       parse_driver_options(smmu);
>>
>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>               smmu->irqs[i] = irq;
>>       }
>>
>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>> +     if (err)
>> +             return err;
>> +
>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>> +     if (err)
>> +             return err;
>> +
>>       err = arm_smmu_device_cfg_probe(smmu);
>>       if (err)
>>               return err;
>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>
>>       /* Turn the thing off */
>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>> +
>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>> +
>>       return 0;
>>  }
>>
>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>       return 0;
>>  }
>>
>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>> +{
>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>> +
>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>> +}
>> +
>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>> +{
>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>> +
>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>> +
>> +     return 0;
>> +}
>> +
>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>
> This is suspicious.
>
> If you need a runtime suspend method, why do you think that it is not necessary
> to suspend the device during system-wide transitions?

Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
In that case the clocks have to be enabled in the resume path too.

I remember Tomasz pointed to that we shouldn't need clock enable in resume
path [1].

[1] https://lkml.org/lkml/2018/3/15/60


Best regards
Vivek

>
>> +     SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend,
>> +                        arm_smmu_runtime_resume, NULL)
>> +};
>>
>>  static struct platform_driver arm_smmu_driver = {
>>       .driver = {
>>
>
>



-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> Hi Rafael,
>
>
> On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>> From: Sricharan R <sricharan@codeaurora.org>
>>>
>>> The smmu needs to be functional only when the respective
>>> master's using it are active. The device_link feature
>>> helps to track such functional dependencies, so that the
>>> iommu gets powered when the master device enables itself
>>> using pm_runtime. So by adapting the smmu driver for
>>> runtime pm, above said dependency can be addressed.
>>>
>>> This patch adds the pm runtime/sleep callbacks to the
>>> driver and also the functions to parse the smmu clocks
>>> from DT and enable them in resume/suspend.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>> [vivek: Clock rework to request bulk of clocks]
>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>> ---
>>>
>>>  - No change since v11.
>>>
>>>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>>  1 file changed, 58 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> index f7a96bcf94a6..a01d0dde21dd 100644
>>> --- a/drivers/iommu/arm-smmu.c
>>> +++ b/drivers/iommu/arm-smmu.c
>>> @@ -48,6 +48,7 @@
>>>  #include <linux/of_iommu.h>
>>>  #include <linux/pci.h>
>>>  #include <linux/platform_device.h>
>>> +#include <linux/pm_runtime.h>
>>>  #include <linux/slab.h>
>>>  #include <linux/spinlock.h>
>>>
>>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>>       u32                             num_global_irqs;
>>>       u32                             num_context_irqs;
>>>       unsigned int                    *irqs;
>>> +     struct clk_bulk_data            *clks;
>>> +     int                             num_clks;
>>>
>>>       u32                             cavium_id_base; /* Specific to Cavium */
>>>
>>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>>  struct arm_smmu_match_data {
>>>       enum arm_smmu_arch_version version;
>>>       enum arm_smmu_implementation model;
>>> +     const char * const *clks;
>>> +     int num_clks;
>>>  };
>>>
>>>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>
>>>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>>  };
>>>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>>
>>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>> +                                const char * const *clks)
>>> +{
>>> +     int i;
>>> +
>>> +     if (smmu->num_clks < 1)
>>> +             return;
>>> +
>>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>> +     if (!smmu->clks)
>>> +             return;
>>> +
>>> +     for (i = 0; i < smmu->num_clks; i++)
>>> +             smmu->clks[i].id = clks[i];
>>> +}
>>> +
>>>  #ifdef CONFIG_ACPI
>>>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>>  {
>>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>>       data = of_device_get_match_data(dev);
>>>       smmu->version = data->version;
>>>       smmu->model = data->model;
>>> +     smmu->num_clks = data->num_clks;
>>> +
>>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>>
>>>       parse_driver_options(smmu);
>>>
>>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>               smmu->irqs[i] = irq;
>>>       }
>>>
>>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>> +     if (err)
>>> +             return err;
>>> +
>>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>> +     if (err)
>>> +             return err;
>>> +
>>>       err = arm_smmu_device_cfg_probe(smmu);
>>>       if (err)
>>>               return err;
>>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>>
>>>       /* Turn the thing off */
>>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>> +
>>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>> +
>>>       return 0;
>>>  }
>>>
>>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>>       return 0;
>>>  }
>>>
>>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>> +{
>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>> +
>>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>> +}
>>> +
>>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>> +{
>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>> +
>>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>> +
>>> +     return 0;
>>> +}
>>> +
>>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>
>> This is suspicious.
>>
>> If you need a runtime suspend method, why do you think that it is not necessary
>> to suspend the device during system-wide transitions?
>
> Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
> In that case the clocks have to be enabled in the resume path too.
>
> I remember Tomasz pointed to that we shouldn't need clock enable in resume
> path [1].
>
> [1] https://lkml.org/lkml/2018/3/15/60

Honestly, I just don't know. :-)

It just looks odd the way it is done.  I think the clock should be
gated during system-wide suspend too, because the system can spend
much more time in a sleep state than in the working state, on average.

And note that you cannot rely on runtime PM to always do it for you,
because it may be disabled at a client device or even blocked by user
space via power/control in sysfs and that shouldn't matter for
system-wide PM.

Thanks,
Rafael

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Tomasz Figa]

On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
> <vivek.gautam@codeaurora.org> wrote:
> > Hi Rafael,
> >
> >
> > On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
> >>> From: Sricharan R <sricharan@codeaurora.org>
> >>>
> >>> The smmu needs to be functional only when the respective
> >>> master's using it are active. The device_link feature
> >>> helps to track such functional dependencies, so that the
> >>> iommu gets powered when the master device enables itself
> >>> using pm_runtime. So by adapting the smmu driver for
> >>> runtime pm, above said dependency can be addressed.
> >>>
> >>> This patch adds the pm runtime/sleep callbacks to the
> >>> driver and also the functions to parse the smmu clocks
> >>> from DT and enable them in resume/suspend.
> >>>
> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> >>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
> >>> [vivek: Clock rework to request bulk of clocks]
> >>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> >>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
> >>> ---
> >>>
> >>>  - No change since v11.
> >>>
> >>>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
> >>>  1 file changed, 58 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> >>> index f7a96bcf94a6..a01d0dde21dd 100644
> >>> --- a/drivers/iommu/arm-smmu.c
> >>> +++ b/drivers/iommu/arm-smmu.c
> >>> @@ -48,6 +48,7 @@
> >>>  #include <linux/of_iommu.h>
> >>>  #include <linux/pci.h>
> >>>  #include <linux/platform_device.h>
> >>> +#include <linux/pm_runtime.h>
> >>>  #include <linux/slab.h>
> >>>  #include <linux/spinlock.h>
> >>>
> >>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
> >>>       u32                             num_global_irqs;
> >>>       u32                             num_context_irqs;
> >>>       unsigned int                    *irqs;
> >>> +     struct clk_bulk_data            *clks;
> >>> +     int                             num_clks;
> >>>
> >>>       u32                             cavium_id_base; /* Specific to Cavium */
> >>>
> >>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
> >>>  struct arm_smmu_match_data {
> >>>       enum arm_smmu_arch_version version;
> >>>       enum arm_smmu_implementation model;
> >>> +     const char * const *clks;
> >>> +     int num_clks;
> >>>  };
> >>>
> >>>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
> >>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
> >>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
> >>>
> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
> >>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
> >>>  };
> >>>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
> >>>
> >>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
> >>> +                                const char * const *clks)
> >>> +{
> >>> +     int i;
> >>> +
> >>> +     if (smmu->num_clks < 1)
> >>> +             return;
> >>> +
> >>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
> >>> +                               sizeof(*smmu->clks), GFP_KERNEL);
> >>> +     if (!smmu->clks)
> >>> +             return;
> >>> +
> >>> +     for (i = 0; i < smmu->num_clks; i++)
> >>> +             smmu->clks[i].id = clks[i];
> >>> +}
> >>> +
> >>>  #ifdef CONFIG_ACPI
> >>>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
> >>>  {
> >>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
> >>>       data = of_device_get_match_data(dev);
> >>>       smmu->version = data->version;
> >>>       smmu->model = data->model;
> >>> +     smmu->num_clks = data->num_clks;
> >>> +
> >>> +     arm_smmu_fill_clk_data(smmu, data->clks);
> >>>
> >>>       parse_driver_options(smmu);
> >>>
> >>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
> >>>               smmu->irqs[i] = irq;
> >>>       }
> >>>
> >>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
> >>> +     if (err)
> >>> +             return err;
> >>> +
> >>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
> >>> +     if (err)
> >>> +             return err;
> >>> +
> >>>       err = arm_smmu_device_cfg_probe(smmu);
> >>>       if (err)
> >>>               return err;
> >>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
> >>>
> >>>       /* Turn the thing off */
> >>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> >>> +
> >>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
> >>> +
> >>>       return 0;
> >>>  }
> >>>
> >>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
> >>>       return 0;
> >>>  }
> >>>
> >>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
> >>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
> >>> +{
> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> >>> +
> >>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
> >>> +}
> >>> +
> >>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
> >>> +{
> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> >>> +
> >>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
> >>> +
> >>> +     return 0;
> >>> +}
> >>> +
> >>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
> >>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
> >>
> >> This is suspicious.
> >>
> >> If you need a runtime suspend method, why do you think that it is not necessary
> >> to suspend the device during system-wide transitions?
> >
> > Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
> > In that case the clocks have to be enabled in the resume path too.
> >
> > I remember Tomasz pointed to that we shouldn't need clock enable in resume
> > path [1].
> >
> > [1] https://lkml.org/lkml/2018/3/15/60

That was an answer for a different question. I don't remember
suggesting having no suspend function. Although, given the PM
subsystem internals, the suspend function wouldn't be called on SMMU
implementation needed power control (since they would have runtime PM
enabled) and on others, it would be called but do nothing (since no
clocks).

>
> Honestly, I just don't know. :-)
>
> It just looks odd the way it is done.  I think the clock should be
> gated during system-wide suspend too, because the system can spend
> much more time in a sleep state than in the working state, on average.
>
> And note that you cannot rely on runtime PM to always do it for you,
> because it may be disabled at a client device or even blocked by user
> space via power/control in sysfs and that shouldn't matter for
> system-wide PM.

User space blocking runtime PM through sysfs is a good point. I'm not
100% sure how the PM subsystem deals with that in case of system-wide
suspend. I guess for consistency and safety, we should have the
suspend callback.

Best regards,
Tomasz

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Marek Szyprowski]

Hi Tomasz,

On 2018-07-11 14:51, Tomasz Figa wrote:
> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:
>>> On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>>> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>
>>>>> The smmu needs to be functional only when the respective
>>>>> master's using it are active. The device_link feature
>>>>> helps to track such functional dependencies, so that the
>>>>> iommu gets powered when the master device enables itself
>>>>> using pm_runtime. So by adapting the smmu driver for
>>>>> runtime pm, above said dependency can be addressed.
>>>>>
>>>>> This patch adds the pm runtime/sleep callbacks to the
>>>>> driver and also the functions to parse the smmu clocks
>>>>> from DT and enable them in resume/suspend.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>>>> [vivek: Clock rework to request bulk of clocks]
>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>> ---
>>>>>
>>>>>   - No change since v11.
>>>>>
>>>>>   drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>>>>   1 file changed, 58 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>> index f7a96bcf94a6..a01d0dde21dd 100644
>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>> @@ -48,6 +48,7 @@
>>>>>   #include <linux/of_iommu.h>
>>>>>   #include <linux/pci.h>
>>>>>   #include <linux/platform_device.h>
>>>>> +#include <linux/pm_runtime.h>
>>>>>   #include <linux/slab.h>
>>>>>   #include <linux/spinlock.h>
>>>>>
>>>>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>>>>        u32                             num_global_irqs;
>>>>>        u32                             num_context_irqs;
>>>>>        unsigned int                    *irqs;
>>>>> +     struct clk_bulk_data            *clks;
>>>>> +     int                             num_clks;
>>>>>
>>>>>        u32                             cavium_id_base; /* Specific to Cavium */
>>>>>
>>>>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>>>>   struct arm_smmu_match_data {
>>>>>        enum arm_smmu_arch_version version;
>>>>>        enum arm_smmu_implementation model;
>>>>> +     const char * const *clks;
>>>>> +     int num_clks;
>>>>>   };
>>>>>
>>>>>   #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>>>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>
>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>>>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>>>>   };
>>>>>   MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>>>>
>>>>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>>>> +                                const char * const *clks)
>>>>> +{
>>>>> +     int i;
>>>>> +
>>>>> +     if (smmu->num_clks < 1)
>>>>> +             return;
>>>>> +
>>>>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>>>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>>>> +     if (!smmu->clks)
>>>>> +             return;
>>>>> +
>>>>> +     for (i = 0; i < smmu->num_clks; i++)
>>>>> +             smmu->clks[i].id = clks[i];
>>>>> +}
>>>>> +
>>>>>   #ifdef CONFIG_ACPI
>>>>>   static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>>>>   {
>>>>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>>>>        data = of_device_get_match_data(dev);
>>>>>        smmu->version = data->version;
>>>>>        smmu->model = data->model;
>>>>> +     smmu->num_clks = data->num_clks;
>>>>> +
>>>>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>>>>
>>>>>        parse_driver_options(smmu);
>>>>>
>>>>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>>                smmu->irqs[i] = irq;
>>>>>        }
>>>>>
>>>>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>>>> +     if (err)
>>>>> +             return err;
>>>>> +
>>>>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>>>> +     if (err)
>>>>> +             return err;
>>>>> +
>>>>>        err = arm_smmu_device_cfg_probe(smmu);
>>>>>        if (err)
>>>>>                return err;
>>>>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>>>>
>>>>>        /* Turn the thing off */
>>>>>        writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>>>> +
>>>>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>>>> +
>>>>>        return 0;
>>>>>   }
>>>>>
>>>>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>>>>        return 0;
>>>>>   }
>>>>>
>>>>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>>>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>>>> +{
>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>> +
>>>>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>>>> +}
>>>>> +
>>>>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>>>> +{
>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>> +
>>>>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>>> +
>>>>> +     return 0;
>>>>> +}
>>>>> +
>>>>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>>>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>>> This is suspicious.
>>>>
>>>> If you need a runtime suspend method, why do you think that it is not necessary
>>>> to suspend the device during system-wide transitions?
>>> Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>>> In that case the clocks have to be enabled in the resume path too.
>>>
>>> I remember Tomasz pointed to that we shouldn't need clock enable in resume
>>> path [1].
>>>
>>> [1] https://lkml.org/lkml/2018/3/15/60
> That was an answer for a different question. I don't remember
> suggesting having no suspend function. Although, given the PM
> subsystem internals, the suspend function wouldn't be called on SMMU
> implementation needed power control (since they would have runtime PM
> enabled) and on others, it would be called but do nothing (since no
> clocks).
>
>> Honestly, I just don't know. :-)
>>
>> It just looks odd the way it is done.  I think the clock should be
>> gated during system-wide suspend too, because the system can spend
>> much more time in a sleep state than in the working state, on average.
>>
>> And note that you cannot rely on runtime PM to always do it for you,
>> because it may be disabled at a client device or even blocked by user
>> space via power/control in sysfs and that shouldn't matter for
>> system-wide PM.
> User space blocking runtime PM through sysfs is a good point. I'm not
> 100% sure how the PM subsystem deals with that in case of system-wide
> suspend. I guess for consistency and safety, we should have the
> suspend callback.

Frankly, if there are no other reasons I suggest to wire system
suspend/resume to pm_runtime_force_* helpers:
SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
                         pm_runtime_force_resume).

This way you will have everything related to suspending and resuming in
one place and you would not need to bother about all possible cases (like
suspending from runtime pm active and suspending from runtime pm suspended
cases as well as restoring proper device state on resume). This is
especially important in recent kernel releases, where devices are
system-suspended regardless their runtime pm states (in older kernels
devices were first runtime resumed for system suspend, what made code
simpler, but wasn't best from power consumption perspective).

If you go this way, You only need to ensure that runtime resume will also
restore proper device state besides enabling all the clocks. This will
also prepare your driver to properly operate inside power domain, where it
is possible for device to loose its internal state after runtime suspend
when respective power domain has been turned off.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

On Wed, Jul 11, 2018 at 3:40 PM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Tomasz,
>
> On 2018-07-11 14:51, Tomasz Figa wrote:
>> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>>> <vivek.gautam@codeaurora.org> wrote:
>>>> On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>>>> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>>
>>>>>> The smmu needs to be functional only when the respective
>>>>>> master's using it are active. The device_link feature
>>>>>> helps to track such functional dependencies, so that the
>>>>>> iommu gets powered when the master device enables itself
>>>>>> using pm_runtime. So by adapting the smmu driver for
>>>>>> runtime pm, above said dependency can be addressed.
>>>>>>
>>>>>> This patch adds the pm runtime/sleep callbacks to the
>>>>>> driver and also the functions to parse the smmu clocks
>>>>>> from DT and enable them in resume/suspend.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>>>>> [vivek: Clock rework to request bulk of clocks]
>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>>> ---
>>>>>>
>>>>>>   - No change since v11.
>>>>>>
>>>>>>   drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>>>>>   1 file changed, 58 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>>> index f7a96bcf94a6..a01d0dde21dd 100644
>>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>>> @@ -48,6 +48,7 @@
>>>>>>   #include <linux/of_iommu.h>
>>>>>>   #include <linux/pci.h>
>>>>>>   #include <linux/platform_device.h>
>>>>>> +#include <linux/pm_runtime.h>
>>>>>>   #include <linux/slab.h>
>>>>>>   #include <linux/spinlock.h>
>>>>>>
>>>>>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>>>>>        u32                             num_global_irqs;
>>>>>>        u32                             num_context_irqs;
>>>>>>        unsigned int                    *irqs;
>>>>>> +     struct clk_bulk_data            *clks;
>>>>>> +     int                             num_clks;
>>>>>>
>>>>>>        u32                             cavium_id_base; /* Specific to Cavium */
>>>>>>
>>>>>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>>>>>   struct arm_smmu_match_data {
>>>>>>        enum arm_smmu_arch_version version;
>>>>>>        enum arm_smmu_implementation model;
>>>>>> +     const char * const *clks;
>>>>>> +     int num_clks;
>>>>>>   };
>>>>>>
>>>>>>   #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>>>>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>>
>>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>>>>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>>>>>   };
>>>>>>   MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>>>>>
>>>>>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>>>>> +                                const char * const *clks)
>>>>>> +{
>>>>>> +     int i;
>>>>>> +
>>>>>> +     if (smmu->num_clks < 1)
>>>>>> +             return;
>>>>>> +
>>>>>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>>>>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>>>>> +     if (!smmu->clks)
>>>>>> +             return;
>>>>>> +
>>>>>> +     for (i = 0; i < smmu->num_clks; i++)
>>>>>> +             smmu->clks[i].id = clks[i];
>>>>>> +}
>>>>>> +
>>>>>>   #ifdef CONFIG_ACPI
>>>>>>   static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>>>>>   {
>>>>>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>>>>>        data = of_device_get_match_data(dev);
>>>>>>        smmu->version = data->version;
>>>>>>        smmu->model = data->model;
>>>>>> +     smmu->num_clks = data->num_clks;
>>>>>> +
>>>>>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>>>>>
>>>>>>        parse_driver_options(smmu);
>>>>>>
>>>>>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>>>                smmu->irqs[i] = irq;
>>>>>>        }
>>>>>>
>>>>>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>>>>> +     if (err)
>>>>>> +             return err;
>>>>>> +
>>>>>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>>>>> +     if (err)
>>>>>> +             return err;
>>>>>> +
>>>>>>        err = arm_smmu_device_cfg_probe(smmu);
>>>>>>        if (err)
>>>>>>                return err;
>>>>>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>>>>>
>>>>>>        /* Turn the thing off */
>>>>>>        writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>>>>> +
>>>>>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>>>>> +
>>>>>>        return 0;
>>>>>>   }
>>>>>>
>>>>>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>>>>>        return 0;
>>>>>>   }
>>>>>>
>>>>>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>>>>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>>>>> +{
>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>> +
>>>>>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>>>>> +}
>>>>>> +
>>>>>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>>>>> +{
>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>> +
>>>>>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>>>> +
>>>>>> +     return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>>>>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>>>> This is suspicious.
>>>>>
>>>>> If you need a runtime suspend method, why do you think that it is not necessary
>>>>> to suspend the device during system-wide transitions?
>>>> Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>>>> In that case the clocks have to be enabled in the resume path too.
>>>>
>>>> I remember Tomasz pointed to that we shouldn't need clock enable in resume
>>>> path [1].
>>>>
>>>> [1] https://lkml.org/lkml/2018/3/15/60
>> That was an answer for a different question. I don't remember
>> suggesting having no suspend function. Although, given the PM
>> subsystem internals, the suspend function wouldn't be called on SMMU
>> implementation needed power control (since they would have runtime PM
>> enabled) and on others, it would be called but do nothing (since no
>> clocks).
>>
>>> Honestly, I just don't know. :-)
>>>
>>> It just looks odd the way it is done.  I think the clock should be
>>> gated during system-wide suspend too, because the system can spend
>>> much more time in a sleep state than in the working state, on average.
>>>
>>> And note that you cannot rely on runtime PM to always do it for you,
>>> because it may be disabled at a client device or even blocked by user
>>> space via power/control in sysfs and that shouldn't matter for
>>> system-wide PM.
>> User space blocking runtime PM through sysfs is a good point. I'm not
>> 100% sure how the PM subsystem deals with that in case of system-wide
>> suspend. I guess for consistency and safety, we should have the
>> suspend callback.
>
> Frankly, if there are no other reasons I suggest to wire system
> suspend/resume to pm_runtime_force_* helpers:
> SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>                          pm_runtime_force_resume).

Not a good idea at all IMO.

Use PM driver flags rather I'd say.

> This way you will have everything related to suspending and resuming in
> one place and you would not need to bother about all possible cases (like
> suspending from runtime pm active and suspending from runtime pm suspended
> cases as well as restoring proper device state on resume). This is
> especially important in recent kernel releases, where devices are
> system-suspended regardless their runtime pm states (in older kernels
> devices were first runtime resumed for system suspend, what made code
> simpler, but wasn't best from power consumption perspective).
>
> If you go this way, You only need to ensure that runtime resume will also
> restore proper device state besides enabling all the clocks. This will
> also prepare your driver to properly operate inside power domain, where it
> is possible for device to loose its internal state after runtime suspend
> when respective power domain has been turned off.

I'm not sure if you are aware of the pm_runtime_force_* limitations, though.

Thanks,
Rafael

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Marek Szyprowski]

Hi Rafael,

On 2018-07-11 22:36, Rafael J. Wysocki wrote:
> On Wed, Jul 11, 2018 at 3:40 PM, Marek Szyprowski
> <m.szyprowski@samsung.com> wrote:
>> Hi Tomasz,
>>
>> On 2018-07-11 14:51, Tomasz Figa wrote:
>>> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>>>> <vivek.gautam@codeaurora.org> wrote:
>>>>> On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>>>>> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> The smmu needs to be functional only when the respective
>>>>>>> master's using it are active. The device_link feature
>>>>>>> helps to track such functional dependencies, so that the
>>>>>>> iommu gets powered when the master device enables itself
>>>>>>> using pm_runtime. So by adapting the smmu driver for
>>>>>>> runtime pm, above said dependency can be addressed.
>>>>>>>
>>>>>>> This patch adds the pm runtime/sleep callbacks to the
>>>>>>> driver and also the functions to parse the smmu clocks
>>>>>>> from DT and enable them in resume/suspend.
>>>>>>>
>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>>>>>> [vivek: Clock rework to request bulk of clocks]
>>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>>>> ---
>>>>>>>
>>>>>>>    - No change since v11.
>>>>>>>
>>>>>>>    drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>>>>>>    1 file changed, 58 insertions(+), 2 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>>>> index f7a96bcf94a6..a01d0dde21dd 100644
>>>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>>>> @@ -48,6 +48,7 @@
>>>>>>>    #include <linux/of_iommu.h>
>>>>>>>    #include <linux/pci.h>
>>>>>>>    #include <linux/platform_device.h>
>>>>>>> +#include <linux/pm_runtime.h>
>>>>>>>    #include <linux/slab.h>
>>>>>>>    #include <linux/spinlock.h>
>>>>>>>
>>>>>>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>>>>>>         u32                             num_global_irqs;
>>>>>>>         u32                             num_context_irqs;
>>>>>>>         unsigned int                    *irqs;
>>>>>>> +     struct clk_bulk_data            *clks;
>>>>>>> +     int                             num_clks;
>>>>>>>
>>>>>>>         u32                             cavium_id_base; /* Specific to Cavium */
>>>>>>>
>>>>>>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>>>>>>    struct arm_smmu_match_data {
>>>>>>>         enum arm_smmu_arch_version version;
>>>>>>>         enum arm_smmu_implementation model;
>>>>>>> +     const char * const *clks;
>>>>>>> +     int num_clks;
>>>>>>>    };
>>>>>>>
>>>>>>>    #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>>>>>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>>>
>>>>>>>    ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>>>>>>    ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>>>>>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>>>>>>    };
>>>>>>>    MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>>>>>>
>>>>>>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>>>>>> +                                const char * const *clks)
>>>>>>> +{
>>>>>>> +     int i;
>>>>>>> +
>>>>>>> +     if (smmu->num_clks < 1)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>>>>>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>>>>>> +     if (!smmu->clks)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     for (i = 0; i < smmu->num_clks; i++)
>>>>>>> +             smmu->clks[i].id = clks[i];
>>>>>>> +}
>>>>>>> +
>>>>>>>    #ifdef CONFIG_ACPI
>>>>>>>    static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>>>>>>    {
>>>>>>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>>>>>>         data = of_device_get_match_data(dev);
>>>>>>>         smmu->version = data->version;
>>>>>>>         smmu->model = data->model;
>>>>>>> +     smmu->num_clks = data->num_clks;
>>>>>>> +
>>>>>>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>>>>>>
>>>>>>>         parse_driver_options(smmu);
>>>>>>>
>>>>>>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>>>>                 smmu->irqs[i] = irq;
>>>>>>>         }
>>>>>>>
>>>>>>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>>>>>> +     if (err)
>>>>>>> +             return err;
>>>>>>> +
>>>>>>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>>>>>> +     if (err)
>>>>>>> +             return err;
>>>>>>> +
>>>>>>>         err = arm_smmu_device_cfg_probe(smmu);
>>>>>>>         if (err)
>>>>>>>                 return err;
>>>>>>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>>>>>>
>>>>>>>         /* Turn the thing off */
>>>>>>>         writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>>>>>> +
>>>>>>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>>>>>> +
>>>>>>>         return 0;
>>>>>>>    }
>>>>>>>
>>>>>>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>>>>>>         return 0;
>>>>>>>    }
>>>>>>>
>>>>>>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>>>>>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>>>>>> +{
>>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>>> +
>>>>>>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>>>>>> +{
>>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>>> +
>>>>>>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>>>>> +
>>>>>>> +     return 0;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>>>>>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>>>>> This is suspicious.
>>>>>>
>>>>>> If you need a runtime suspend method, why do you think that it is not necessary
>>>>>> to suspend the device during system-wide transitions?
>>>>> Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>>>>> In that case the clocks have to be enabled in the resume path too.
>>>>>
>>>>> I remember Tomasz pointed to that we shouldn't need clock enable in resume
>>>>> path [1].
>>>>>
>>>>> [1] https://lkml.org/lkml/2018/3/15/60
>>> That was an answer for a different question. I don't remember
>>> suggesting having no suspend function. Although, given the PM
>>> subsystem internals, the suspend function wouldn't be called on SMMU
>>> implementation needed power control (since they would have runtime PM
>>> enabled) and on others, it would be called but do nothing (since no
>>> clocks).
>>>
>>>> Honestly, I just don't know. :-)
>>>>
>>>> It just looks odd the way it is done.  I think the clock should be
>>>> gated during system-wide suspend too, because the system can spend
>>>> much more time in a sleep state than in the working state, on average.
>>>>
>>>> And note that you cannot rely on runtime PM to always do it for you,
>>>> because it may be disabled at a client device or even blocked by user
>>>> space via power/control in sysfs and that shouldn't matter for
>>>> system-wide PM.
>>> User space blocking runtime PM through sysfs is a good point. I'm not
>>> 100% sure how the PM subsystem deals with that in case of system-wide
>>> suspend. I guess for consistency and safety, we should have the
>>> suspend callback.
>> Frankly, if there are no other reasons I suggest to wire system
>> suspend/resume to pm_runtime_force_* helpers:
>> SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>                           pm_runtime_force_resume).
> Not a good idea at all IMO.
>
> Use PM driver flags rather I'd say.

Frankly, till now I wasn't aware of the DPM_FLAG_* in struct dev_pm_info
'driver_flags'. I've briefly checked them but I don't see the equivalent
of using SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
pm_runtime_force_resume): keep device suspend if it was runtime suspended
AND really call pm_runtime_suspend if it was not runtime suspended on
system suspend.

>> This way you will have everything related to suspending and resuming in
>> one place and you would not need to bother about all possible cases (like
>> suspending from runtime pm active and suspending from runtime pm suspended
>> cases as well as restoring proper device state on resume). This is
>> especially important in recent kernel releases, where devices are
>> system-suspended regardless their runtime pm states (in older kernels
>> devices were first runtime resumed for system suspend, what made code
>> simpler, but wasn't best from power consumption perspective).
>>
>> If you go this way, You only need to ensure that runtime resume will also
>> restore proper device state besides enabling all the clocks. This will
>> also prepare your driver to properly operate inside power domain, where it
>> is possible for device to loose its internal state after runtime suspend
>> when respective power domain has been turned off.
> I'm not sure if you are aware of the pm_runtime_force_* limitations, though.

What are those limitations?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

On Mon, Jul 23, 2018 at 7:59 AM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Rafael,
>
> On 2018-07-11 22:36, Rafael J. Wysocki wrote:
>> On Wed, Jul 11, 2018 at 3:40 PM, Marek Szyprowski
>> <m.szyprowski@samsung.com> wrote:

[cut]

>>> Frankly, if there are no other reasons I suggest to wire system
>>> suspend/resume to pm_runtime_force_* helpers:
>>> SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>>                           pm_runtime_force_resume).
>> Not a good idea at all IMO.
>>
>> Use PM driver flags rather I'd say.
>
> Frankly, till now I wasn't aware of the DPM_FLAG_* in struct dev_pm_info
> 'driver_flags'.

They are a relatively recent addition.

> I've briefly checked them but I don't see the equivalent
> of using SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> pm_runtime_force_resume): keep device suspend if it was runtime suspended
> AND really call pm_runtime_suspend if it was not runtime suspended on
> system suspend.

DPM_FLAG_SMART_SUSPEND is expected to cause that to happen.  If you
want the device to remain in suspend after the system-wide resume from
sleep (if possible), you can set DPM_FLAG_LEAVE_SUSPENDED for it too.

Currently a caveat is that genpd still doesn't support the flags.  I
have patches for that, but I haven't got to posting them yet.  If you
are interested, I can push this work forward relatively quickly, so
please let me know.

>>> This way you will have everything related to suspending and resuming in
>>> one place and you would not need to bother about all possible cases (like
>>> suspending from runtime pm active and suspending from runtime pm suspended
>>> cases as well as restoring proper device state on resume). This is
>>> especially important in recent kernel releases, where devices are
>>> system-suspended regardless their runtime pm states (in older kernels
>>> devices were first runtime resumed for system suspend, what made code
>>> simpler, but wasn't best from power consumption perspective).
>>>
>>> If you go this way, You only need to ensure that runtime resume will also
>>> restore proper device state besides enabling all the clocks. This will
>>> also prepare your driver to properly operate inside power domain, where it
>>> is possible for device to loose its internal state after runtime suspend
>>> when respective power domain has been turned off.
>> I'm not sure if you are aware of the pm_runtime_force_* limitations, though.
>
> What are those limitations?

First off, they don't work with middle-layer code implementing its own
PM callbacks that actually operate on devices (like the PCI bus type
or the generic ACPI PM domain).  This means that drivers using them
may not work on systems with ACPI, for example.

Second, pm_runtime_force_resume() will always try to leave the device
in suspend during system-wide resume from sleep which may not be
desirable.

Finally, they expect the runtime PM status to be updated by
system-wide PM callbacks of devices below the one using them (eg. its
children and their children etc) which generally is not required and
may not take place unless the drivers of those devices use
pm_runtime_force_* themselves.

So careful there.

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Vivek Gautam]

Hi,


On Wed, Jul 11, 2018 at 6:21 PM, Tomasz Figa <tfiga@chromium.org> wrote:
> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>
>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:
>> > Hi Rafael,
>> >
>> >
>> > On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> >> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>> >>> From: Sricharan R <sricharan@codeaurora.org>
>> >>>
>> >>> The smmu needs to be functional only when the respective
>> >>> master's using it are active. The device_link feature
>> >>> helps to track such functional dependencies, so that the
>> >>> iommu gets powered when the master device enables itself
>> >>> using pm_runtime. So by adapting the smmu driver for
>> >>> runtime pm, above said dependency can be addressed.
>> >>>
>> >>> This patch adds the pm runtime/sleep callbacks to the
>> >>> driver and also the functions to parse the smmu clocks
>> >>> from DT and enable them in resume/suspend.
>> >>>
>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> >>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>> >>> [vivek: Clock rework to request bulk of clocks]
>> >>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>> >>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>> >>> ---
>> >>>
>> >>>  - No change since v11.
>> >>>
>> >>>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>> >>>  1 file changed, 58 insertions(+), 2 deletions(-)
>> >>>
>> >>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> >>> index f7a96bcf94a6..a01d0dde21dd 100644
>> >>> --- a/drivers/iommu/arm-smmu.c
>> >>> +++ b/drivers/iommu/arm-smmu.c
>> >>> @@ -48,6 +48,7 @@
>> >>>  #include <linux/of_iommu.h>
>> >>>  #include <linux/pci.h>
>> >>>  #include <linux/platform_device.h>
>> >>> +#include <linux/pm_runtime.h>
>> >>>  #include <linux/slab.h>
>> >>>  #include <linux/spinlock.h>
>> >>>
>> >>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>> >>>       u32                             num_global_irqs;
>> >>>       u32                             num_context_irqs;
>> >>>       unsigned int                    *irqs;
>> >>> +     struct clk_bulk_data            *clks;
>> >>> +     int                             num_clks;
>> >>>
>> >>>       u32                             cavium_id_base; /* Specific to Cavium */
>> >>>
>> >>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>> >>>  struct arm_smmu_match_data {
>> >>>       enum arm_smmu_arch_version version;
>> >>>       enum arm_smmu_implementation model;
>> >>> +     const char * const *clks;
>> >>> +     int num_clks;
>> >>>  };
>> >>>
>> >>>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>> >>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>> >>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>> >>>
>> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>> >>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>> >>>  };
>> >>>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>> >>>
>> >>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>> >>> +                                const char * const *clks)
>> >>> +{
>> >>> +     int i;
>> >>> +
>> >>> +     if (smmu->num_clks < 1)
>> >>> +             return;
>> >>> +
>> >>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>> >>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>> >>> +     if (!smmu->clks)
>> >>> +             return;
>> >>> +
>> >>> +     for (i = 0; i < smmu->num_clks; i++)
>> >>> +             smmu->clks[i].id = clks[i];
>> >>> +}
>> >>> +
>> >>>  #ifdef CONFIG_ACPI
>> >>>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>> >>>  {
>> >>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>> >>>       data = of_device_get_match_data(dev);
>> >>>       smmu->version = data->version;
>> >>>       smmu->model = data->model;
>> >>> +     smmu->num_clks = data->num_clks;
>> >>> +
>> >>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>> >>>
>> >>>       parse_driver_options(smmu);
>> >>>
>> >>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>> >>>               smmu->irqs[i] = irq;
>> >>>       }
>> >>>
>> >>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>> >>> +     if (err)
>> >>> +             return err;
>> >>> +
>> >>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>> >>> +     if (err)
>> >>> +             return err;
>> >>> +
>> >>>       err = arm_smmu_device_cfg_probe(smmu);
>> >>>       if (err)
>> >>>               return err;
>> >>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>> >>>
>> >>>       /* Turn the thing off */
>> >>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>> >>> +
>> >>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>> >>> +
>> >>>       return 0;
>> >>>  }
>> >>>
>> >>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>> >>>       return 0;
>> >>>  }
>> >>>
>> >>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>> >>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>> >>> +{
>> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>> >>> +
>> >>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>> >>> +}
>> >>> +
>> >>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>> >>> +{
>> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>> >>> +
>> >>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>> >>> +
>> >>> +     return 0;
>> >>> +}
>> >>> +
>> >>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>> >>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>> >>
>> >> This is suspicious.
>> >>
>> >> If you need a runtime suspend method, why do you think that it is not necessary
>> >> to suspend the device during system-wide transitions?
>> >
>> > Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>> > In that case the clocks have to be enabled in the resume path too.
>> >
>> > I remember Tomasz pointed to that we shouldn't need clock enable in resume
>> > path [1].
>> >
>> > [1] https://lkml.org/lkml/2018/3/15/60
>
> That was an answer for a different question. I don't remember
> suggesting having no suspend function.

My bad, apologies. You are right, we were discussing if we need any additional
handling of power for arm_smmu_device_reset() in arm_smmu_pm_resume().

> Although, given the PM
> subsystem internals, the suspend function wouldn't be called on SMMU
> implementation needed power control (since they would have runtime PM
> enabled) and on others, it would be called but do nothing (since no
> clocks).
>
>>
>> Honestly, I just don't know. :-)
>>
>> It just looks odd the way it is done.  I think the clock should be
>> gated during system-wide suspend too, because the system can spend
>> much more time in a sleep state than in the working state, on average.
>>
>> And note that you cannot rely on runtime PM to always do it for you,
>> because it may be disabled at a client device or even blocked by user
>> space via power/control in sysfs and that shouldn't matter for
>> system-wide PM.
>
> User space blocking runtime PM through sysfs is a good point. I'm not
> 100% sure how the PM subsystem deals with that in case of system-wide
> suspend. I guess for consistency and safety, we should have the
> suspend callback.

Will add the following suspend callback (same as arm_smmu_runtime_suspend):

 static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
 {
         struct arm_smmu_device *smmu = dev_get_drvdata(dev);

         clk_bulk_disable(smmu->num_clks, smmu->clks);

         return 0;
 }


Best regards
Vivek

>
> Best regards,
> Tomasz
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu



-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

On Thu, Jul 12, 2018 at 12:57 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> Hi,
>
>
> On Wed, Jul 11, 2018 at 6:21 PM, Tomasz Figa <tfiga@chromium.org> wrote:
>> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>>
>>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>>> <vivek.gautam@codeaurora.org> wrote:
>>> > Hi Rafael,
>>> >
>>> >
>>> > On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>> >> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>> >>> From: Sricharan R <sricharan@codeaurora.org>
>>> >>>
>>> >>> The smmu needs to be functional only when the respective
>>> >>> master's using it are active. The device_link feature
>>> >>> helps to track such functional dependencies, so that the
>>> >>> iommu gets powered when the master device enables itself
>>> >>> using pm_runtime. So by adapting the smmu driver for
>>> >>> runtime pm, above said dependency can be addressed.
>>> >>>
>>> >>> This patch adds the pm runtime/sleep callbacks to the
>>> >>> driver and also the functions to parse the smmu clocks
>>> >>> from DT and enable them in resume/suspend.
>>> >>>
>>> >>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> >>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>> >>> [vivek: Clock rework to request bulk of clocks]
>>> >>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>> >>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>> >>> ---
>>> >>>
>>> >>>  - No change since v11.
>>> >>>
>>> >>>  drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>> >>>  1 file changed, 58 insertions(+), 2 deletions(-)
>>> >>>
>>> >>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> >>> index f7a96bcf94a6..a01d0dde21dd 100644
>>> >>> --- a/drivers/iommu/arm-smmu.c
>>> >>> +++ b/drivers/iommu/arm-smmu.c
>>> >>> @@ -48,6 +48,7 @@
>>> >>>  #include <linux/of_iommu.h>
>>> >>>  #include <linux/pci.h>
>>> >>>  #include <linux/platform_device.h>
>>> >>> +#include <linux/pm_runtime.h>
>>> >>>  #include <linux/slab.h>
>>> >>>  #include <linux/spinlock.h>
>>> >>>
>>> >>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>> >>>       u32                             num_global_irqs;
>>> >>>       u32                             num_context_irqs;
>>> >>>       unsigned int                    *irqs;
>>> >>> +     struct clk_bulk_data            *clks;
>>> >>> +     int                             num_clks;
>>> >>>
>>> >>>       u32                             cavium_id_base; /* Specific to Cavium */
>>> >>>
>>> >>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>> >>>  struct arm_smmu_match_data {
>>> >>>       enum arm_smmu_arch_version version;
>>> >>>       enum arm_smmu_implementation model;
>>> >>> +     const char * const *clks;
>>> >>> +     int num_clks;
>>> >>>  };
>>> >>>
>>> >>>  #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>> >>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>> >>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>> >>>
>>> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>> >>>  ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>> >>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>> >>>  };
>>> >>>  MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>> >>>
>>> >>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>> >>> +                                const char * const *clks)
>>> >>> +{
>>> >>> +     int i;
>>> >>> +
>>> >>> +     if (smmu->num_clks < 1)
>>> >>> +             return;
>>> >>> +
>>> >>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>> >>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>> >>> +     if (!smmu->clks)
>>> >>> +             return;
>>> >>> +
>>> >>> +     for (i = 0; i < smmu->num_clks; i++)
>>> >>> +             smmu->clks[i].id = clks[i];
>>> >>> +}
>>> >>> +
>>> >>>  #ifdef CONFIG_ACPI
>>> >>>  static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>> >>>  {
>>> >>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>> >>>       data = of_device_get_match_data(dev);
>>> >>>       smmu->version = data->version;
>>> >>>       smmu->model = data->model;
>>> >>> +     smmu->num_clks = data->num_clks;
>>> >>> +
>>> >>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>> >>>
>>> >>>       parse_driver_options(smmu);
>>> >>>
>>> >>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>> >>>               smmu->irqs[i] = irq;
>>> >>>       }
>>> >>>
>>> >>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>> >>> +     if (err)
>>> >>> +             return err;
>>> >>> +
>>> >>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>> >>> +     if (err)
>>> >>> +             return err;
>>> >>> +
>>> >>>       err = arm_smmu_device_cfg_probe(smmu);
>>> >>>       if (err)
>>> >>>               return err;
>>> >>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>> >>>
>>> >>>       /* Turn the thing off */
>>> >>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>> >>> +
>>> >>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>> >>> +
>>> >>>       return 0;
>>> >>>  }
>>> >>>
>>> >>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>> >>>       return 0;
>>> >>>  }
>>> >>>
>>> >>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>> >>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>> >>> +{
>>> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>> >>> +
>>> >>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>> >>> +}
>>> >>> +
>>> >>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>> >>> +{
>>> >>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>> >>> +
>>> >>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>> >>> +
>>> >>> +     return 0;
>>> >>> +}
>>> >>> +
>>> >>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>> >>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>> >>
>>> >> This is suspicious.
>>> >>
>>> >> If you need a runtime suspend method, why do you think that it is not necessary
>>> >> to suspend the device during system-wide transitions?
>>> >
>>> > Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>>> > In that case the clocks have to be enabled in the resume path too.
>>> >
>>> > I remember Tomasz pointed to that we shouldn't need clock enable in resume
>>> > path [1].
>>> >
>>> > [1] https://lkml.org/lkml/2018/3/15/60
>>
>> That was an answer for a different question. I don't remember
>> suggesting having no suspend function.
>
> My bad, apologies. You are right, we were discussing if we need any additional
> handling of power for arm_smmu_device_reset() in arm_smmu_pm_resume().
>
>> Although, given the PM
>> subsystem internals, the suspend function wouldn't be called on SMMU
>> implementation needed power control (since they would have runtime PM
>> enabled) and on others, it would be called but do nothing (since no
>> clocks).
>>
>>>
>>> Honestly, I just don't know. :-)
>>>
>>> It just looks odd the way it is done.  I think the clock should be
>>> gated during system-wide suspend too, because the system can spend
>>> much more time in a sleep state than in the working state, on average.
>>>
>>> And note that you cannot rely on runtime PM to always do it for you,
>>> because it may be disabled at a client device or even blocked by user
>>> space via power/control in sysfs and that shouldn't matter for
>>> system-wide PM.
>>
>> User space blocking runtime PM through sysfs is a good point. I'm not
>> 100% sure how the PM subsystem deals with that in case of system-wide
>> suspend. I guess for consistency and safety, we should have the
>> suspend callback.
>
> Will add the following suspend callback (same as arm_smmu_runtime_suspend):
>
>  static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
>  {
>          struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>
>          clk_bulk_disable(smmu->num_clks, smmu->clks);
>
>          return 0;
>  }

I think you also need to check if the clock has already been disabled
by runtime PM.  Otherwise you may end up disabling it twice in a row.

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Vivek Gautam]

HI Rafael,


On 7/16/2018 2:21 PM, Rafael J. Wysocki wrote:
> On Thu, Jul 12, 2018 at 12:57 PM, Vivek Gautam
> <vivek.gautam@codeaurora.org> wrote:
>> Hi,
>>
>>
>> On Wed, Jul 11, 2018 at 6:21 PM, Tomasz Figa <tfiga@chromium.org> wrote:
>>> On Wed, Jul 11, 2018 at 8:11 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>>> On Wed, Jul 11, 2018 at 12:55 PM, Vivek Gautam
>>>> <vivek.gautam@codeaurora.org> wrote:
>>>>> Hi Rafael,
>>>>>
>>>>>
>>>>> On Wed, Jul 11, 2018 at 3:20 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>>>>> On Sunday, July 8, 2018 7:34:10 PM CEST Vivek Gautam wrote:
>>>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> The smmu needs to be functional only when the respective
>>>>>>> master's using it are active. The device_link feature
>>>>>>> helps to track such functional dependencies, so that the
>>>>>>> iommu gets powered when the master device enables itself
>>>>>>> using pm_runtime. So by adapting the smmu driver for
>>>>>>> runtime pm, above said dependency can be addressed.
>>>>>>>
>>>>>>> This patch adds the pm runtime/sleep callbacks to the
>>>>>>> driver and also the functions to parse the smmu clocks
>>>>>>> from DT and enable them in resume/suspend.
>>>>>>>
>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>> Signed-off-by: Archit Taneja <architt@codeaurora.org>
>>>>>>> [vivek: Clock rework to request bulk of clocks]
>>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>>>> ---
>>>>>>>
>>>>>>>   - No change since v11.
>>>>>>>
>>>>>>>   drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
>>>>>>>   1 file changed, 58 insertions(+), 2 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>>>> index f7a96bcf94a6..a01d0dde21dd 100644
>>>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>>>> @@ -48,6 +48,7 @@
>>>>>>>   #include <linux/of_iommu.h>
>>>>>>>   #include <linux/pci.h>
>>>>>>>   #include <linux/platform_device.h>
>>>>>>> +#include <linux/pm_runtime.h>
>>>>>>>   #include <linux/slab.h>
>>>>>>>   #include <linux/spinlock.h>
>>>>>>>
>>>>>>> @@ -205,6 +206,8 @@ struct arm_smmu_device {
>>>>>>>        u32                             num_global_irqs;
>>>>>>>        u32                             num_context_irqs;
>>>>>>>        unsigned int                    *irqs;
>>>>>>> +     struct clk_bulk_data            *clks;
>>>>>>> +     int                             num_clks;
>>>>>>>
>>>>>>>        u32                             cavium_id_base; /* Specific to Cavium */
>>>>>>>
>>>>>>> @@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>>>>>>>   struct arm_smmu_match_data {
>>>>>>>        enum arm_smmu_arch_version version;
>>>>>>>        enum arm_smmu_implementation model;
>>>>>>> +     const char * const *clks;
>>>>>>> +     int num_clks;
>>>>>>>   };
>>>>>>>
>>>>>>>   #define ARM_SMMU_MATCH_DATA(name, ver, imp)  \
>>>>>>> -static struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>>> +static const struct arm_smmu_match_data name = { .version = ver, .model = imp }
>>>>>>>
>>>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
>>>>>>>   ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
>>>>>>> @@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
>>>>>>>   };
>>>>>>>   MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
>>>>>>>
>>>>>>> +static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
>>>>>>> +                                const char * const *clks)
>>>>>>> +{
>>>>>>> +     int i;
>>>>>>> +
>>>>>>> +     if (smmu->num_clks < 1)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
>>>>>>> +                               sizeof(*smmu->clks), GFP_KERNEL);
>>>>>>> +     if (!smmu->clks)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     for (i = 0; i < smmu->num_clks; i++)
>>>>>>> +             smmu->clks[i].id = clks[i];
>>>>>>> +}
>>>>>>> +
>>>>>>>   #ifdef CONFIG_ACPI
>>>>>>>   static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
>>>>>>>   {
>>>>>>> @@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
>>>>>>>        data = of_device_get_match_data(dev);
>>>>>>>        smmu->version = data->version;
>>>>>>>        smmu->model = data->model;
>>>>>>> +     smmu->num_clks = data->num_clks;
>>>>>>> +
>>>>>>> +     arm_smmu_fill_clk_data(smmu, data->clks);
>>>>>>>
>>>>>>>        parse_driver_options(smmu);
>>>>>>>
>>>>>>> @@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>>>>>>                smmu->irqs[i] = irq;
>>>>>>>        }
>>>>>>>
>>>>>>> +     err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>>>>>> +     if (err)
>>>>>>> +             return err;
>>>>>>> +
>>>>>>> +     err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
>>>>>>> +     if (err)
>>>>>>> +             return err;
>>>>>>> +
>>>>>>>        err = arm_smmu_device_cfg_probe(smmu);
>>>>>>>        if (err)
>>>>>>>                return err;
>>>>>>> @@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>>>>>>>
>>>>>>>        /* Turn the thing off */
>>>>>>>        writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>>>>>>> +
>>>>>>> +     clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>>>>>> +
>>>>>>>        return 0;
>>>>>>>   }
>>>>>>>
>>>>>>> @@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
>>>>>>>        return 0;
>>>>>>>   }
>>>>>>>
>>>>>>> -static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
>>>>>>> +static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
>>>>>>> +{
>>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>>> +
>>>>>>> +     return clk_bulk_enable(smmu->num_clks, smmu->clks);
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
>>>>>>> +{
>>>>>>> +     struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>>>>> +
>>>>>>> +     clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>>>>> +
>>>>>>> +     return 0;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static const struct dev_pm_ops arm_smmu_pm_ops = {
>>>>>>> +     SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
>>>>>> This is suspicious.
>>>>>>
>>>>>> If you need a runtime suspend method, why do you think that it is not necessary
>>>>>> to suspend the device during system-wide transitions?
>>>>> Okay, so you suggest to put clock disabling in say arm_smmu_pm_suspend()?
>>>>> In that case the clocks have to be enabled in the resume path too.
>>>>>
>>>>> I remember Tomasz pointed to that we shouldn't need clock enable in resume
>>>>> path [1].
>>>>>
>>>>> [1] https://lkml.org/lkml/2018/3/15/60
>>> That was an answer for a different question. I don't remember
>>> suggesting having no suspend function.
>> My bad, apologies. You are right, we were discussing if we need any additional
>> handling of power for arm_smmu_device_reset() in arm_smmu_pm_resume().
>>
>>> Although, given the PM
>>> subsystem internals, the suspend function wouldn't be called on SMMU
>>> implementation needed power control (since they would have runtime PM
>>> enabled) and on others, it would be called but do nothing (since no
>>> clocks).
>>>
>>>> Honestly, I just don't know. :-)
>>>>
>>>> It just looks odd the way it is done.  I think the clock should be
>>>> gated during system-wide suspend too, because the system can spend
>>>> much more time in a sleep state than in the working state, on average.
>>>>
>>>> And note that you cannot rely on runtime PM to always do it for you,
>>>> because it may be disabled at a client device or even blocked by user
>>>> space via power/control in sysfs and that shouldn't matter for
>>>> system-wide PM.
>>> User space blocking runtime PM through sysfs is a good point. I'm not
>>> 100% sure how the PM subsystem deals with that in case of system-wide
>>> suspend. I guess for consistency and safety, we should have the
>>> suspend callback.
>> Will add the following suspend callback (same as arm_smmu_runtime_suspend):
>>
>>   static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
>>   {
>>           struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>
>>           clk_bulk_disable(smmu->num_clks, smmu->clks);
>>
>>           return 0;
>>   }
> I think you also need to check if the clock has already been disabled
> by runtime PM.  Otherwise you may end up disabling it twice in a row.

Should I rather call a pm_runtime_put() in suspend callback? Or an 
expanded form
something similar to:
https://elixir.bootlin.com/linux/v4.18-rc5/source/drivers/slimbus/qcom-ctrl.c#L695


Best regards
Vivek

Re: [PATCH v12 1/4] iommu/arm-smmu: Add pm_runtime/sleep ops

[Rafael J. Wysocki]

Hi,

On Mon, Jul 16, 2018 at 12:11 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> HI Rafael,
>
>
>
> On 7/16/2018 2:21 PM, Rafael J. Wysocki wrote:
>>
>> On Thu, Jul 12, 2018 at 12:57 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:

[cut]

>>>> Although, given the PM
>>>> subsystem internals, the suspend function wouldn't be called on SMMU
>>>> implementation needed power control (since they would have runtime PM
>>>> enabled) and on others, it would be called but do nothing (since no
>>>> clocks).
>>>>
>>>>> Honestly, I just don't know. :-)
>>>>>
>>>>> It just looks odd the way it is done.  I think the clock should be
>>>>> gated during system-wide suspend too, because the system can spend
>>>>> much more time in a sleep state than in the working state, on average.
>>>>>
>>>>> And note that you cannot rely on runtime PM to always do it for you,
>>>>> because it may be disabled at a client device or even blocked by user
>>>>> space via power/control in sysfs and that shouldn't matter for
>>>>> system-wide PM.
>>>>
>>>> User space blocking runtime PM through sysfs is a good point. I'm not
>>>> 100% sure how the PM subsystem deals with that in case of system-wide
>>>> suspend. I guess for consistency and safety, we should have the
>>>> suspend callback.
>>>
>>> Will add the following suspend callback (same as
>>> arm_smmu_runtime_suspend):
>>>
>>>   static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
>>>   {
>>>           struct arm_smmu_device *smmu = dev_get_drvdata(dev);
>>>
>>>           clk_bulk_disable(smmu->num_clks, smmu->clks);
>>>
>>>           return 0;
>>>   }
>>
>> I think you also need to check if the clock has already been disabled
>> by runtime PM.  Otherwise you may end up disabling it twice in a row.
>
>
> Should I rather call a pm_runtime_put() in suspend callback?

That wouldn't work as runtime PM may be effectively disabled by user
space via sysfs.  That's one of the reasons why you need the extra
system-wide suspend callback in the first place. :-)

> Or an expanded form something similar to:
> https://elixir.bootlin.com/linux/v4.18-rc5/source/drivers/slimbus/qcom-ctrl.c#L695

Yes, you can do something like that, but be careful to make sure that
the state of the device after system-wide resume is consistent with
its runtime PM status in all cases.

[PATCH v12 2/4] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Vivek Gautam]

From: Sricharan R <sricharan@codeaurora.org>

The smmu device probe/remove and add/remove master device callbacks
gets called when the smmu is not linked to its master, that is without
the context of the master device. So calling runtime apis in those places
separately.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
[vivek: Cleanup pm runtime calls]
Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
---

 - Change since v11
   * Replaced pm_runtime_disable() with pm_runtime_force_suspend()
     to avoid warning about " Unpreparing enabled clock".
     Full warning text mentioned in cover patch.

 drivers/iommu/arm-smmu.c | 92 +++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 84 insertions(+), 8 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index a01d0dde21dd..09265e206e2d 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
 	{ 0, NULL},
 };
 
+static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
+{
+	if (pm_runtime_enabled(smmu->dev))
+		return pm_runtime_get_sync(smmu->dev);
+
+	return 0;
+}
+
+static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
+{
+	if (pm_runtime_enabled(smmu->dev))
+		pm_runtime_put(smmu->dev);
+}
+
 static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
 {
 	return container_of(dom, struct arm_smmu_domain, domain);
@@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_device *smmu = smmu_domain->smmu;
 	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
-	int irq;
+	int ret, irq;
 
 	if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
 		return;
 
+	ret = arm_smmu_rpm_get(smmu);
+	if (ret < 0)
+		return;
+
 	/*
 	 * Disable the context bank and free the page tables before freeing
 	 * it.
@@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
 
 	free_io_pgtable_ops(smmu_domain->pgtbl_ops);
 	__arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
+
+	arm_smmu_rpm_put(smmu);
 }
 
 static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
@@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		return -ENODEV;
 
 	smmu = fwspec_smmu(fwspec);
+
+	ret = arm_smmu_rpm_get(smmu);
+	if (ret < 0)
+		return ret;
+
 	/* Ensure that the domain is finalised */
 	ret = arm_smmu_init_domain_context(domain, smmu);
 	if (ret < 0)
-		return ret;
+		goto rpm_put;
 
 	/*
 	 * Sanity check the domain. We don't support domains across
@@ -1226,33 +1251,50 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		dev_err(dev,
 			"cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n",
 			dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev));
-		return -EINVAL;
+		ret = -EINVAL;
+		goto rpm_put;
 	}
 
 	/* Looks ok, so add the device to the domain */
-	return arm_smmu_domain_add_master(smmu_domain, fwspec);
+	ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
+
+rpm_put:
+	arm_smmu_rpm_put(smmu);
+	return ret;
 }
 
 static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
 			phys_addr_t paddr, size_t size, int prot)
 {
 	struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+	struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu;
+	int ret;
 
 	if (!ops)
 		return -ENODEV;
 
-	return ops->map(ops, iova, paddr, size, prot);
+	arm_smmu_rpm_get(smmu);
+	ret = ops->map(ops, iova, paddr, size, prot);
+	arm_smmu_rpm_put(smmu);
+
+	return ret;
 }
 
 static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
 			     size_t size)
 {
 	struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+	struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu;
+	size_t ret;
 
 	if (!ops)
 		return 0;
 
-	return ops->unmap(ops, iova, size);
+	arm_smmu_rpm_get(smmu);
+	ret = ops->unmap(ops, iova, size);
+	arm_smmu_rpm_put(smmu);
+
+	return ret;
 }
 
 static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
@@ -1407,7 +1449,13 @@ static int arm_smmu_add_device(struct device *dev)
 	while (i--)
 		cfg->smendx[i] = INVALID_SMENDX;
 
+	ret = arm_smmu_rpm_get(smmu);
+	if (ret < 0)
+		goto out_cfg_free;
+
 	ret = arm_smmu_master_alloc_smes(dev);
+	arm_smmu_rpm_put(smmu);
+
 	if (ret)
 		goto out_cfg_free;
 
@@ -1427,7 +1475,7 @@ static void arm_smmu_remove_device(struct device *dev)
 	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 	struct arm_smmu_master_cfg *cfg;
 	struct arm_smmu_device *smmu;
-
+	int ret;
 
 	if (!fwspec || fwspec->ops != &arm_smmu_ops)
 		return;
@@ -1435,8 +1483,15 @@ static void arm_smmu_remove_device(struct device *dev)
 	cfg  = fwspec->iommu_priv;
 	smmu = cfg->smmu;
 
+	ret = arm_smmu_rpm_get(smmu);
+	if (ret < 0)
+		return;
+
 	iommu_device_unlink(&smmu->iommu, dev);
 	arm_smmu_master_free_smes(fwspec);
+
+	arm_smmu_rpm_put(smmu);
+
 	iommu_group_remove_device(dev);
 	kfree(fwspec->iommu_priv);
 	iommu_fwspec_free(dev);
@@ -2124,6 +2179,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		smmu->irqs[i] = irq;
 	}
 
+	platform_set_drvdata(pdev, smmu);
+
 	err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
 	if (err)
 		return err;
@@ -2132,6 +2189,19 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	if (err)
 		return err;
 
+	/*
+	 * We want to avoid touching dev->power.lock in fastpaths unless
+	 * it's really going to do something useful - pm_runtime_enabled()
+	 * can serve as an ideal proxy for that decision. So, conditionally
+	 * enable pm_runtime.
+	 */
+	if (dev->pm_domain)
+		pm_runtime_enable(dev);
+
+	err = arm_smmu_rpm_get(smmu);
+	if (err < 0)
+		return err;
+
 	err = arm_smmu_device_cfg_probe(smmu);
 	if (err)
 		return err;
@@ -2173,10 +2243,11 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		return err;
 	}
 
-	platform_set_drvdata(pdev, smmu);
 	arm_smmu_device_reset(smmu);
 	arm_smmu_test_smr_masks(smmu);
 
+	arm_smmu_rpm_put(smmu);
+
 	/*
 	 * For ACPI and generic DT bindings, an SMMU will be probed before
 	 * any device which might need it, so we want the bus ops in place
@@ -2212,8 +2283,13 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
 		dev_err(&pdev->dev, "removing device with active domains!\n");
 
+	arm_smmu_rpm_get(smmu);
 	/* Turn the thing off */
 	writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+	arm_smmu_rpm_put(smmu);
+
+	if (pm_runtime_enabled(smmu->dev))
+		pm_runtime_force_suspend(smmu->dev);
 
 	clk_bulk_unprepare(smmu->num_clks, smmu->clks);
 
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 2/4] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Rafael J. Wysocki]

On Sunday, July 8, 2018 7:34:11 PM CEST Vivek Gautam wrote:
> From: Sricharan R <sricharan@codeaurora.org>
> 
> The smmu device probe/remove and add/remove master device callbacks
> gets called when the smmu is not linked to its master, that is without
> the context of the master device. So calling runtime apis in those places
> separately.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> [vivek: Cleanup pm runtime calls]
> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
> ---
> 
>  - Change since v11
>    * Replaced pm_runtime_disable() with pm_runtime_force_suspend()
>      to avoid warning about " Unpreparing enabled clock".
>      Full warning text mentioned in cover patch.
> 
>  drivers/iommu/arm-smmu.c | 92 +++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 84 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index a01d0dde21dd..09265e206e2d 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>  	{ 0, NULL},
>  };
>  
> +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
> +{
> +	if (pm_runtime_enabled(smmu->dev))

Why do you need the pm_runtime_enabled() checks here and below?

pm_runtime_get_sync() and pm_runtime_put() should work just fine if
runtime PM is not enabled.

> +		return pm_runtime_get_sync(smmu->dev);
> +
> +	return 0;
> +}
> +
> +static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
> +{
> +	if (pm_runtime_enabled(smmu->dev))
> +		pm_runtime_put(smmu->dev);
> +}
> +
>  static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
>  {
>  	return container_of(dom, struct arm_smmu_domain, domain);
> @@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>  	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>  	struct arm_smmu_device *smmu = smmu_domain->smmu;
>  	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> -	int irq;
> +	int ret, irq;
>  
>  	if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
>  		return;
>  
> +	ret = arm_smmu_rpm_get(smmu);
> +	if (ret < 0)
> +		return;
> +
>  	/*
>  	 * Disable the context bank and free the page tables before freeing
>  	 * it.
> @@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>  
>  	free_io_pgtable_ops(smmu_domain->pgtbl_ops);
>  	__arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
> +
> +	arm_smmu_rpm_put(smmu);
>  }
>  
>  static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> @@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
>  		return -ENODEV;
>  
>  	smmu = fwspec_smmu(fwspec);
> +
> +	ret = arm_smmu_rpm_get(smmu);
> +	if (ret < 0)
> +		return ret;
> +
>  	/* Ensure that the domain is finalised */
>  	ret = arm_smmu_init_domain_context(domain, smmu);
>  	if (ret < 0)
> -		return ret;
> +		goto rpm_put;
>  
>  	/*
>  	 * Sanity check the domain. We don't support domains across
> @@ -1226,33 +1251,50 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
>  		dev_err(dev,
>  			"cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n",
>  			dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev));
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto rpm_put;
>  	}
>  
>  	/* Looks ok, so add the device to the domain */
> -	return arm_smmu_domain_add_master(smmu_domain, fwspec);
> +	ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
> +
> +rpm_put:
> +	arm_smmu_rpm_put(smmu);
> +	return ret;
>  }
>  
>  static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
>  			phys_addr_t paddr, size_t size, int prot)
>  {
>  	struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
> +	struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu;
> +	int ret;
>  
>  	if (!ops)
>  		return -ENODEV;
>  
> -	return ops->map(ops, iova, paddr, size, prot);
> +	arm_smmu_rpm_get(smmu);
> +	ret = ops->map(ops, iova, paddr, size, prot);
> +	arm_smmu_rpm_put(smmu);
> +
> +	return ret;
>  }
>  
>  static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
>  			     size_t size)
>  {
>  	struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
> +	struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu;
> +	size_t ret;
>  
>  	if (!ops)
>  		return 0;
>  
> -	return ops->unmap(ops, iova, size);
> +	arm_smmu_rpm_get(smmu);
> +	ret = ops->unmap(ops, iova, size);
> +	arm_smmu_rpm_put(smmu);
> +
> +	return ret;
>  }
>  
>  static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
> @@ -1407,7 +1449,13 @@ static int arm_smmu_add_device(struct device *dev)
>  	while (i--)
>  		cfg->smendx[i] = INVALID_SMENDX;
>  
> +	ret = arm_smmu_rpm_get(smmu);
> +	if (ret < 0)
> +		goto out_cfg_free;
> +
>  	ret = arm_smmu_master_alloc_smes(dev);
> +	arm_smmu_rpm_put(smmu);
> +
>  	if (ret)
>  		goto out_cfg_free;
>  
> @@ -1427,7 +1475,7 @@ static void arm_smmu_remove_device(struct device *dev)
>  	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>  	struct arm_smmu_master_cfg *cfg;
>  	struct arm_smmu_device *smmu;
> -
> +	int ret;
>  
>  	if (!fwspec || fwspec->ops != &arm_smmu_ops)
>  		return;
> @@ -1435,8 +1483,15 @@ static void arm_smmu_remove_device(struct device *dev)
>  	cfg  = fwspec->iommu_priv;
>  	smmu = cfg->smmu;
>  
> +	ret = arm_smmu_rpm_get(smmu);
> +	if (ret < 0)
> +		return;
> +
>  	iommu_device_unlink(&smmu->iommu, dev);
>  	arm_smmu_master_free_smes(fwspec);
> +
> +	arm_smmu_rpm_put(smmu);
> +
>  	iommu_group_remove_device(dev);
>  	kfree(fwspec->iommu_priv);
>  	iommu_fwspec_free(dev);
> @@ -2124,6 +2179,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>  		smmu->irqs[i] = irq;
>  	}
>  
> +	platform_set_drvdata(pdev, smmu);
> +
>  	err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>  	if (err)
>  		return err;
> @@ -2132,6 +2189,19 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>  	if (err)
>  		return err;
>  
> +	/*
> +	 * We want to avoid touching dev->power.lock in fastpaths unless
> +	 * it's really going to do something useful - pm_runtime_enabled()
> +	 * can serve as an ideal proxy for that decision. So, conditionally
> +	 * enable pm_runtime.
> +	 */
> +	if (dev->pm_domain)
> +		pm_runtime_enable(dev);
> +
> +	err = arm_smmu_rpm_get(smmu);
> +	if (err < 0)
> +		return err;
> +
>  	err = arm_smmu_device_cfg_probe(smmu);
>  	if (err)
>  		return err;
> @@ -2173,10 +2243,11 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>  		return err;
>  	}
>  
> -	platform_set_drvdata(pdev, smmu);
>  	arm_smmu_device_reset(smmu);
>  	arm_smmu_test_smr_masks(smmu);
>  
> +	arm_smmu_rpm_put(smmu);
> +
>  	/*
>  	 * For ACPI and generic DT bindings, an SMMU will be probed before
>  	 * any device which might need it, so we want the bus ops in place
> @@ -2212,8 +2283,13 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>  	if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
>  		dev_err(&pdev->dev, "removing device with active domains!\n");
>  
> +	arm_smmu_rpm_get(smmu);
>  	/* Turn the thing off */
>  	writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> +	arm_smmu_rpm_put(smmu);
> +
> +	if (pm_runtime_enabled(smmu->dev))
> +		pm_runtime_force_suspend(smmu->dev);
>  
>  	clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>  
> 

Re: [PATCH v12 2/4] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Tomasz Figa]

Hi Rafael,

Thanks for review.

On Wed, Jul 11, 2018 at 6:53 PM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>
> On Sunday, July 8, 2018 7:34:11 PM CEST Vivek Gautam wrote:
> > From: Sricharan R <sricharan@codeaurora.org>
> >
> > The smmu device probe/remove and add/remove master device callbacks
> > gets called when the smmu is not linked to its master, that is without
> > the context of the master device. So calling runtime apis in those places
> > separately.
> >
> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> > [vivek: Cleanup pm runtime calls]
> > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> > Reviewed-by: Tomasz Figa <tfiga@chromium.org>
> > ---
> >
> >  - Change since v11
> >    * Replaced pm_runtime_disable() with pm_runtime_force_suspend()
> >      to avoid warning about " Unpreparing enabled clock".
> >      Full warning text mentioned in cover patch.
> >
> >  drivers/iommu/arm-smmu.c | 92 +++++++++++++++++++++++++++++++++++++++++++-----
> >  1 file changed, 84 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> > index a01d0dde21dd..09265e206e2d 100644
> > --- a/drivers/iommu/arm-smmu.c
> > +++ b/drivers/iommu/arm-smmu.c
> > @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
> >       { 0, NULL},
> >  };
> >
> > +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
> > +{
> > +     if (pm_runtime_enabled(smmu->dev))
>
> Why do you need the pm_runtime_enabled() checks here and below?
>
> pm_runtime_get_sync() and pm_runtime_put() should work just fine if
> runtime PM is not enabled.

Because pm_runtime_get_sync() acquires a spin lock, even if only for
the short time of checking if runtime PM is enabled and SMMU driver
maintainers didn't want any spin locks in certain IOMMU API code paths
on hardware implementations that don't need runtime PM, while we still
need to be able to control runtime PM there on hardware
implementations that need so.

Best regards,
Tomasz

Re: [PATCH v12 2/4] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Rafael J. Wysocki]

On Wed, Jul 11, 2018 at 12:05 PM, Tomasz Figa <tfiga@chromium.org> wrote:
> Hi Rafael,
>
> Thanks for review.
>
> On Wed, Jul 11, 2018 at 6:53 PM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>
>> On Sunday, July 8, 2018 7:34:11 PM CEST Vivek Gautam wrote:
>> > From: Sricharan R <sricharan@codeaurora.org>
>> >
>> > The smmu device probe/remove and add/remove master device callbacks
>> > gets called when the smmu is not linked to its master, that is without
>> > the context of the master device. So calling runtime apis in those places
>> > separately.
>> >
>> > Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> > [vivek: Cleanup pm runtime calls]
>> > Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>> > Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>> > ---
>> >
>> >  - Change since v11
>> >    * Replaced pm_runtime_disable() with pm_runtime_force_suspend()
>> >      to avoid warning about " Unpreparing enabled clock".
>> >      Full warning text mentioned in cover patch.
>> >
>> >  drivers/iommu/arm-smmu.c | 92 +++++++++++++++++++++++++++++++++++++++++++-----
>> >  1 file changed, 84 insertions(+), 8 deletions(-)
>> >
>> > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> > index a01d0dde21dd..09265e206e2d 100644
>> > --- a/drivers/iommu/arm-smmu.c
>> > +++ b/drivers/iommu/arm-smmu.c
>> > @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>> >       { 0, NULL},
>> >  };
>> >
>> > +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
>> > +{
>> > +     if (pm_runtime_enabled(smmu->dev))
>>
>> Why do you need the pm_runtime_enabled() checks here and below?
>>
>> pm_runtime_get_sync() and pm_runtime_put() should work just fine if
>> runtime PM is not enabled.
>
> Because pm_runtime_get_sync() acquires a spin lock, even if only for
> the short time of checking if runtime PM is enabled and SMMU driver
> maintainers didn't want any spin locks in certain IOMMU API code paths
> on hardware implementations that don't need runtime PM, while we still
> need to be able to control runtime PM there on hardware
> implementations that need so.

OK, so it is an optimization.  It would be good to put a comment in
there to that effect.

Re: [PATCH v12 2/4] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Vivek Gautam]

On 7/11/2018 4:29 PM, Rafael J. Wysocki wrote:
> On Wed, Jul 11, 2018 at 12:05 PM, Tomasz Figa <tfiga@chromium.org> wrote:
>> Hi Rafael,
>>
>> Thanks for review.
>>
>> On Wed, Jul 11, 2018 at 6:53 PM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>>> On Sunday, July 8, 2018 7:34:11 PM CEST Vivek Gautam wrote:
>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> The smmu device probe/remove and add/remove master device callbacks
>>>> gets called when the smmu is not linked to its master, that is without
>>>> the context of the master device. So calling runtime apis in those places
>>>> separately.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> [vivek: Cleanup pm runtime calls]
>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>> ---
>>>>
>>>>   - Change since v11
>>>>     * Replaced pm_runtime_disable() with pm_runtime_force_suspend()
>>>>       to avoid warning about " Unpreparing enabled clock".
>>>>       Full warning text mentioned in cover patch.
>>>>
>>>>   drivers/iommu/arm-smmu.c | 92 +++++++++++++++++++++++++++++++++++++++++++-----
>>>>   1 file changed, 84 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>> index a01d0dde21dd..09265e206e2d 100644
>>>> --- a/drivers/iommu/arm-smmu.c
>>>> +++ b/drivers/iommu/arm-smmu.c
>>>> @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>>>>        { 0, NULL},
>>>>   };
>>>>
>>>> +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
>>>> +{
>>>> +     if (pm_runtime_enabled(smmu->dev))
>>> Why do you need the pm_runtime_enabled() checks here and below?
>>>
>>> pm_runtime_get_sync() and pm_runtime_put() should work just fine if
>>> runtime PM is not enabled.
>> Because pm_runtime_get_sync() acquires a spin lock, even if only for
>> the short time of checking if runtime PM is enabled and SMMU driver
>> maintainers didn't want any spin locks in certain IOMMU API code paths
>> on hardware implementations that don't need runtime PM, while we still
>> need to be able to control runtime PM there on hardware
>> implementations that need so.
> OK, so it is an optimization.  It would be good to put a comment in
> there to that effect.

Yea, actually there's a comment placed in arm_smmu_device_probe()
  where the runtime PM is conditionally enabled.
I can add comments for these wrappers too if you would like.

Thanks & Regards
Vivek

[PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

From: Sricharan R <sricharan@codeaurora.org>

Finally add the device link between the master device and
smmu, so that the smmu gets runtime enabled/disabled only when the
master needs it. This is done from add_device callback which gets
called once when the master is added to the smmu.

Signed-off-by: Sricharan R <sricharan@codeaurora.org>
Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Lukas Wunner <lukas@wunner.de>
---

 - Change since v11
   * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.

 drivers/iommu/arm-smmu.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 09265e206e2d..916cde4954d2 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
 
 	iommu_device_link(&smmu->iommu, dev);
 
+	if (pm_runtime_enabled(smmu->dev) &&
+	    !device_link_add(dev, smmu->dev,
+			DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER)) {
+		dev_err(smmu->dev, "Unable to add link to the consumer %s\n",
+			dev_name(dev));
+		ret = -ENODEV;
+		goto out_unlink;
+	}
+
 	return 0;
 
+out_unlink:
+	iommu_device_unlink(&smmu->iommu, dev);
+	arm_smmu_master_free_smes(fwspec);
 out_cfg_free:
 	kfree(cfg);
 out_free:
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Rafael J. Wysocki]

On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
> From: Sricharan R <sricharan@codeaurora.org>
> 
> Finally add the device link between the master device and
> smmu, so that the smmu gets runtime enabled/disabled only when the
> master needs it. This is done from add_device callback which gets
> called once when the master is added to the smmu.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Lukas Wunner <lukas@wunner.de>
> ---
> 
>  - Change since v11
>    * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
> 
>  drivers/iommu/arm-smmu.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 09265e206e2d..916cde4954d2 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>  
>  	iommu_device_link(&smmu->iommu, dev);
>  
> +	if (pm_runtime_enabled(smmu->dev) &&

Why does the creation of the link depend on whether or not runtime PM
is enabled for the MMU device?

What about system-wide PM and system shutdown?  Are they always guaranteed
to happen in the right order without the link?

> +	    !device_link_add(dev, smmu->dev,
> +			DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER)) {
> +		dev_err(smmu->dev, "Unable to add link to the consumer %s\n",
> +			dev_name(dev));
> +		ret = -ENODEV;
> +		goto out_unlink;
> +	}
> +
>  	return 0;
>  
> +out_unlink:
> +	iommu_device_unlink(&smmu->iommu, dev);
> +	arm_smmu_master_free_smes(fwspec);
>  out_cfg_free:
>  	kfree(cfg);
>  out_free:
> 

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

Hi Rafael,


On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>> From: Sricharan R <sricharan@codeaurora.org>
>>
>> Finally add the device link between the master device and
>> smmu, so that the smmu gets runtime enabled/disabled only when the
>> master needs it. This is done from add_device callback which gets
>> called once when the master is added to the smmu.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Lukas Wunner <lukas@wunner.de>
>> ---
>>
>>   - Change since v11
>>     * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>
>>   drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 09265e206e2d..916cde4954d2 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>   
>>   	iommu_device_link(&smmu->iommu, dev);
>>   
>> +	if (pm_runtime_enabled(smmu->dev) &&
> Why does the creation of the link depend on whether or not runtime PM
> is enabled for the MMU device?

The main purpose of this device link is to handle the runtime PM 
synchronization
between the supplier (iommu) and consumer (client devices, such as 
GPU/display).
Moreover, the runtime pm is conditionally enabled for smmu devices that 
support
such [1].
>
> What about system-wide PM and system shutdown?  Are they always guaranteed
> to happen in the right order without the link?

When there's no runtime PM, there's no clocks, and other resources to be 
handled.
So, we don't need device link for system-wide PM and system shutdown to 
work correctly.
That's the case with current arm-smmu driver.
Is it something that i am missing here?

[1] https://lkml.org/lkml/2018/3/8/775

Thanks
Vivek
>> +	    !device_link_add(dev, smmu->dev,
>> +			DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER)) {
>> +		dev_err(smmu->dev, "Unable to add link to the consumer %s\n",
>> +			dev_name(dev));
>> +		ret = -ENODEV;
>> +		goto out_unlink;
>> +	}
>> +
>>   	return 0;
>>   
>> +out_unlink:
>> +	iommu_device_unlink(&smmu->iommu, dev);
>> +	arm_smmu_master_free_smes(fwspec);
>>   out_cfg_free:
>>   	kfree(cfg);
>>   out_free:
>>
>

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

Hi Rafael,


On Wed, Jul 11, 2018 at 4:06 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> Hi Rafael,
>
>
>
> On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
>>
>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>
>>> From: Sricharan R <sricharan@codeaurora.org>
>>>
>>> Finally add the device link between the master device and
>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>> master needs it. This is done from add_device callback which gets
>>> called once when the master is added to the smmu.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Lukas Wunner <lukas@wunner.de>
>>> ---
>>>
>>>   - Change since v11
>>>     * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>
>>>   drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>   1 file changed, 12 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> index 09265e206e2d..916cde4954d2 100644
>>> --- a/drivers/iommu/arm-smmu.c
>>> +++ b/drivers/iommu/arm-smmu.c
>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>>         iommu_device_link(&smmu->iommu, dev);
>>>   +     if (pm_runtime_enabled(smmu->dev) &&
>>
>> Why does the creation of the link depend on whether or not runtime PM
>> is enabled for the MMU device?
>
>
> The main purpose of this device link is to handle the runtime PM
> synchronization
> between the supplier (iommu) and consumer (client devices, such as
> GPU/display).
> Moreover, the runtime pm is conditionally enabled for smmu devices that
> support
> such [1].

Is there something you would like me to modify in this patch?

Best regards
Vivek

>>
>>
>> What about system-wide PM and system shutdown?  Are they always guaranteed
>> to happen in the right order without the link?
>
>
> When there's no runtime PM, there's no clocks, and other resources to be
> handled.
> So, we don't need device link for system-wide PM and system shutdown to work
> correctly.
> That's the case with current arm-smmu driver.
> Is it something that i am missing here?
>
> [1] https://lkml.org/lkml/2018/3/8/775
>
> Thanks
> Vivek
>>>
>>> +           !device_link_add(dev, smmu->dev,
>>> +                       DL_FLAG_PM_RUNTIME |
>>> DL_FLAG_AUTOREMOVE_SUPPLIER)) {
>>> +               dev_err(smmu->dev, "Unable to add link to the consumer
>>> %s\n",
>>> +                       dev_name(dev));
>>> +               ret = -ENODEV;
>>> +               goto out_unlink;
>>> +       }
>>> +
>>>         return 0;
>>>   +out_unlink:
>>> +       iommu_device_unlink(&smmu->iommu, dev);
>>> +       arm_smmu_master_free_smes(fwspec);
>>>   out_cfg_free:
>>>         kfree(cfg);
>>>   out_free:
>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Rafael J. Wysocki]

On Thu, Jul 12, 2018 at 2:41 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
> Hi Rafael,
>
>
> On Wed, Jul 11, 2018 at 4:06 PM, Vivek Gautam
> <vivek.gautam@codeaurora.org> wrote:
>> Hi Rafael,
>>
>>
>>
>> On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
>>>
>>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>>
>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> Finally add the device link between the master device and
>>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>>> master needs it. This is done from add_device callback which gets
>>>> called once when the master is added to the smmu.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>> Cc: Lukas Wunner <lukas@wunner.de>
>>>> ---
>>>>
>>>>   - Change since v11
>>>>     * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>>
>>>>   drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>>   1 file changed, 12 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>> index 09265e206e2d..916cde4954d2 100644
>>>> --- a/drivers/iommu/arm-smmu.c
>>>> +++ b/drivers/iommu/arm-smmu.c
>>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>>>         iommu_device_link(&smmu->iommu, dev);
>>>>   +     if (pm_runtime_enabled(smmu->dev) &&
>>>
>>> Why does the creation of the link depend on whether or not runtime PM
>>> is enabled for the MMU device?
>>
>>
>> The main purpose of this device link is to handle the runtime PM
>> synchronization
>> between the supplier (iommu) and consumer (client devices, such as
>> GPU/display).
>> Moreover, the runtime pm is conditionally enabled for smmu devices that
>> support
>> such [1].
>
> Is there something you would like me to modify in this patch?

Not really, as long as you are sure that it is correct. :-)

You need to remember, however, that if you add system-wide PM
callbacks to the driver, the ordering between them and the client
device callbacks during system-wide suspend matters as well.  Don't
you need the link the ensure the correct system-wide suspend ordering
too?

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

On 7/16/2018 2:25 PM, Rafael J. Wysocki wrote:
> On Thu, Jul 12, 2018 at 2:41 PM, Vivek Gautam
> <vivek.gautam@codeaurora.org> wrote:
>> Hi Rafael,
>>
>>
>> On Wed, Jul 11, 2018 at 4:06 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:
>>> Hi Rafael,
>>>
>>>
>>>
>>> On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
>>>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>
>>>>> Finally add the device link between the master device and
>>>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>>>> master needs it. This is done from add_device callback which gets
>>>>> called once when the master is added to the smmu.
>>>>>
>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>>> Cc: Lukas Wunner <lukas@wunner.de>
>>>>> ---
>>>>>
>>>>>    - Change since v11
>>>>>      * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>>>
>>>>>    drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>>>    1 file changed, 12 insertions(+)
>>>>>
>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>> index 09265e206e2d..916cde4954d2 100644
>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>>>>          iommu_device_link(&smmu->iommu, dev);
>>>>>    +     if (pm_runtime_enabled(smmu->dev) &&
>>>> Why does the creation of the link depend on whether or not runtime PM
>>>> is enabled for the MMU device?
>>>
>>> The main purpose of this device link is to handle the runtime PM
>>> synchronization
>>> between the supplier (iommu) and consumer (client devices, such as
>>> GPU/display).
>>> Moreover, the runtime pm is conditionally enabled for smmu devices that
>>> support
>>> such [1].
>> Is there something you would like me to modify in this patch?
> Not really, as long as you are sure that it is correct. :-)
>
> You need to remember, however, that if you add system-wide PM
> callbacks to the driver, the ordering between them and the client
> device callbacks during system-wide suspend matters as well.  Don't
> you need the link the ensure the correct system-wide suspend ordering
> too?

The fact that currently we handle clocks only through runtime pm callbacks,
would it be better to call runtime pm put/get in system-wide PM callbacks.
This would be same as i mentioned in the other thread.

Best regards
Vivek

> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Rafael J. Wysocki]

On Mon, Jul 16, 2018 at 1:46 PM, Vivek Gautam
<vivek.gautam@codeaurora.org> wrote:
>
>
> On 7/16/2018 2:25 PM, Rafael J. Wysocki wrote:
>>
>> On Thu, Jul 12, 2018 at 2:41 PM, Vivek Gautam
>> <vivek.gautam@codeaurora.org> wrote:
>>>
>>> Hi Rafael,
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:06 PM, Vivek Gautam
>>> <vivek.gautam@codeaurora.org> wrote:
>>>>
>>>> Hi Rafael,
>>>>
>>>>
>>>>
>>>> On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
>>>>>
>>>>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>>>>
>>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>>
>>>>>> Finally add the device link between the master device and
>>>>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>>>>> master needs it. This is done from add_device callback which gets
>>>>>> called once when the master is added to the smmu.
>>>>>>
>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>>>> Cc: Lukas Wunner <lukas@wunner.de>
>>>>>> ---
>>>>>>
>>>>>>    - Change since v11
>>>>>>      * Replaced DL_FLAG_AUTOREMOVE flag with
>>>>>> DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>>>>
>>>>>>    drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>>>>    1 file changed, 12 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>>> index 09265e206e2d..916cde4954d2 100644
>>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device
>>>>>> *dev)
>>>>>>          iommu_device_link(&smmu->iommu, dev);
>>>>>>    +     if (pm_runtime_enabled(smmu->dev) &&
>>>>>
>>>>> Why does the creation of the link depend on whether or not runtime PM
>>>>> is enabled for the MMU device?
>>>>
>>>>
>>>> The main purpose of this device link is to handle the runtime PM
>>>> synchronization
>>>> between the supplier (iommu) and consumer (client devices, such as
>>>> GPU/display).
>>>> Moreover, the runtime pm is conditionally enabled for smmu devices that
>>>> support
>>>> such [1].
>>>
>>> Is there something you would like me to modify in this patch?
>>
>> Not really, as long as you are sure that it is correct. :-)
>>
>> You need to remember, however, that if you add system-wide PM
>> callbacks to the driver, the ordering between them and the client
>> device callbacks during system-wide suspend matters as well.  Don't
>> you need the link the ensure the correct system-wide suspend ordering
>> too?
>
>
> The fact that currently we handle clocks only through runtime pm callbacks,
> would it be better to call runtime pm put/get in system-wide PM callbacks.
> This would be same as i mentioned in the other thread.

Well, my point is that there's no reason for system-wide suspend to
depend directly on runtime PM (which can be effectively disabled by
user space as mentioned for multiple times in this thread).

It simply is not efficient to let the clock run while the system as a
whole is sleeping (even if power/control has been set to "on" for this
particular device) and it should not be too hard to prevent that from
happening.  You may need an additional flag in the driver for that or
similar, but it definitely should be doable.

Now, that's my advice and I'm not the maintainer of that code, so it
is your call (as long as the maintainer agrees with it).

Thanks,
Rafael

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

On 7/17/2018 1:16 PM, Rafael J. Wysocki wrote:
> On Mon, Jul 16, 2018 at 1:46 PM, Vivek Gautam
> <vivek.gautam@codeaurora.org> wrote:
>>
>> On 7/16/2018 2:25 PM, Rafael J. Wysocki wrote:
>>> On Thu, Jul 12, 2018 at 2:41 PM, Vivek Gautam
>>> <vivek.gautam@codeaurora.org> wrote:
>>>> Hi Rafael,
>>>>
>>>>
>>>> On Wed, Jul 11, 2018 at 4:06 PM, Vivek Gautam
>>>> <vivek.gautam@codeaurora.org> wrote:
>>>>> Hi Rafael,
>>>>>
>>>>>
>>>>>
>>>>> On 7/11/2018 3:23 PM, Rafael J. Wysocki wrote:
>>>>>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>>>>
>>>>>>> Finally add the device link between the master device and
>>>>>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>>>>>> master needs it. This is done from add_device callback which gets
>>>>>>> called once when the master is added to the smmu.
>>>>>>>
>>>>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>>>>> Cc: Lukas Wunner <lukas@wunner.de>
>>>>>>> ---
>>>>>>>
>>>>>>>     - Change since v11
>>>>>>>       * Replaced DL_FLAG_AUTOREMOVE flag with
>>>>>>> DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>>>>>
>>>>>>>     drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>>>>>     1 file changed, 12 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>>>>> index 09265e206e2d..916cde4954d2 100644
>>>>>>> --- a/drivers/iommu/arm-smmu.c
>>>>>>> +++ b/drivers/iommu/arm-smmu.c
>>>>>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device
>>>>>>> *dev)
>>>>>>>           iommu_device_link(&smmu->iommu, dev);
>>>>>>>     +     if (pm_runtime_enabled(smmu->dev) &&
>>>>>> Why does the creation of the link depend on whether or not runtime PM
>>>>>> is enabled for the MMU device?
>>>>>
>>>>> The main purpose of this device link is to handle the runtime PM
>>>>> synchronization
>>>>> between the supplier (iommu) and consumer (client devices, such as
>>>>> GPU/display).
>>>>> Moreover, the runtime pm is conditionally enabled for smmu devices that
>>>>> support
>>>>> such [1].
>>>> Is there something you would like me to modify in this patch?
>>> Not really, as long as you are sure that it is correct. :-)
>>>
>>> You need to remember, however, that if you add system-wide PM
>>> callbacks to the driver, the ordering between them and the client
>>> device callbacks during system-wide suspend matters as well.  Don't
>>> you need the link the ensure the correct system-wide suspend ordering
>>> too?
>>
>> The fact that currently we handle clocks only through runtime pm callbacks,
>> would it be better to call runtime pm put/get in system-wide PM callbacks.
>> This would be same as i mentioned in the other thread.
> Well, my point is that there's no reason for system-wide suspend to
> depend directly on runtime PM (which can be effectively disabled by
> user space as mentioned for multiple times in this thread).
>
> It simply is not efficient to let the clock run while the system as a
> whole is sleeping (even if power/control has been set to "on" for this
> particular device) and it should not be too hard to prevent that from
> happening.  You may need an additional flag in the driver for that or
> similar, but it definitely should be doable.

Right, I will modify the things are required. Thanks again for pointing 
this out.

Best regards
Vivek

>
> Now, that's my advice and I'm not the maintainer of that code, so it
> is your call (as long as the maintainer agrees with it).
>
> Thanks,
> Rafael

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

On Wed, Jul 11, 2018 at 3:23 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>> From: Sricharan R <sricharan@codeaurora.org>
>>
>> Finally add the device link between the master device and
>> smmu, so that the smmu gets runtime enabled/disabled only when the
>> master needs it. This is done from add_device callback which gets
>> called once when the master is added to the smmu.
>>
>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Lukas Wunner <lukas@wunner.de>
>> ---
>>
>>  - Change since v11
>>    * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>
>>  drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>  1 file changed, 12 insertions(+)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 09265e206e2d..916cde4954d2 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>
>>       iommu_device_link(&smmu->iommu, dev);
>>
>> +     if (pm_runtime_enabled(smmu->dev) &&
>
> Why does the creation of the link depend on whether or not runtime PM
> is enabled for the MMU device?
>
> What about system-wide PM and system shutdown?  Are they always guaranteed
> to happen in the right order without the link?

Hi Robin,

As Rafael pointed, we should the device link creation should not depend on
runtime PM being enabled or not, as we would also want to guarantee
that system wide PM callbacks are called in the right order for smmu
and clients.

Does this change of removing the check for pm_runtime_enabled() from here
looks okay to you?

FYI, as discussed in the first patch [1] of this series, I will add a
system wide
suspend callback - arm_smmu_pm_suspend, that would do clock disable, and will
add corresponding clock enable calls in arm_smmu_pm_resume().

[1] https://lore.kernel.org/patchwork/patch/960460/


Best regards
Vivek

>
>> +         !device_link_add(dev, smmu->dev,
>> +                     DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER)) {
>> +             dev_err(smmu->dev, "Unable to add link to the consumer %s\n",
>> +                     dev_name(dev));
>> +             ret = -ENODEV;
>> +             goto out_unlink;
>> +     }
>> +
>>       return 0;
>>
>> +out_unlink:
>> +     iommu_device_unlink(&smmu->iommu, dev);
>> +     arm_smmu_master_free_smes(fwspec);
>>  out_cfg_free:
>>       kfree(cfg);
>>  out_free:
>>
>
>



-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Robin Murphy]

On 18/07/18 10:30, Vivek Gautam wrote:
> On Wed, Jul 11, 2018 at 3:23 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>> From: Sricharan R <sricharan@codeaurora.org>
>>>
>>> Finally add the device link between the master device and
>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>> master needs it. This is done from add_device callback which gets
>>> called once when the master is added to the smmu.
>>>
>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Lukas Wunner <lukas@wunner.de>
>>> ---
>>>
>>>   - Change since v11
>>>     * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>
>>>   drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>   1 file changed, 12 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> index 09265e206e2d..916cde4954d2 100644
>>> --- a/drivers/iommu/arm-smmu.c
>>> +++ b/drivers/iommu/arm-smmu.c
>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device *dev)
>>>
>>>        iommu_device_link(&smmu->iommu, dev);
>>>
>>> +     if (pm_runtime_enabled(smmu->dev) &&
>>
>> Why does the creation of the link depend on whether or not runtime PM
>> is enabled for the MMU device?
>>
>> What about system-wide PM and system shutdown?  Are they always guaranteed
>> to happen in the right order without the link?
> 
> Hi Robin,
> 
> As Rafael pointed, we should the device link creation should not depend on
> runtime PM being enabled or not, as we would also want to guarantee
> that system wide PM callbacks are called in the right order for smmu
> and clients.
> 
> Does this change of removing the check for pm_runtime_enabled() from here
> looks okay to you?

FWIW the existing system PM ops make no claim to be perfect, and I 
wouldn't be at all surprised if it was only by coincidence that my 
devices happened to put on the relevant lists in the right order to 
start with. If we no longer need to worry about explicit device_link 
housekeeping in the SMMU driver, then creating them unconditionally 
sounds like the sensible thing to do. I'd be inclined to treat failure 
as non-fatal like we do for the sysfs link, though, since it's another 
thing that correct SMMU operation doesn't actually depend on (at this 
point we don't necessarily know if this consumer even has a driver at all).

> FYI, as discussed in the first patch [1] of this series, I will add a
> system wide
> suspend callback - arm_smmu_pm_suspend, that would do clock disable, and will
> add corresponding clock enable calls in arm_smmu_pm_resume().

OK, I still don't really understand the finer points of how system PM 
and runtime PM interact, but if making it robust is just a case of 
calling the runtime suspend/resume hooks as appropriate from the system 
ones, that sounds reasonable.

Robin.

> 
> [1] https://lore.kernel.org/patchwork/patch/960460/
> 
> 
> Best regards
> Vivek
> 
>>
>>> +         !device_link_add(dev, smmu->dev,
>>> +                     DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER)) {
>>> +             dev_err(smmu->dev, "Unable to add link to the consumer %s\n",
>>> +                     dev_name(dev));
>>> +             ret = -ENODEV;
>>> +             goto out_unlink;
>>> +     }
>>> +
>>>        return 0;
>>>
>>> +out_unlink:
>>> +     iommu_device_unlink(&smmu->iommu, dev);
>>> +     arm_smmu_master_free_smes(fwspec);
>>>   out_cfg_free:
>>>        kfree(cfg);
>>>   out_free:
>>>
>>
>>
> 
> 
> 

Re: [PATCH v12 3/4] iommu/arm-smmu: Add the device_link between masters and smmu

[Vivek Gautam]

On Wed, Jul 18, 2018 at 6:13 PM, Robin Murphy <robin.murphy@arm.com> wrote:
> On 18/07/18 10:30, Vivek Gautam wrote:
>>
>> On Wed, Jul 11, 2018 at 3:23 PM, Rafael J. Wysocki <rjw@rjwysocki.net>
>> wrote:
>>>
>>> On Sunday, July 8, 2018 7:34:12 PM CEST Vivek Gautam wrote:
>>>>
>>>> From: Sricharan R <sricharan@codeaurora.org>
>>>>
>>>> Finally add the device link between the master device and
>>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>>> master needs it. This is done from add_device callback which gets
>>>> called once when the master is added to the smmu.
>>>>
>>>> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
>>>> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
>>>> Reviewed-by: Tomasz Figa <tfiga@chromium.org>
>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>> Cc: Lukas Wunner <lukas@wunner.de>
>>>> ---
>>>>
>>>>   - Change since v11
>>>>     * Replaced DL_FLAG_AUTOREMOVE flag with DL_FLAG_AUTOREMOVE_SUPPLIER.
>>>>
>>>>   drivers/iommu/arm-smmu.c | 12 ++++++++++++
>>>>   1 file changed, 12 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>>> index 09265e206e2d..916cde4954d2 100644
>>>> --- a/drivers/iommu/arm-smmu.c
>>>> +++ b/drivers/iommu/arm-smmu.c
>>>> @@ -1461,8 +1461,20 @@ static int arm_smmu_add_device(struct device
>>>> *dev)
>>>>
>>>>        iommu_device_link(&smmu->iommu, dev);
>>>>
>>>> +     if (pm_runtime_enabled(smmu->dev) &&
>>>
>>>
>>> Why does the creation of the link depend on whether or not runtime PM
>>> is enabled for the MMU device?
>>>
>>> What about system-wide PM and system shutdown?  Are they always
>>> guaranteed
>>> to happen in the right order without the link?
>>
>>
>> Hi Robin,
>>
>> As Rafael pointed, we should the device link creation should not depend on
>> runtime PM being enabled or not, as we would also want to guarantee
>> that system wide PM callbacks are called in the right order for smmu
>> and clients.
>>
>> Does this change of removing the check for pm_runtime_enabled() from here
>> looks okay to you?
>
>
> FWIW the existing system PM ops make no claim to be perfect, and I wouldn't
> be at all surprised if it was only by coincidence that my devices happened
> to put on the relevant lists in the right order to start with. If we no
> longer need to worry about explicit device_link housekeeping in the SMMU
> driver, then creating them unconditionally sounds like the sensible thing to
> do. I'd be inclined to treat failure as non-fatal like we do for the sysfs
> link, though, since it's another thing that correct SMMU operation doesn't
> actually depend on (at this point we don't necessarily know if this consumer
> even has a driver at all).

Thanks. I will then respin the patch taking care of treating failure
as non-fatal.

>> FYI, as discussed in the first patch [1] of this series, I will add a
>> system wide
>> suspend callback - arm_smmu_pm_suspend, that would do clock disable, and
>> will
>> add corresponding clock enable calls in arm_smmu_pm_resume().
>
>
> OK, I still don't really understand the finer points of how system PM and
> runtime PM interact, but if making it robust is just a case of calling the
> runtime suspend/resume hooks as appropriate from the system ones, that
> sounds reasonable.

Sure. Thanks.

Best regards
Vivek

>
> Robin.
>
>>
>> [1] https://lore.kernel.org/patchwork/patch/960460/
>>
>>
>> Best regards
>> Vivek
>>
>>>
>>>> +         !device_link_add(dev, smmu->dev,
>>>> +                     DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER))
>>>> {
>>>> +             dev_err(smmu->dev, "Unable to add link to the consumer
>>>> %s\n",
>>>> +                     dev_name(dev));
>>>> +             ret = -ENODEV;
>>>> +             goto out_unlink;
>>>> +     }
>>>> +
>>>>        return 0;
>>>>
>>>> +out_unlink:
>>>> +     iommu_device_unlink(&smmu->iommu, dev);
>>>> +     arm_smmu_master_free_smes(fwspec);
>>>>   out_cfg_free:
>>>>        kfree(cfg);
>>>>   out_free:
>>>>
>>>
>>>
>>
>>
>>
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu



-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

[PATCH v12 4/4] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

[Vivek Gautam]

qcom,smmu-v2 is an arm,smmu-v2 implementation with specific
clock and power requirements. This smmu core is used with
multiple masters on msm8996, viz. mdss, video, etc.
Add bindings for the same.

Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
Reviewed-by: Rob Herring <robh@kernel.org>
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
---

 - No change since v11.

 .../devicetree/bindings/iommu/arm,smmu.txt         | 42 ++++++++++++++++++++++
 drivers/iommu/arm-smmu.c                           | 14 ++++++++
 2 files changed, 56 insertions(+)

diff --git a/Documentation/devicetree/bindings/iommu/arm,smmu.txt b/Documentation/devicetree/bindings/iommu/arm,smmu.txt
index 8a6ffce12af5..7c71a6ed465a 100644
--- a/Documentation/devicetree/bindings/iommu/arm,smmu.txt
+++ b/Documentation/devicetree/bindings/iommu/arm,smmu.txt
@@ -17,10 +17,19 @@ conditions.
                         "arm,mmu-401"
                         "arm,mmu-500"
                         "cavium,smmu-v2"
+                        "qcom,<soc>-smmu-v2", "qcom,smmu-v2"
 
                   depending on the particular implementation and/or the
                   version of the architecture implemented.
 
+                  A number of Qcom SoCs use qcom,smmu-v2 version of the IP.
+                  "qcom,<soc>-smmu-v2" represents a soc specific compatible
+                  string that should be present along with the "qcom,smmu-v2"
+                  to facilitate SoC specific clocks/power connections and to
+                  address specific bug fixes.
+                  An example string would be -
+                  "qcom,msm8996-smmu-v2", "qcom,smmu-v2".
+
 - reg           : Base address and size of the SMMU.
 
 - #global-interrupts : The number of global interrupts exposed by the
@@ -71,6 +80,22 @@ conditions.
                   or using stream matching with #iommu-cells = <2>, and
                   may be ignored if present in such cases.
 
+- clock-names:    List of the names of clocks input to the device. The
+                  required list depends on particular implementation and
+                  is as follows:
+                  - for "qcom,smmu-v2":
+                    - "bus": clock required for downstream bus access and
+                             for the smmu ptw,
+                    - "iface": clock required to access smmu's registers
+                               through the TCU's programming interface.
+                  - unspecified for other implementations.
+
+- clocks:         Specifiers for all clocks listed in the clock-names property,
+                  as per generic clock bindings.
+
+- power-domains:  Specifiers for power domains required to be powered on for
+                  the SMMU to operate, as per generic power domain bindings.
+
 ** Deprecated properties:
 
 - mmu-masters (deprecated in favour of the generic "iommus" binding) :
@@ -137,3 +162,20 @@ conditions.
                 iommu-map = <0 &smmu3 0 0x400>;
                 ...
         };
+
+	/* Qcom's arm,smmu-v2 implementation */
+	smmu4: iommu {
+		compatible = "qcom,msm8996-smmu-v2", "qcom,smmu-v2";
+		reg = <0xd00000 0x10000>;
+
+		#global-interrupts = <1>;
+		interrupts = <GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 320 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 321 IRQ_TYPE_LEVEL_HIGH>;
+		#iommu-cells = <1>;
+		power-domains = <&mmcc MDSS_GDSC>;
+
+		clocks = <&mmcc SMMU_MDP_AXI_CLK>,
+			 <&mmcc SMMU_MDP_AHB_CLK>;
+		clock-names = "bus", "iface";
+	};
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 916cde4954d2..a9edb17f09bf 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -119,6 +119,7 @@ enum arm_smmu_implementation {
 	GENERIC_SMMU,
 	ARM_MMU500,
 	CAVIUM_SMMUV2,
+	QCOM_SMMUV2,
 };
 
 struct arm_smmu_s2cr {
@@ -1980,6 +1981,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, ARM_SMMU_V1_64K, GENERIC_SMMU);
 ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500);
 ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2);
 
+static const char * const qcom_smmuv2_clks[] = {
+	"bus", "iface",
+};
+
+static const struct arm_smmu_match_data qcom_smmuv2 = {
+	.version = ARM_SMMU_V2,
+	.model = QCOM_SMMUV2,
+	.clks = qcom_smmuv2_clks,
+	.num_clks = ARRAY_SIZE(qcom_smmuv2_clks),
+};
+
 static const struct of_device_id arm_smmu_of_match[] = {
 	{ .compatible = "arm,smmu-v1", .data = &smmu_generic_v1 },
 	{ .compatible = "arm,smmu-v2", .data = &smmu_generic_v2 },
@@ -1987,6 +1999,7 @@ static const struct of_device_id arm_smmu_of_match[] = {
 	{ .compatible = "arm,mmu-401", .data = &arm_mmu401 },
 	{ .compatible = "arm,mmu-500", .data = &arm_mmu500 },
 	{ .compatible = "cavium,smmu-v2", .data = &cavium_smmuv2 },
+	{ .compatible = "qcom,smmu-v2", .data = &qcom_smmuv2 },
 	{ },
 };
 MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
@@ -2361,6 +2374,7 @@ IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400");
 IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401");
 IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500");
 IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2");
+IOMMU_OF_DECLARE(qcom_smmuv2, "qcom,smmu-v2");
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
 MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation