[PATCH v4 0/4] amba/dma: pl330: add Power Management support

[PATCH v4 0/4] amba/dma: pl330: add Power Management support

[Krzysztof Kozlowski]

Hi,


This is fourth version of patches adding runtime PM to the pl330
driver.

The runtime PM of pl330 driver requires interrupt safe suspend/resume
callbacks which is in conflict with current amba bus driver.
The latter also unprepares and prepares the AMBA bus clock which
is not safe for atomic context.

The patchset solves this in patch 3/4 by handling clocks in different
way if device driver set interrupt safe runtime PM.

Any comments are welcome.


Tested on boards with pl330 DMA driver:
 - Trats2 (Exynos4212)
 - Samsung Gear2-like development board (Exynos3250)


TODO/ideas:
===========
Consider renaming existing pm_runtime_irq_safe() function to keep
up with naming convention after adding pm_runtime_is_irq_safe() in
patch 1/4.
This naming:
 - pm_runtime_irq_safe() as setter,
 - pm_runtime_is_irq_safe() as getter,
is contradictory to naming of other PM runtime functions, e.g.:
 - pm_runtime_set_active() as setter,
 - pm_runtime_active() as getter.
I didn't change the naming of pm_runtime_irq_safe() because I wanted
to minimize the impact of the patches.


Changes since v3:
=================
1. Patch 1/4: Document new API in Documentation/power/runtime_pm.txt
   (pointed by Alan Stern).

Changes since v2:
=================
1. Add patch 1 (PM / Runtime: Add getter for quering the IRQ safe option)
2. Add patch 2 (amba: Add helper macros for (un)preparing AMBA clock)
3. Patch 3/4: Rewrite the idea. If IRQ safe runtime PM is set then
   do not unprepare/prepare the clocks. Suggested by Russell King.
4. Patch 4/4: During system sleep unprepare the clock.

Changes since v1:
=================
1. Add patch 1 (amba: Allow AMBA drivers to use their own runtime PM).
2. Patch 2/2: Apply Michal Simek's suggestions.
3. Patch 2/2: Fix atomic context safeness in pl330_issue_pending().


Best regards,
Krzysztof Kozlowski


Krzysztof Kozlowski (4):
  PM / Runtime: Add getter for quering the IRQ safe option
  amba: Add helper macros for (un)preparing AMBA clock
  amba: Don't unprepare the clocks if device driver wants IRQ safe
    runtime PM
  dma: pl330: add Power Management support

 Documentation/power/runtime_pm.txt |  4 ++
 drivers/amba/bus.c                 | 30 +++++++++++++--
 drivers/dma/pl330.c                | 77 ++++++++++++++++++++++++++++++++++++--
 include/linux/amba/bus.h           |  7 ++++
 include/linux/pm_runtime.h         |  6 +++
 5 files changed, 116 insertions(+), 8 deletions(-)

-- 
1.9.1

[PATCH v4 1/4] PM / Runtime: Add getter for quering the IRQ safe option

[Krzysztof Kozlowski]

Add a simple getter pm_runtime_is_irq_safe() for quering whether runtime
PM IRQ safe was set or not.

Various bus drivers implementing runtime PM may use choose to suspend
differently based on IRQ safeness status of child driver (e.g. do not
unprepare the clock if IRQ safe is not set).

Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
---
 Documentation/power/runtime_pm.txt | 4 ++++
 include/linux/pm_runtime.h         | 6 ++++++
 2 files changed, 10 insertions(+)

diff --git a/Documentation/power/runtime_pm.txt b/Documentation/power/runtime_pm.txt
index f32ce5419573..397b81593142 100644
--- a/Documentation/power/runtime_pm.txt
+++ b/Documentation/power/runtime_pm.txt
@@ -468,6 +468,10 @@ drivers/base/power/runtime.c and include/linux/pm_runtime.h:
     - set the power.irq_safe flag for the device, causing the runtime-PM
       callbacks to be invoked with interrupts off
 
+  bool pm_runtime_is_irq_safe(struct device *dev);
+    - return true if power.irq_safe flag was set for the device, causing
+      the runtime-PM callbacks to be invoked with interrupts off
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
diff --git a/include/linux/pm_runtime.h b/include/linux/pm_runtime.h
index 367f49b9a1c9..44d74f0f182e 100644
--- a/include/linux/pm_runtime.h
+++ b/include/linux/pm_runtime.h
@@ -128,6 +128,11 @@ static inline void pm_runtime_mark_last_busy(struct device *dev)
 	ACCESS_ONCE(dev->power.last_busy) = jiffies;
 }
 
+static inline bool pm_runtime_is_irq_safe(struct device *dev)
+{
+	return dev->power.irq_safe;
+}
+
 #else /* !CONFIG_PM_RUNTIME */
 
 static inline int __pm_runtime_idle(struct device *dev, int rpmflags)
@@ -167,6 +172,7 @@ static inline bool pm_runtime_enabled(struct device *dev) { return false; }
 
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
 static inline void pm_runtime_irq_safe(struct device *dev) {}
+static inline bool pm_runtime_is_irq_safe(struct device *dev) { return false; }
 
 static inline bool pm_runtime_callbacks_present(struct device *dev) { return false; }
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
-- 
1.9.1

[PATCH v4 2/4] amba: Add helper macros for (un)preparing AMBA clock

[Krzysztof Kozlowski]

Add amba_pclk_prepare() and amba_pclk_unprepare() macros for handling
the AMBA bus clock by device drivers.

Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
---
 include/linux/amba/bus.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/linux/amba/bus.h b/include/linux/amba/bus.h
index fdd7e1b61f60..ad52027a9cbf 100644
--- a/include/linux/amba/bus.h
+++ b/include/linux/amba/bus.h
@@ -92,6 +92,12 @@ void amba_release_regions(struct amba_device *);
 #define amba_pclk_disable(d)	\
 	do { if (!IS_ERR((d)->pclk)) clk_disable((d)->pclk); } while (0)
 
+#define amba_pclk_prepare(d)	\
+	(IS_ERR((d)->pclk) ? 0 : clk_prepare((d)->pclk))
+
+#define amba_pclk_unprepare(d)	\
+	do { if (!IS_ERR((d)->pclk)) clk_unprepare((d)->pclk); } while (0)
+
 /* Some drivers don't use the struct amba_device */
 #define AMBA_CONFIG_BITS(a) (((a) >> 24) & 0xff)
 #define AMBA_REV_BITS(a) (((a) >> 20) & 0x0f)
-- 
1.9.1

[PATCH v4 3/4] amba: Don't unprepare the clocks if device driver wants IRQ safe runtime PM

[Krzysztof Kozlowski]

The AMBA bus driver defines runtime Power Management functions which
disable and unprepare AMBA bus clock. This is problematic for runtime PM
because unpreparing a clock might sleep so it is not interrupt safe.

However some drivers may want to implement runtime PM functions in
interrupt-safe way (see pm_runtime_irq_safe()). In such case the AMBA
bus driver should not unprepare/prepare the clock in runtime suspend and
resume callbacks.

Detect the device driver behavior after calling its probe function and
store it. During runtime suspend/resume deal with clocks according to
stored value.

Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
---
 drivers/amba/bus.c       | 30 ++++++++++++++++++++++++++----
 include/linux/amba/bus.h |  1 +
 2 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/drivers/amba/bus.c b/drivers/amba/bus.c
index 3cf61a127ee5..4aee6e61bed2 100644
--- a/drivers/amba/bus.c
+++ b/drivers/amba/bus.c
@@ -94,8 +94,18 @@ static int amba_pm_runtime_suspend(struct device *dev)
 	struct amba_device *pcdev = to_amba_device(dev);
 	int ret = pm_generic_runtime_suspend(dev);
 
-	if (ret == 0 && dev->driver)
-		clk_disable_unprepare(pcdev->pclk);
+	if (ret == 0 && dev->driver) {
+		/*
+		 * Drivers should not change pm_runtime_irq_safe()
+		 * after probe.
+		 */
+		WARN_ON(pcdev->irq_safe != pm_runtime_is_irq_safe(dev));
+
+		if (pcdev->irq_safe)
+			clk_disable(pcdev->pclk);
+		else
+			clk_disable_unprepare(pcdev->pclk);
+	}
 
 	return ret;
 }
@@ -106,7 +116,16 @@ static int amba_pm_runtime_resume(struct device *dev)
 	int ret;
 
 	if (dev->driver) {
-		ret = clk_prepare_enable(pcdev->pclk);
+		/*
+		 * Drivers should not change pm_runtime_irq_safe()
+		 * after probe.
+		 */
+		WARN_ON(pcdev->irq_safe != pm_runtime_is_irq_safe(dev));
+
+		if (pcdev->irq_safe)
+			ret = clk_enable(pcdev->pclk);
+		else
+			ret = clk_prepare_enable(pcdev->pclk);
 		/* Failure is probably fatal to the system, but... */
 		if (ret)
 			return ret;
@@ -191,8 +210,11 @@ static int amba_probe(struct device *dev)
 		pm_runtime_enable(dev);
 
 		ret = pcdrv->probe(pcdev, id);
-		if (ret == 0)
+		if (ret == 0) {
+			if (pm_runtime_is_irq_safe(dev))
+				pcdev->irq_safe = 1;
 			break;
+		}
 
 		pm_runtime_disable(dev);
 		pm_runtime_set_suspended(dev);
diff --git a/include/linux/amba/bus.h b/include/linux/amba/bus.h
index ad52027a9cbf..ce101e4497d6 100644
--- a/include/linux/amba/bus.h
+++ b/include/linux/amba/bus.h
@@ -32,6 +32,7 @@ struct amba_device {
 	struct clk		*pclk;
 	unsigned int		periphid;
 	unsigned int		irq[AMBA_NR_IRQS];
+	unsigned int		irq_safe:1;
 };
 
 struct amba_driver {
-- 
1.9.1

[PATCH v4 4/4] dma: pl330: add Power Management support

[Krzysztof Kozlowski]

This patch adds both normal PM suspend/resume support and runtime PM
support to pl330 DMA engine driver.

The runtime power management for pl330 DMA driver allows gating of AMBA
clock (PDMA) in FSYS clock domain, when the device is not processing any
requests. This is necessary to enter low power modes on Exynos SoCs
(e.g. LPA on Exynos4x12 or W-AFTR on Exynos3250).

Runtime PM resuming of the device may happen in atomic context (during
call device_issue_pending()) so pm_runtime_irq_safe() is used. This will
lead only to disabling/enabling of the clock but this is sufficient for
gating the clock and for reducing energy usage.

During system sleep the AMBA bus clock is also unprepared.

Suggested-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
---
 drivers/dma/pl330.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 73 insertions(+), 4 deletions(-)

diff --git a/drivers/dma/pl330.c b/drivers/dma/pl330.c
index 4839bfa74a10..9c014874c33d 100644
--- a/drivers/dma/pl330.c
+++ b/drivers/dma/pl330.c
@@ -27,6 +27,7 @@
 #include <linux/of.h>
 #include <linux/of_dma.h>
 #include <linux/err.h>
+#include <linux/pm_runtime.h>
 
 #include "dmaengine.h"
 #define PL330_MAX_CHAN		8
@@ -1958,6 +1959,7 @@ static void pl330_tasklet(unsigned long data)
 	struct dma_pl330_chan *pch = (struct dma_pl330_chan *)data;
 	struct dma_pl330_desc *desc, *_dt;
 	unsigned long flags;
+	bool power_down = false;
 
 	spin_lock_irqsave(&pch->lock, flags);
 
@@ -1972,10 +1974,17 @@ static void pl330_tasklet(unsigned long data)
 	/* Try to submit a req imm. next to the last completed cookie */
 	fill_queue(pch);
 
-	/* Make sure the PL330 Channel thread is active */
-	spin_lock(&pch->thread->dmac->lock);
-	_start(pch->thread);
-	spin_unlock(&pch->thread->dmac->lock);
+	if (list_empty(&pch->work_list)) {
+		spin_lock(&pch->thread->dmac->lock);
+		_stop(pch->thread);
+		spin_unlock(&pch->thread->dmac->lock);
+		power_down = true;
+	} else {
+		/* Make sure the PL330 Channel thread is active */
+		spin_lock(&pch->thread->dmac->lock);
+		_start(pch->thread);
+		spin_unlock(&pch->thread->dmac->lock);
+	}
 
 	while (!list_empty(&pch->completed_list)) {
 		dma_async_tx_callback callback;
@@ -1990,6 +1999,12 @@ static void pl330_tasklet(unsigned long data)
 		if (pch->cyclic) {
 			desc->status = PREP;
 			list_move_tail(&desc->node, &pch->work_list);
+			if (power_down) {
+				spin_lock(&pch->thread->dmac->lock);
+				_start(pch->thread);
+				spin_unlock(&pch->thread->dmac->lock);
+				power_down = false;
+			}
 		} else {
 			desc->status = FREE;
 			list_move_tail(&desc->node, &pch->dmac->desc_pool);
@@ -2004,6 +2019,10 @@ static void pl330_tasklet(unsigned long data)
 		}
 	}
 	spin_unlock_irqrestore(&pch->lock, flags);
+
+	/* If work list empty, power down */
+	if (power_down)
+		pm_runtime_put(pch->dmac->ddma.dev);
 }
 
 bool pl330_filter(struct dma_chan *chan, void *param)
@@ -2073,6 +2092,7 @@ static int pl330_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, unsigned
 
 	switch (cmd) {
 	case DMA_TERMINATE_ALL:
+		pm_runtime_get_sync(pl330->ddma.dev);
 		spin_lock_irqsave(&pch->lock, flags);
 
 		spin_lock(&pl330->lock);
@@ -2099,10 +2119,14 @@ static int pl330_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, unsigned
 			dma_cookie_complete(&desc->txd);
 		}
 
+		if (!list_empty(&pch->work_list))
+			pm_runtime_put(pl330->ddma.dev);
+
 		list_splice_tail_init(&pch->submitted_list, &pl330->desc_pool);
 		list_splice_tail_init(&pch->work_list, &pl330->desc_pool);
 		list_splice_tail_init(&pch->completed_list, &pl330->desc_pool);
 		spin_unlock_irqrestore(&pch->lock, flags);
+		pm_runtime_put(pl330->ddma.dev);
 		break;
 	case DMA_SLAVE_CONFIG:
 		slave_config = (struct dma_slave_config *)arg;
@@ -2138,6 +2162,7 @@ static void pl330_free_chan_resources(struct dma_chan *chan)
 
 	tasklet_kill(&pch->task);
 
+	pm_runtime_get_sync(pch->dmac->ddma.dev);
 	spin_lock_irqsave(&pch->lock, flags);
 
 	pl330_release_channel(pch->thread);
@@ -2147,6 +2172,7 @@ static void pl330_free_chan_resources(struct dma_chan *chan)
 		list_splice_tail_init(&pch->work_list, &pch->dmac->desc_pool);
 
 	spin_unlock_irqrestore(&pch->lock, flags);
+	pm_runtime_put(pch->dmac->ddma.dev);
 }
 
 static enum dma_status
@@ -2162,6 +2188,15 @@ static void pl330_issue_pending(struct dma_chan *chan)
 	unsigned long flags;
 
 	spin_lock_irqsave(&pch->lock, flags);
+	if (list_empty(&pch->work_list)) {
+		/*
+		 * Warn on nothing pending. Empty submitted_list may
+		 * break our pm_runtime usage counter as it is
+		 * updated on work_list emptiness status.
+		 */
+		WARN_ON(list_empty(&pch->submitted_list));
+		pm_runtime_get_sync(pch->dmac->ddma.dev);
+	}
 	list_splice_tail_init(&pch->submitted_list, &pch->work_list);
 	spin_unlock_irqrestore(&pch->lock, flags);
 
@@ -2585,6 +2620,34 @@ static int pl330_dma_device_slave_caps(struct dma_chan *dchan,
 	return 0;
 }
 
+/*
+ * Assume that IRQ safe runtime PM is chosen in probe and amba bus driver
+ * will only disable/enable the clock in runtime PM suspend/resume.
+ */
+static int __maybe_unused pl330_suspend(struct device *dev)
+{
+	struct amba_device *pcdev = to_amba_device(dev);
+
+	if (!pm_runtime_suspended(dev))
+		amba_pclk_disable(pcdev);
+	amba_pclk_unprepare(pcdev);
+
+	return 0;
+}
+
+static int __maybe_unused pl330_resume(struct device *dev)
+{
+	struct amba_device *pcdev = to_amba_device(dev);
+
+	amba_pclk_prepare(pcdev);
+	if (!pm_runtime_suspended(dev))
+		return amba_pclk_enable(pcdev);
+
+	return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(pl330_pm, pl330_suspend, pl330_resume);
+
 static int
 pl330_probe(struct amba_device *adev, const struct amba_id *id)
 {
@@ -2738,6 +2801,9 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
 		pcfg->data_buf_dep, pcfg->data_bus_width / 8, pcfg->num_chan,
 		pcfg->num_peri, pcfg->num_events);
 
+	pm_runtime_irq_safe(&adev->dev);
+	pm_runtime_put_noidle(&adev->dev);
+
 	return 0;
 probe_err3:
 	/* Idle the DMAC */
@@ -2764,6 +2830,8 @@ static int pl330_remove(struct amba_device *adev)
 	struct pl330_dmac *pl330 = amba_get_drvdata(adev);
 	struct dma_pl330_chan *pch, *_p;
 
+	pm_runtime_get_noresume(pl330->ddma.dev);
+
 	if (adev->dev.of_node)
 		of_dma_controller_free(adev->dev.of_node);
 
@@ -2802,6 +2870,7 @@ static struct amba_driver pl330_driver = {
 	.drv = {
 		.owner = THIS_MODULE,
 		.name = "dma-pl330",
+		.pm = &pl330_pm,
 	},
 	.id_table = pl330_ids,
 	.probe = pl330_probe,
-- 
1.9.1

Re: [PATCH v4 3/4] amba: Don't unprepare the clocks if device driver wants IRQ safe runtime PM

[Russell King - ARM Linux]

On Tue, Sep 16, 2014 at 10:51:33AM +0200, Krzysztof Kozlowski wrote:
> @@ -191,8 +210,11 @@ static int amba_probe(struct device *dev)
>  		pm_runtime_enable(dev);
>  
>  		ret = pcdrv->probe(pcdev, id);
> -		if (ret == 0)
> +		if (ret == 0) {
> +			if (pm_runtime_is_irq_safe(dev))
> +				pcdev->irq_safe = 1;

Patch looks good, though I'd prefer this to always explicitly initialise
pcdev->irq_safe each time a probe succeeds.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

Re: [PATCH v4 3/4] amba: Don't unprepare the clocks if device driver wants IRQ safe runtime PM

[Krzysztof Kozlowski]

On wto, 2014-09-16 at 10:20 +0100, Russell King - ARM Linux wrote:
> On Tue, Sep 16, 2014 at 10:51:33AM +0200, Krzysztof Kozlowski wrote:
> > @@ -191,8 +210,11 @@ static int amba_probe(struct device *dev)
> >  		pm_runtime_enable(dev);
> >  
> >  		ret = pcdrv->probe(pcdev, id);
> > -		if (ret == 0)
> > +		if (ret == 0) {
> > +			if (pm_runtime_is_irq_safe(dev))
> > +				pcdev->irq_safe = 1;
> 
> Patch looks good, though I'd prefer this to always explicitly initialise
> pcdev->irq_safe each time a probe succeeds.

No problem, I'll add this and resend.

Thank you for feedback,
Krzysztof

Re: [PATCH v4 4/4] dma: pl330: add Power Management support

[Ulf Hansson]

On 16 September 2014 10:51, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:
> This patch adds both normal PM suspend/resume support and runtime PM
> support to pl330 DMA engine driver.
>
> The runtime power management for pl330 DMA driver allows gating of AMBA
> clock (PDMA) in FSYS clock domain, when the device is not processing any
> requests. This is necessary to enter low power modes on Exynos SoCs
> (e.g. LPA on Exynos4x12 or W-AFTR on Exynos3250).
>
> Runtime PM resuming of the device may happen in atomic context (during
> call device_issue_pending()) so pm_runtime_irq_safe() is used. This will
> lead only to disabling/enabling of the clock but this is sufficient for
> gating the clock and for reducing energy usage.
>
> During system sleep the AMBA bus clock is also unprepared.
>
> Suggested-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
> Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
> ---
>  drivers/dma/pl330.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 73 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/dma/pl330.c b/drivers/dma/pl330.c
> index 4839bfa74a10..9c014874c33d 100644
> --- a/drivers/dma/pl330.c
> +++ b/drivers/dma/pl330.c
> @@ -27,6 +27,7 @@
>  #include <linux/of.h>
>  #include <linux/of_dma.h>
>  #include <linux/err.h>
> +#include <linux/pm_runtime.h>
>
>  #include "dmaengine.h"
>  #define PL330_MAX_CHAN         8
> @@ -1958,6 +1959,7 @@ static void pl330_tasklet(unsigned long data)
>         struct dma_pl330_chan *pch = (struct dma_pl330_chan *)data;
>         struct dma_pl330_desc *desc, *_dt;
>         unsigned long flags;
> +       bool power_down = false;
>
>         spin_lock_irqsave(&pch->lock, flags);
>
> @@ -1972,10 +1974,17 @@ static void pl330_tasklet(unsigned long data)
>         /* Try to submit a req imm. next to the last completed cookie */
>         fill_queue(pch);
>
> -       /* Make sure the PL330 Channel thread is active */
> -       spin_lock(&pch->thread->dmac->lock);
> -       _start(pch->thread);
> -       spin_unlock(&pch->thread->dmac->lock);
> +       if (list_empty(&pch->work_list)) {
> +               spin_lock(&pch->thread->dmac->lock);
> +               _stop(pch->thread);
> +               spin_unlock(&pch->thread->dmac->lock);
> +               power_down = true;
> +       } else {
> +               /* Make sure the PL330 Channel thread is active */
> +               spin_lock(&pch->thread->dmac->lock);
> +               _start(pch->thread);
> +               spin_unlock(&pch->thread->dmac->lock);
> +       }
>
>         while (!list_empty(&pch->completed_list)) {
>                 dma_async_tx_callback callback;
> @@ -1990,6 +1999,12 @@ static void pl330_tasklet(unsigned long data)
>                 if (pch->cyclic) {
>                         desc->status = PREP;
>                         list_move_tail(&desc->node, &pch->work_list);
> +                       if (power_down) {
> +                               spin_lock(&pch->thread->dmac->lock);
> +                               _start(pch->thread);
> +                               spin_unlock(&pch->thread->dmac->lock);
> +                               power_down = false;
> +                       }
>                 } else {
>                         desc->status = FREE;
>                         list_move_tail(&desc->node, &pch->dmac->desc_pool);
> @@ -2004,6 +2019,10 @@ static void pl330_tasklet(unsigned long data)
>                 }
>         }
>         spin_unlock_irqrestore(&pch->lock, flags);
> +
> +       /* If work list empty, power down */
> +       if (power_down)
> +               pm_runtime_put(pch->dmac->ddma.dev);
>  }
>
>  bool pl330_filter(struct dma_chan *chan, void *param)
> @@ -2073,6 +2092,7 @@ static int pl330_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, unsigned
>
>         switch (cmd) {
>         case DMA_TERMINATE_ALL:
> +               pm_runtime_get_sync(pl330->ddma.dev);
>                 spin_lock_irqsave(&pch->lock, flags);
>
>                 spin_lock(&pl330->lock);
> @@ -2099,10 +2119,14 @@ static int pl330_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, unsigned
>                         dma_cookie_complete(&desc->txd);
>                 }
>
> +               if (!list_empty(&pch->work_list))
> +                       pm_runtime_put(pl330->ddma.dev);
> +
>                 list_splice_tail_init(&pch->submitted_list, &pl330->desc_pool);
>                 list_splice_tail_init(&pch->work_list, &pl330->desc_pool);
>                 list_splice_tail_init(&pch->completed_list, &pl330->desc_pool);
>                 spin_unlock_irqrestore(&pch->lock, flags);
> +               pm_runtime_put(pl330->ddma.dev);
>                 break;
>         case DMA_SLAVE_CONFIG:
>                 slave_config = (struct dma_slave_config *)arg;
> @@ -2138,6 +2162,7 @@ static void pl330_free_chan_resources(struct dma_chan *chan)
>
>         tasklet_kill(&pch->task);
>
> +       pm_runtime_get_sync(pch->dmac->ddma.dev);
>         spin_lock_irqsave(&pch->lock, flags);
>
>         pl330_release_channel(pch->thread);
> @@ -2147,6 +2172,7 @@ static void pl330_free_chan_resources(struct dma_chan *chan)
>                 list_splice_tail_init(&pch->work_list, &pch->dmac->desc_pool);
>
>         spin_unlock_irqrestore(&pch->lock, flags);
> +       pm_runtime_put(pch->dmac->ddma.dev);
>  }
>
>  static enum dma_status
> @@ -2162,6 +2188,15 @@ static void pl330_issue_pending(struct dma_chan *chan)
>         unsigned long flags;
>
>         spin_lock_irqsave(&pch->lock, flags);
> +       if (list_empty(&pch->work_list)) {
> +               /*
> +                * Warn on nothing pending. Empty submitted_list may
> +                * break our pm_runtime usage counter as it is
> +                * updated on work_list emptiness status.
> +                */
> +               WARN_ON(list_empty(&pch->submitted_list));
> +               pm_runtime_get_sync(pch->dmac->ddma.dev);
> +       }
>         list_splice_tail_init(&pch->submitted_list, &pch->work_list);
>         spin_unlock_irqrestore(&pch->lock, flags);
>
> @@ -2585,6 +2620,34 @@ static int pl330_dma_device_slave_caps(struct dma_chan *dchan,
>         return 0;
>  }
>
> +/*
> + * Assume that IRQ safe runtime PM is chosen in probe and amba bus driver
> + * will only disable/enable the clock in runtime PM suspend/resume.
> + */
> +static int __maybe_unused pl330_suspend(struct device *dev)
> +{
> +       struct amba_device *pcdev = to_amba_device(dev);
> +
> +       if (!pm_runtime_suspended(dev))
> +               amba_pclk_disable(pcdev);

I would suggest to use pm_runtime_force_suspend() instead of the above.

> +       amba_pclk_unprepare(pcdev);
> +
> +       return 0;
> +}
> +
> +static int __maybe_unused pl330_resume(struct device *dev)
> +{
> +       struct amba_device *pcdev = to_amba_device(dev);
> +
> +       amba_pclk_prepare(pcdev);
> +       if (!pm_runtime_suspended(dev))
> +               return amba_pclk_enable(pcdev);

The above if statement could be replaced with pm_runtime_force_resume().

Doing that, means the amba_pclk_enable|disable() don't need to be
exported from the AMBA bus header, but entirely handled by the AMBA
bus itself.

> +
> +       return 0;
> +}
> +
> +static SIMPLE_DEV_PM_OPS(pl330_pm, pl330_suspend, pl330_resume);
> +
>  static int
>  pl330_probe(struct amba_device *adev, const struct amba_id *id)
>  {
> @@ -2738,6 +2801,9 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
>                 pcfg->data_buf_dep, pcfg->data_bus_width / 8, pcfg->num_chan,
>                 pcfg->num_peri, pcfg->num_events);
>

You need pm_runtime_set_active() here as well.

> +       pm_runtime_irq_safe(&adev->dev);
> +       pm_runtime_put_noidle(&adev->dev);

Why pm_runtime_put_noidle(), that seems like you might end up leaving
the device in active state - unless you get some request. Likely not
what you want?

A final question, have you considered using runtime PM autosuspend
feature. Any reason to why not?

> +
>         return 0;
>  probe_err3:
>         /* Idle the DMAC */
> @@ -2764,6 +2830,8 @@ static int pl330_remove(struct amba_device *adev)
>         struct pl330_dmac *pl330 = amba_get_drvdata(adev);
>         struct dma_pl330_chan *pch, *_p;
>
> +       pm_runtime_get_noresume(pl330->ddma.dev);
> +
>         if (adev->dev.of_node)
>                 of_dma_controller_free(adev->dev.of_node);
>
> @@ -2802,6 +2870,7 @@ static struct amba_driver pl330_driver = {
>         .drv = {
>                 .owner = THIS_MODULE,
>                 .name = "dma-pl330",
> +               .pm = &pl330_pm,
>         },
>         .id_table = pl330_ids,
>         .probe = pl330_probe,
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH v4 4/4] dma: pl330: add Power Management support

[Krzysztof Kozlowski]

On śro, 2014-09-17 at 20:42 +0200, Ulf Hansson wrote:
> On 16 September 2014 10:51, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:

[...]

> >
> > @@ -2585,6 +2620,34 @@ static int pl330_dma_device_slave_caps(struct dma_chan *dchan,
> >         return 0;
> >  }
> >
> > +/*
> > + * Assume that IRQ safe runtime PM is chosen in probe and amba bus driver
> > + * will only disable/enable the clock in runtime PM suspend/resume.
> > + */
> > +static int __maybe_unused pl330_suspend(struct device *dev)
> > +{
> > +       struct amba_device *pcdev = to_amba_device(dev);
> > +
> > +       if (!pm_runtime_suspended(dev))
> > +               amba_pclk_disable(pcdev);
> 
> I would suggest to use pm_runtime_force_suspend() instead of the above.

Sure, I can change it but... (see below)

> 
> > +       amba_pclk_unprepare(pcdev);
> > +
> > +       return 0;
> > +}
> > +
> > +static int __maybe_unused pl330_resume(struct device *dev)
> > +{
> > +       struct amba_device *pcdev = to_amba_device(dev);
> > +
> > +       amba_pclk_prepare(pcdev);
> > +       if (!pm_runtime_suspended(dev))
> > +               return amba_pclk_enable(pcdev);
> 
> The above if statement could be replaced with pm_runtime_force_resume().

But that would lead to runtime resuming the device even when it is not
needed. We don't have to fully wakeup the device during resume operation
when the device was runtime suspended before suspend.

> 
> Doing that, means the amba_pclk_enable|disable() don't need to be
> exported from the AMBA bus header, but entirely handled by the AMBA
> bus itself.
> 
> > +
> > +       return 0;
> > +}
> > +
> > +static SIMPLE_DEV_PM_OPS(pl330_pm, pl330_suspend, pl330_resume);
> > +
> >  static int
> >  pl330_probe(struct amba_device *adev, const struct amba_id *id)
> >  {
> > @@ -2738,6 +2801,9 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
> >                 pcfg->data_buf_dep, pcfg->data_bus_width / 8, pcfg->num_chan,
> >                 pcfg->num_peri, pcfg->num_events);
> >
> 
> You need pm_runtime_set_active() here as well.

This is done by amba/bus.c. Do I have to do it again?

> 
> > +       pm_runtime_irq_safe(&adev->dev);
> > +       pm_runtime_put_noidle(&adev->dev);
> 
> Why pm_runtime_put_noidle(), that seems like you might end up leaving
> the device in active state - unless you get some request. Likely not
> what you want?

Hmmm... I am sorry but I do not get the point here. It could be
pm_runtime_put() as well because initially we start with counter
incremented by amba/bus.c.


> A final question, have you considered using runtime PM autosuspend
> feature. Any reason to why not?

There is no reason, I just implemented the easier way. I was hoping for
such ideas of improvement. I'll add autosuspend.


Thanks for feedback, I really appreciate it!

Best regards,
Krzysztof

> 
> > +
> >         return 0;
> >  probe_err3:
> >         /* Idle the DMAC */
> > @@ -2764,6 +2830,8 @@ static int pl330_remove(struct amba_device *adev)
> >         struct pl330_dmac *pl330 = amba_get_drvdata(adev);
> >         struct dma_pl330_chan *pch, *_p;
> >
> > +       pm_runtime_get_noresume(pl330->ddma.dev);
> > +
> >         if (adev->dev.of_node)
> >                 of_dma_controller_free(adev->dev.of_node);
> >
> > @@ -2802,6 +2870,7 @@ static struct amba_driver pl330_driver = {
> >         .drv = {
> >                 .owner = THIS_MODULE,
> >                 .name = "dma-pl330",
> > +               .pm = &pl330_pm,
> >         },
> >         .id_table = pl330_ids,
> >         .probe = pl330_probe,
> > --
> > 1.9.1
> >
> 
> Kind regards
> Uffe

Re: [PATCH v4 4/4] dma: pl330: add Power Management support

[Ulf Hansson]

On 18 September 2014 12:09, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:
> On śro, 2014-09-17 at 20:42 +0200, Ulf Hansson wrote:
>> On 16 September 2014 10:51, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:
>
> [...]
>
>> >
>> > @@ -2585,6 +2620,34 @@ static int pl330_dma_device_slave_caps(struct dma_chan *dchan,
>> >         return 0;
>> >  }
>> >
>> > +/*
>> > + * Assume that IRQ safe runtime PM is chosen in probe and amba bus driver
>> > + * will only disable/enable the clock in runtime PM suspend/resume.
>> > + */
>> > +static int __maybe_unused pl330_suspend(struct device *dev)
>> > +{
>> > +       struct amba_device *pcdev = to_amba_device(dev);
>> > +
>> > +       if (!pm_runtime_suspended(dev))
>> > +               amba_pclk_disable(pcdev);
>>
>> I would suggest to use pm_runtime_force_suspend() instead of the above.
>
> Sure, I can change it but... (see below)
>
>>
>> > +       amba_pclk_unprepare(pcdev);
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static int __maybe_unused pl330_resume(struct device *dev)
>> > +{
>> > +       struct amba_device *pcdev = to_amba_device(dev);
>> > +
>> > +       amba_pclk_prepare(pcdev);
>> > +       if (!pm_runtime_suspended(dev))
>> > +               return amba_pclk_enable(pcdev);
>>
>> The above if statement could be replaced with pm_runtime_force_resume().
>
> But that would lead to runtime resuming the device even when it is not
> needed. We don't have to fully wakeup the device during resume operation
> when the device was runtime suspended before suspend.

That's true - but that's not a big issue I believe. I think the below
problems are worse. :-)

1) You don't disable runtime PM, thus not preventing the runtime PM
callbacks from being invoked.
2) You don't update the runtime PM status, thus not reflecting the
actual state of the device.

Both the above may cause unbalanced clk_enable|disable calls.

Don't get me wrong, I certainly think the "don't do runtime PM resume
during system PM resume" is interesting to solve. But that it's a
common problem and should be supported by the runtime PM API somehow
instead. I have been thinking of extending the
pm_runtime_force_resume() API to support this, let's see when that
happens. :-)

>
>>
>> Doing that, means the amba_pclk_enable|disable() don't need to be
>> exported from the AMBA bus header, but entirely handled by the AMBA
>> bus itself.
>>
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static SIMPLE_DEV_PM_OPS(pl330_pm, pl330_suspend, pl330_resume);
>> > +
>> >  static int
>> >  pl330_probe(struct amba_device *adev, const struct amba_id *id)
>> >  {
>> > @@ -2738,6 +2801,9 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
>> >                 pcfg->data_buf_dep, pcfg->data_bus_width / 8, pcfg->num_chan,
>> >                 pcfg->num_peri, pcfg->num_events);
>> >
>>
>> You need pm_runtime_set_active() here as well.
>
> This is done by amba/bus.c. Do I have to do it again?

Ohh, you are right! No need to add it!

>
>>
>> > +       pm_runtime_irq_safe(&adev->dev);
>> > +       pm_runtime_put_noidle(&adev->dev);
>>
>> Why pm_runtime_put_noidle(), that seems like you might end up leaving
>> the device in active state - unless you get some request. Likely not
>> what you want?
>
> Hmmm... I am sorry but I do not get the point here. It could be
> pm_runtime_put() as well because initially we start with counter
> incremented by amba/bus.c.

The "no_idle" version will prevent the device from going inactive,
unless a new cycle of pm_runtime_get() to pm_runtime_put() happens,
which there are no guarantees of.

Kind regards
Uffe

Re: [PATCH v4 4/4] dma: pl330: add Power Management support

[Krzysztof Kozlowski]

On wto, 2014-09-23 at 17:39 +0200, Ulf Hansson wrote:
> On 18 September 2014 12:09, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:
> > On śro, 2014-09-17 at 20:42 +0200, Ulf Hansson wrote:
> >> On 16 September 2014 10:51, Krzysztof Kozlowski <k.kozlowski@samsung.com> wrote:
> >
> > [...]
> >
> >> >
> >> > @@ -2585,6 +2620,34 @@ static int pl330_dma_device_slave_caps(struct dma_chan *dchan,
> >> >         return 0;
> >> >  }
> >> >
> >> > +/*
> >> > + * Assume that IRQ safe runtime PM is chosen in probe and amba bus driver
> >> > + * will only disable/enable the clock in runtime PM suspend/resume.
> >> > + */
> >> > +static int __maybe_unused pl330_suspend(struct device *dev)
> >> > +{
> >> > +       struct amba_device *pcdev = to_amba_device(dev);
> >> > +
> >> > +       if (!pm_runtime_suspended(dev))
> >> > +               amba_pclk_disable(pcdev);
> >>
> >> I would suggest to use pm_runtime_force_suspend() instead of the above.
> >
> > Sure, I can change it but... (see below)
> >
> >>
> >> > +       amba_pclk_unprepare(pcdev);
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int __maybe_unused pl330_resume(struct device *dev)
> >> > +{
> >> > +       struct amba_device *pcdev = to_amba_device(dev);
> >> > +
> >> > +       amba_pclk_prepare(pcdev);
> >> > +       if (!pm_runtime_suspended(dev))
> >> > +               return amba_pclk_enable(pcdev);
> >>
> >> The above if statement could be replaced with pm_runtime_force_resume().
> >
> > But that would lead to runtime resuming the device even when it is not
> > needed. We don't have to fully wakeup the device during resume operation
> > when the device was runtime suspended before suspend.
> 
> That's true - but that's not a big issue I believe. I think the below
> problems are worse. :-)
> 
> 1) You don't disable runtime PM, thus not preventing the runtime PM
> callbacks from being invoked.
> 2) You don't update the runtime PM status, thus not reflecting the
> actual state of the device.
> 
> Both the above may cause unbalanced clk_enable|disable calls.
> 
> Don't get me wrong, I certainly think the "don't do runtime PM resume
> during system PM resume" is interesting to solve. But that it's a
> common problem and should be supported by the runtime PM API somehow
> instead. I have been thinking of extending the
> pm_runtime_force_resume() API to support this, let's see when that
> happens. :-)

OK, I get the point. I'll fix this.


> >
> >>
> >> Doing that, means the amba_pclk_enable|disable() don't need to be
> >> exported from the AMBA bus header, but entirely handled by the AMBA
> >> bus itself.
> >>
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static SIMPLE_DEV_PM_OPS(pl330_pm, pl330_suspend, pl330_resume);
> >> > +
> >> >  static int
> >> >  pl330_probe(struct amba_device *adev, const struct amba_id *id)
> >> >  {
> >> > @@ -2738,6 +2801,9 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
> >> >                 pcfg->data_buf_dep, pcfg->data_bus_width / 8, pcfg->num_chan,
> >> >                 pcfg->num_peri, pcfg->num_events);
> >> >
> >>
> >> You need pm_runtime_set_active() here as well.
> >
> > This is done by amba/bus.c. Do I have to do it again?
> 
> Ohh, you are right! No need to add it!
> 
> >
> >>
> >> > +       pm_runtime_irq_safe(&adev->dev);
> >> > +       pm_runtime_put_noidle(&adev->dev);
> >>
> >> Why pm_runtime_put_noidle(), that seems like you might end up leaving
> >> the device in active state - unless you get some request. Likely not
> >> what you want?
> >
> > Hmmm... I am sorry but I do not get the point here. It could be
> > pm_runtime_put() as well because initially we start with counter
> > incremented by amba/bus.c.
> 
> The "no_idle" version will prevent the device from going inactive,
> unless a new cycle of pm_runtime_get() to pm_runtime_put() happens,
> which there are no guarantees of.

Right.

Thanks for reviewing the code. I'll send the next version removing also
the amba_pclk_enable|disable() macros.

Best regards,
Krzysztof

> 
> Kind regards
> Uffe