[PATCH V5 0/9] CPU PM domains

[PATCH V5 0/9] CPU PM domains

[Lina Iyer]

Hi all,

This is the revised patchset after good discussions at Plumbers' conference.
So far, the design fashioned a PM domain around a cluster of CPUs. After
discussions, it was agreed that each CPU will be considered a PM domain, for it
can be powered on/off individually. This series implements that change.

This set adds support for CPU PM domains. One of users of CPU PM Domains is
PSCI on ARM architectures. A following patchset will add that support.

Changes since V4[1]:
- CPU PM domains re-organized
- Update patchset to use hotplug mechanism
- Rebase on top of rwy/linux-next

The patches do the following -

#1:    Allows CPU's idle states be presented using domain-idle-states property
#2,#3: Add runtime PM support for CPU devices
#4:    Create PM domains for CPUs
#5:    Export timer for CPU wakeup
#6:    Determine CPU cluster wakeup
#7,#8: Data and DT support for CPU PM domains
#9:    Add Documentation

Thanks,
Lina

[1]. http://lists.infradead.org/pipermail/linux-arm-kernel/2016-October/463419.html

Lina Iyer (9):
  PM / Domains: Ignore domain-idle-states that are not compatible
  drivers: cpu: Setup CPU devices to do runtime PM
  kernel/cpu_pm: Add runtime PM support for CPUs
  PM / cpu_domains: Setup PM domains for CPUs/clusters
  timer: Export next wake up of a CPU
  PM / cpu_domains: Add PM Domain governor for CPUs
  PM / Domains: allow platform specific data for genpd states
  PM / cpu_domains: Initialize CPU PM domains from DT
  doc / cpu_domains: Describe CPU PM domains setup and governor

 .../devicetree/bindings/power/power_domain.txt     |   4 +-
 Documentation/power/cpu_domains.txt                | 109 +++++
 drivers/base/cpu.c                                 |  15 +
 drivers/base/power/Makefile                        |   2 +-
 drivers/base/power/cpu_domains.c                   | 480 +++++++++++++++++++++
 drivers/base/power/domain.c                        |  16 +-
 include/linux/cpu_domains.h                        |  68 +++
 include/linux/cpuhotplug.h                         |   1 +
 include/linux/pm_domain.h                          |   1 +
 include/linux/tick.h                               |  10 +
 kernel/cpu_pm.c                                    |  47 ++
 kernel/time/tick-sched.c                           |  11 +
 12 files changed, 755 insertions(+), 9 deletions(-)
 create mode 100644 Documentation/power/cpu_domains.txt
 create mode 100644 drivers/base/power/cpu_domains.c
 create mode 100644 include/linux/cpu_domains.h

-- 
2.7.4

[PATCH V5 0/9] CPU PM domains

[Lina Iyer]

Hi all,

This is the revised patchset after good discussions at Plumbers' conference.
So far, the design fashioned a PM domain around a cluster of CPUs. After
discussions, it was agreed that each CPU will be considered a PM domain, for it
can be powered on/off individually. This series implements that change.

This set adds support for CPU PM domains. One of users of CPU PM Domains is
PSCI on ARM architectures. A following patchset will add that support.

Changes since V4[1]:
- CPU PM domains re-organized
- Update patchset to use hotplug mechanism
- Rebase on top of rwy/linux-next

The patches do the following -

#1:    Allows CPU's idle states be presented using domain-idle-states property
#2,#3: Add runtime PM support for CPU devices
#4:    Create PM domains for CPUs
#5:    Export timer for CPU wakeup
#6:    Determine CPU cluster wakeup
#7,#8: Data and DT support for CPU PM domains
#9:    Add Documentation

Thanks,
Lina

[1]. http://lists.infradead.org/pipermail/linux-arm-kernel/2016-October/463419.html

Lina Iyer (9):
  PM / Domains: Ignore domain-idle-states that are not compatible
  drivers: cpu: Setup CPU devices to do runtime PM
  kernel/cpu_pm: Add runtime PM support for CPUs
  PM / cpu_domains: Setup PM domains for CPUs/clusters
  timer: Export next wake up of a CPU
  PM / cpu_domains: Add PM Domain governor for CPUs
  PM / Domains: allow platform specific data for genpd states
  PM / cpu_domains: Initialize CPU PM domains from DT
  doc / cpu_domains: Describe CPU PM domains setup and governor

 .../devicetree/bindings/power/power_domain.txt     |   4 +-
 Documentation/power/cpu_domains.txt                | 109 +++++
 drivers/base/cpu.c                                 |  15 +
 drivers/base/power/Makefile                        |   2 +-
 drivers/base/power/cpu_domains.c                   | 480 +++++++++++++++++++++
 drivers/base/power/domain.c                        |  16 +-
 include/linux/cpu_domains.h                        |  68 +++
 include/linux/cpuhotplug.h                         |   1 +
 include/linux/pm_domain.h                          |   1 +
 include/linux/tick.h                               |  10 +
 kernel/cpu_pm.c                                    |  47 ++
 kernel/time/tick-sched.c                           |  11 +
 12 files changed, 755 insertions(+), 9 deletions(-)
 create mode 100644 Documentation/power/cpu_domains.txt
 create mode 100644 drivers/base/power/cpu_domains.c
 create mode 100644 include/linux/cpu_domains.h

-- 
2.7.4

[PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Lina Iyer]

domain-idle-states property may have phandles to idle state bindings
that may not be compatible with idle state definition defined in [1].
Such phandles would just be ignored and not throw and error when read by
the domain core.

Cc: <devicetree@vger.kernel.org>
Cc: Rob Herring <robh@kernel.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
 drivers/base/power/domain.c                              | 16 +++++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt
index 723e1ad..940707d 100644
--- a/Documentation/devicetree/bindings/power/power_domain.txt
+++ b/Documentation/devicetree/bindings/power/power_domain.txt
@@ -31,7 +31,9 @@ Optional properties:
 
 - domain-idle-states : A phandle of an idle-state that shall be soaked into a
                 generic domain power state. The idle state definitions are
-                compatible with domain-idle-state specified in [1].
+                compatible with domain-idle-state specified in [1]. phandles
+                that are not compatible with domain-idle-state will be
+                ignored.
   The domain-idle-state property reflects the idle state of this PM domain and
   not the idle states of the devices or sub-domains in the PM domain. Devices
   and sub-domains have their own idle-states independent of the parent
diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index e697dec..de3e489 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2079,11 +2079,6 @@ static int genpd_parse_state(struct genpd_power_state *genpd_state,
 	int err;
 	u32 residency;
 	u32 entry_latency, exit_latency;
-	const struct of_device_id *match_id;
-
-	match_id = of_match_node(idle_state_match, state_node);
-	if (!match_id)
-		return -EINVAL;
 
 	err = of_property_read_u32(state_node, "entry-latency-us",
 						&entry_latency);
@@ -2132,6 +2127,7 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	int err, ret;
 	int count;
 	struct of_phandle_iterator it;
+	const struct of_device_id *match_id;
 
 	count = of_count_phandle_with_args(dn, "domain-idle-states", NULL);
 	if (count <= 0)
@@ -2144,6 +2140,9 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	/* Loop over the phandles until all the requested entry is found */
 	of_for_each_phandle(&it, err, dn, "domain-idle-states", NULL, 0) {
 		np = it.node;
+		match_id = of_match_node(idle_state_match, np);
+		if (!match_id)
+			continue;
 		ret = genpd_parse_state(&st[i++], np);
 		if (ret) {
 			pr_err
@@ -2155,8 +2154,11 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 		}
 	}
 
-	*n = count;
-	*states = st;
+	*n = i;
+	if (!i)
+		kfree(st);
+	else
+		*states = st;
 
 	return 0;
 }
-- 
2.7.4

[PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Lina Iyer]

domain-idle-states property may have phandles to idle state bindings
that may not be compatible with idle state definition defined in [1].
Such phandles would just be ignored and not throw and error when read by
the domain core.

Cc: <devicetree@vger.kernel.org>
Cc: Rob Herring <robh@kernel.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
 drivers/base/power/domain.c                              | 16 +++++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt
index 723e1ad..940707d 100644
--- a/Documentation/devicetree/bindings/power/power_domain.txt
+++ b/Documentation/devicetree/bindings/power/power_domain.txt
@@ -31,7 +31,9 @@ Optional properties:
 
 - domain-idle-states : A phandle of an idle-state that shall be soaked into a
                 generic domain power state. The idle state definitions are
-                compatible with domain-idle-state specified in [1].
+                compatible with domain-idle-state specified in [1]. phandles
+                that are not compatible with domain-idle-state will be
+                ignored.
   The domain-idle-state property reflects the idle state of this PM domain and
   not the idle states of the devices or sub-domains in the PM domain. Devices
   and sub-domains have their own idle-states independent of the parent
diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index e697dec..de3e489 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2079,11 +2079,6 @@ static int genpd_parse_state(struct genpd_power_state *genpd_state,
 	int err;
 	u32 residency;
 	u32 entry_latency, exit_latency;
-	const struct of_device_id *match_id;
-
-	match_id = of_match_node(idle_state_match, state_node);
-	if (!match_id)
-		return -EINVAL;
 
 	err = of_property_read_u32(state_node, "entry-latency-us",
 						&entry_latency);
@@ -2132,6 +2127,7 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	int err, ret;
 	int count;
 	struct of_phandle_iterator it;
+	const struct of_device_id *match_id;
 
 	count = of_count_phandle_with_args(dn, "domain-idle-states", NULL);
 	if (count <= 0)
@@ -2144,6 +2140,9 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	/* Loop over the phandles until all the requested entry is found */
 	of_for_each_phandle(&it, err, dn, "domain-idle-states", NULL, 0) {
 		np = it.node;
+		match_id = of_match_node(idle_state_match, np);
+		if (!match_id)
+			continue;
 		ret = genpd_parse_state(&st[i++], np);
 		if (ret) {
 			pr_err
@@ -2155,8 +2154,11 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 		}
 	}
 
-	*n = count;
-	*states = st;
+	*n = i;
+	if (!i)
+		kfree(st);
+	else
+		*states = st;
 
 	return 0;
 }
-- 
2.7.4

Re: [PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Rob Herring]

On Fri, Mar 03, 2017 at 12:41:27PM -0800, Lina Iyer wrote:
> domain-idle-states property may have phandles to idle state bindings
> that may not be compatible with idle state definition defined in [1].
> Such phandles would just be ignored and not throw and error when read by
> the domain core.
> 
> Cc: <devicetree@vger.kernel.org>
> Cc: Rob Herring <robh@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
>  drivers/base/power/domain.c                              | 16 +++++++++-------
>  2 files changed, 12 insertions(+), 8 deletions(-)

Acked-by: Rob Herring <robh@kernel.org>

[PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Rob Herring]

On Fri, Mar 03, 2017 at 12:41:27PM -0800, Lina Iyer wrote:
> domain-idle-states property may have phandles to idle state bindings
> that may not be compatible with idle state definition defined in [1].
> Such phandles would just be ignored and not throw and error when read by
> the domain core.
> 
> Cc: <devicetree@vger.kernel.org>
> Cc: Rob Herring <robh@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
>  drivers/base/power/domain.c                              | 16 +++++++++-------
>  2 files changed, 12 insertions(+), 8 deletions(-)

Acked-by: Rob Herring <robh@kernel.org>

Re: [PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> domain-idle-states property may have phandles to idle state bindings
> that may not be compatible with idle state definition defined in [1].

I can't find the reference to [1] here, perhaps just point to the DT
doc file path is better.

> Such phandles would just be ignored and not throw and error when read by
> the domain core.
>
> Cc: <devicetree@vger.kernel.org>
> Cc: Rob Herring <robh@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>

With an updated change-log, you may add:

Acked-by: Ulf Hansson <ulf.hansson@linaro.org>

Kind regards
Uffe

> ---
>  Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
>  drivers/base/power/domain.c                              | 16 +++++++++-------
>  2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt
> index 723e1ad..940707d 100644
> --- a/Documentation/devicetree/bindings/power/power_domain.txt
> +++ b/Documentation/devicetree/bindings/power/power_domain.txt
> @@ -31,7 +31,9 @@ Optional properties:
>
>  - domain-idle-states : A phandle of an idle-state that shall be soaked into a
>                  generic domain power state. The idle state definitions are
> -                compatible with domain-idle-state specified in [1].
> +                compatible with domain-idle-state specified in [1]. phandles
> +                that are not compatible with domain-idle-state will be
> +                ignored.
>    The domain-idle-state property reflects the idle state of this PM domain and
>    not the idle states of the devices or sub-domains in the PM domain. Devices
>    and sub-domains have their own idle-states independent of the parent
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index e697dec..de3e489 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -2079,11 +2079,6 @@ static int genpd_parse_state(struct genpd_power_state *genpd_state,
>         int err;
>         u32 residency;
>         u32 entry_latency, exit_latency;
> -       const struct of_device_id *match_id;
> -
> -       match_id = of_match_node(idle_state_match, state_node);
> -       if (!match_id)
> -               return -EINVAL;
>
>         err = of_property_read_u32(state_node, "entry-latency-us",
>                                                 &entry_latency);
> @@ -2132,6 +2127,7 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>         int err, ret;
>         int count;
>         struct of_phandle_iterator it;
> +       const struct of_device_id *match_id;
>
>         count = of_count_phandle_with_args(dn, "domain-idle-states", NULL);
>         if (count <= 0)
> @@ -2144,6 +2140,9 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>         /* Loop over the phandles until all the requested entry is found */
>         of_for_each_phandle(&it, err, dn, "domain-idle-states", NULL, 0) {
>                 np = it.node;
> +               match_id = of_match_node(idle_state_match, np);
> +               if (!match_id)
> +                       continue;
>                 ret = genpd_parse_state(&st[i++], np);
>                 if (ret) {
>                         pr_err
> @@ -2155,8 +2154,11 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>                 }
>         }
>
> -       *n = count;
> -       *states = st;
> +       *n = i;
> +       if (!i)
> +               kfree(st);
> +       else
> +               *states = st;
>
>         return 0;
>  }
> --
> 2.7.4
>

[PATCH V5 1/9] PM / Domains: Ignore domain-idle-states that are not compatible

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> domain-idle-states property may have phandles to idle state bindings
> that may not be compatible with idle state definition defined in [1].

I can't find the reference to [1] here, perhaps just point to the DT
doc file path is better.

> Such phandles would just be ignored and not throw and error when read by
> the domain core.
>
> Cc: <devicetree@vger.kernel.org>
> Cc: Rob Herring <robh@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>

With an updated change-log, you may add:

Acked-by: Ulf Hansson <ulf.hansson@linaro.org>

Kind regards
Uffe

> ---
>  Documentation/devicetree/bindings/power/power_domain.txt |  4 +++-
>  drivers/base/power/domain.c                              | 16 +++++++++-------
>  2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt
> index 723e1ad..940707d 100644
> --- a/Documentation/devicetree/bindings/power/power_domain.txt
> +++ b/Documentation/devicetree/bindings/power/power_domain.txt
> @@ -31,7 +31,9 @@ Optional properties:
>
>  - domain-idle-states : A phandle of an idle-state that shall be soaked into a
>                  generic domain power state. The idle state definitions are
> -                compatible with domain-idle-state specified in [1].
> +                compatible with domain-idle-state specified in [1]. phandles
> +                that are not compatible with domain-idle-state will be
> +                ignored.
>    The domain-idle-state property reflects the idle state of this PM domain and
>    not the idle states of the devices or sub-domains in the PM domain. Devices
>    and sub-domains have their own idle-states independent of the parent
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index e697dec..de3e489 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -2079,11 +2079,6 @@ static int genpd_parse_state(struct genpd_power_state *genpd_state,
>         int err;
>         u32 residency;
>         u32 entry_latency, exit_latency;
> -       const struct of_device_id *match_id;
> -
> -       match_id = of_match_node(idle_state_match, state_node);
> -       if (!match_id)
> -               return -EINVAL;
>
>         err = of_property_read_u32(state_node, "entry-latency-us",
>                                                 &entry_latency);
> @@ -2132,6 +2127,7 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>         int err, ret;
>         int count;
>         struct of_phandle_iterator it;
> +       const struct of_device_id *match_id;
>
>         count = of_count_phandle_with_args(dn, "domain-idle-states", NULL);
>         if (count <= 0)
> @@ -2144,6 +2140,9 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>         /* Loop over the phandles until all the requested entry is found */
>         of_for_each_phandle(&it, err, dn, "domain-idle-states", NULL, 0) {
>                 np = it.node;
> +               match_id = of_match_node(idle_state_match, np);
> +               if (!match_id)
> +                       continue;
>                 ret = genpd_parse_state(&st[i++], np);
>                 if (ret) {
>                         pr_err
> @@ -2155,8 +2154,11 @@ int of_genpd_parse_idle_states(struct device_node *dn,
>                 }
>         }
>
> -       *n = count;
> -       *states = st;
> +       *n = i;
> +       if (!i)
> +               kfree(st);
> +       else
> +               *states = st;
>
>         return 0;
>  }
> --
> 2.7.4
>

[PATCH V5 2/9] drivers: cpu: Setup CPU devices to do runtime PM

[Lina Iyer]

CPU devices just like any other device, can do runtime PM. However, CPU
devices may only do runtime only when IRQs are disabled. The devices
must be set as IRQ safe.

Cc: Kevin Hilman <khilman@kernel.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/cpu.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index 2c3b359..77451ad 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -18,6 +18,7 @@
 #include <linux/cpufeature.h>
 #include <linux/tick.h>
 #include <linux/pm_qos.h>
+#include <linux/pm_runtime.h>
 
 #include "base.h"
 
@@ -345,6 +346,19 @@ static int cpu_uevent(struct device *dev, struct kobj_uevent_env *env)
 }
 #endif
 
+#ifdef CONFIG_PM
+static void cpu_runtime_pm_init(struct device *dev)
+{
+	pm_runtime_irq_safe(dev);
+	pm_runtime_enable(dev);
+	if (cpu_online(dev->id))
+		pm_runtime_set_active(dev);
+}
+#else
+static void cpu_runtime_pm_init(struct device *dev)
+{ }
+#endif
+
 /*
  * register_cpu - Setup a sysfs device for a CPU.
  * @cpu - cpu->hotpluggable field set to 1 will generate a control file in
@@ -379,6 +393,7 @@ int register_cpu(struct cpu *cpu, int num)
 	register_cpu_under_node(num, cpu_to_node(num));
 	dev_pm_qos_expose_latency_limit(&cpu->dev, 0);
 
+	cpu_runtime_pm_init(&cpu->dev);
 	return 0;
 }
 
-- 
2.7.4

[PATCH V5 2/9] drivers: cpu: Setup CPU devices to do runtime PM

[Lina Iyer]

CPU devices just like any other device, can do runtime PM. However, CPU
devices may only do runtime only when IRQs are disabled. The devices
must be set as IRQ safe.

Cc: Kevin Hilman <khilman@kernel.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/cpu.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index 2c3b359..77451ad 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -18,6 +18,7 @@
 #include <linux/cpufeature.h>
 #include <linux/tick.h>
 #include <linux/pm_qos.h>
+#include <linux/pm_runtime.h>
 
 #include "base.h"
 
@@ -345,6 +346,19 @@ static int cpu_uevent(struct device *dev, struct kobj_uevent_env *env)
 }
 #endif
 
+#ifdef CONFIG_PM
+static void cpu_runtime_pm_init(struct device *dev)
+{
+	pm_runtime_irq_safe(dev);
+	pm_runtime_enable(dev);
+	if (cpu_online(dev->id))
+		pm_runtime_set_active(dev);
+}
+#else
+static void cpu_runtime_pm_init(struct device *dev)
+{ }
+#endif
+
 /*
  * register_cpu - Setup a sysfs device for a CPU.
  * @cpu - cpu->hotpluggable field set to 1 will generate a control file in
@@ -379,6 +393,7 @@ int register_cpu(struct cpu *cpu, int num)
 	register_cpu_under_node(num, cpu_to_node(num));
 	dev_pm_qos_expose_latency_limit(&cpu->dev, 0);
 
+	cpu_runtime_pm_init(&cpu->dev);
 	return 0;
 }
 
-- 
2.7.4

Re: [PATCH V5 2/9] drivers: cpu: Setup CPU devices to do runtime PM

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> CPU devices just like any other device, can do runtime PM. However, CPU
> devices may only do runtime only when IRQs are disabled. The devices
> must be set as IRQ safe.
>
> Cc: Kevin Hilman <khilman@kernel.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/cpu.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
> index 2c3b359..77451ad 100644
> --- a/drivers/base/cpu.c
> +++ b/drivers/base/cpu.c
> @@ -18,6 +18,7 @@
>  #include <linux/cpufeature.h>
>  #include <linux/tick.h>
>  #include <linux/pm_qos.h>
> +#include <linux/pm_runtime.h>
>
>  #include "base.h"
>
> @@ -345,6 +346,19 @@ static int cpu_uevent(struct device *dev, struct kobj_uevent_env *env)
>  }
>  #endif
>
> +#ifdef CONFIG_PM
> +static void cpu_runtime_pm_init(struct device *dev)
> +{
> +       pm_runtime_irq_safe(dev);
> +       pm_runtime_enable(dev);
> +       if (cpu_online(dev->id))
> +               pm_runtime_set_active(dev);

You shouldn't change the runtime PM status of the device when runtime
PM is enabled. Instead do that before you call pm_runtime_enable().

> +}
> +#else
> +static void cpu_runtime_pm_init(struct device *dev)

There is no need for a stub function, as the runtime PM API already
provides stubs for when CONFIG_PM is unset.

> +{ }
> +#endif
> +
>  /*
>   * register_cpu - Setup a sysfs device for a CPU.
>   * @cpu - cpu->hotpluggable field set to 1 will generate a control file in
> @@ -379,6 +393,7 @@ int register_cpu(struct cpu *cpu, int num)
>         register_cpu_under_node(num, cpu_to_node(num));
>         dev_pm_qos_expose_latency_limit(&cpu->dev, 0);
>
> +       cpu_runtime_pm_init(&cpu->dev);
>         return 0;
>  }
>
> --
> 2.7.4
>

Kind regards
Uffe

[PATCH V5 2/9] drivers: cpu: Setup CPU devices to do runtime PM

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> CPU devices just like any other device, can do runtime PM. However, CPU
> devices may only do runtime only when IRQs are disabled. The devices
> must be set as IRQ safe.
>
> Cc: Kevin Hilman <khilman@kernel.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/cpu.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
> index 2c3b359..77451ad 100644
> --- a/drivers/base/cpu.c
> +++ b/drivers/base/cpu.c
> @@ -18,6 +18,7 @@
>  #include <linux/cpufeature.h>
>  #include <linux/tick.h>
>  #include <linux/pm_qos.h>
> +#include <linux/pm_runtime.h>
>
>  #include "base.h"
>
> @@ -345,6 +346,19 @@ static int cpu_uevent(struct device *dev, struct kobj_uevent_env *env)
>  }
>  #endif
>
> +#ifdef CONFIG_PM
> +static void cpu_runtime_pm_init(struct device *dev)
> +{
> +       pm_runtime_irq_safe(dev);
> +       pm_runtime_enable(dev);
> +       if (cpu_online(dev->id))
> +               pm_runtime_set_active(dev);

You shouldn't change the runtime PM status of the device when runtime
PM is enabled. Instead do that before you call pm_runtime_enable().

> +}
> +#else
> +static void cpu_runtime_pm_init(struct device *dev)

There is no need for a stub function, as the runtime PM API already
provides stubs for when CONFIG_PM is unset.

> +{ }
> +#endif
> +
>  /*
>   * register_cpu - Setup a sysfs device for a CPU.
>   * @cpu - cpu->hotpluggable field set to 1 will generate a control file in
> @@ -379,6 +393,7 @@ int register_cpu(struct cpu *cpu, int num)
>         register_cpu_under_node(num, cpu_to_node(num));
>         dev_pm_qos_expose_latency_limit(&cpu->dev, 0);
>
> +       cpu_runtime_pm_init(&cpu->dev);
>         return 0;
>  }
>
> --
> 2.7.4
>

Kind regards
Uffe

[PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Lina Iyer]

Notify runtime PM when the CPU is going to be powered off in the idle
state. This allows for runtime PM suspend/resume of the CPU as well as
its PM domain.

Cc: Kevin Hilman <khilman@kernel.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/cpuhotplug.h |  1 +
 kernel/cpu_pm.c            | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 921acaa..448226a 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -79,6 +79,7 @@ enum cpuhp_state {
 	CPUHP_AP_OFFLINE,
 	CPUHP_AP_SCHED_STARTING,
 	CPUHP_AP_RCUTREE_DYING,
+	CPUHP_AP_CPU_PM_STARTING,
 	CPUHP_AP_IRQ_GIC_STARTING,
 	CPUHP_AP_IRQ_HIP04_STARTING,
 	CPUHP_AP_IRQ_ARMADA_XP_STARTING,
diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 009cc9a..db95ef3 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -16,9 +16,12 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/cpu.h>
+#include <linux/cpuhotplug.h>
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/pm_runtime.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -99,6 +102,7 @@ int cpu_pm_enter(void)
 {
 	int nr_calls;
 	int ret = 0;
+	struct device *dev = get_cpu_device(smp_processor_id());
 
 	read_lock(&cpu_pm_notifier_lock);
 	ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
@@ -110,6 +114,10 @@ int cpu_pm_enter(void)
 		cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
 	read_unlock(&cpu_pm_notifier_lock);
 
+	/* Notify Runtime PM that we are suspending the CPU */
+	if (!ret && dev)
+		RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_enter);
@@ -129,6 +137,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
 int cpu_pm_exit(void)
 {
 	int ret;
+	struct device *dev = get_cpu_device(smp_processor_id());
+
+	/* Notify Runtime PM that we are resuming the CPU */
+	if (dev)
+		RCU_NONIDLE(pm_runtime_get_sync(dev));
 
 	read_lock(&cpu_pm_notifier_lock);
 	ret = cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
@@ -200,6 +213,40 @@ int cpu_cluster_pm_exit(void)
 }
 EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit);
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int cpu_pm_cpu_dying(unsigned int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (dev)
+		pm_runtime_put_sync_suspend(dev);
+
+	return 0;
+}
+
+static int cpu_pm_cpu_starting(unsigned int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (dev)
+		pm_runtime_get_sync(dev);
+
+	return 0;
+}
+
+static int __init cpu_pm_hotplug_init(void)
+{
+	int ret;
+
+	ret = cpuhp_setup_state(CPUHP_AP_CPU_PM_STARTING,
+				"AP_CPU_PM_STARTING",
+				cpu_pm_cpu_starting, cpu_pm_cpu_dying);
+
+	return ret;
+}
+device_initcall(cpu_pm_hotplug_init);
+#endif
+
 #ifdef CONFIG_PM
 static int cpu_pm_suspend(void)
 {
-- 
2.7.4

[PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Lina Iyer]

Notify runtime PM when the CPU is going to be powered off in the idle
state. This allows for runtime PM suspend/resume of the CPU as well as
its PM domain.

Cc: Kevin Hilman <khilman@kernel.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/cpuhotplug.h |  1 +
 kernel/cpu_pm.c            | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 921acaa..448226a 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -79,6 +79,7 @@ enum cpuhp_state {
 	CPUHP_AP_OFFLINE,
 	CPUHP_AP_SCHED_STARTING,
 	CPUHP_AP_RCUTREE_DYING,
+	CPUHP_AP_CPU_PM_STARTING,
 	CPUHP_AP_IRQ_GIC_STARTING,
 	CPUHP_AP_IRQ_HIP04_STARTING,
 	CPUHP_AP_IRQ_ARMADA_XP_STARTING,
diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 009cc9a..db95ef3 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -16,9 +16,12 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/cpu.h>
+#include <linux/cpuhotplug.h>
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/pm_runtime.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -99,6 +102,7 @@ int cpu_pm_enter(void)
 {
 	int nr_calls;
 	int ret = 0;
+	struct device *dev = get_cpu_device(smp_processor_id());
 
 	read_lock(&cpu_pm_notifier_lock);
 	ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
@@ -110,6 +114,10 @@ int cpu_pm_enter(void)
 		cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
 	read_unlock(&cpu_pm_notifier_lock);
 
+	/* Notify Runtime PM that we are suspending the CPU */
+	if (!ret && dev)
+		RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_enter);
@@ -129,6 +137,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
 int cpu_pm_exit(void)
 {
 	int ret;
+	struct device *dev = get_cpu_device(smp_processor_id());
+
+	/* Notify Runtime PM that we are resuming the CPU */
+	if (dev)
+		RCU_NONIDLE(pm_runtime_get_sync(dev));
 
 	read_lock(&cpu_pm_notifier_lock);
 	ret = cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
@@ -200,6 +213,40 @@ int cpu_cluster_pm_exit(void)
 }
 EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit);
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int cpu_pm_cpu_dying(unsigned int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (dev)
+		pm_runtime_put_sync_suspend(dev);
+
+	return 0;
+}
+
+static int cpu_pm_cpu_starting(unsigned int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (dev)
+		pm_runtime_get_sync(dev);
+
+	return 0;
+}
+
+static int __init cpu_pm_hotplug_init(void)
+{
+	int ret;
+
+	ret = cpuhp_setup_state(CPUHP_AP_CPU_PM_STARTING,
+				"AP_CPU_PM_STARTING",
+				cpu_pm_cpu_starting, cpu_pm_cpu_dying);
+
+	return ret;
+}
+device_initcall(cpu_pm_hotplug_init);
+#endif
+
 #ifdef CONFIG_PM
 static int cpu_pm_suspend(void)
 {
-- 
2.7.4

Re: [PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Notify runtime PM when the CPU is going to be powered off in the idle
> state. This allows for runtime PM suspend/resume of the CPU as well as
> its PM domain.
>
> Cc: Kevin Hilman <khilman@kernel.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  include/linux/cpuhotplug.h |  1 +
>  kernel/cpu_pm.c            | 47 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 48 insertions(+)
>
> diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
> index 921acaa..448226a 100644
> --- a/include/linux/cpuhotplug.h
> +++ b/include/linux/cpuhotplug.h
> @@ -79,6 +79,7 @@ enum cpuhp_state {
>         CPUHP_AP_OFFLINE,
>         CPUHP_AP_SCHED_STARTING,
>         CPUHP_AP_RCUTREE_DYING,
> +       CPUHP_AP_CPU_PM_STARTING,
>         CPUHP_AP_IRQ_GIC_STARTING,
>         CPUHP_AP_IRQ_HIP04_STARTING,
>         CPUHP_AP_IRQ_ARMADA_XP_STARTING,
> diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
> index 009cc9a..db95ef3 100644
> --- a/kernel/cpu_pm.c
> +++ b/kernel/cpu_pm.c
> @@ -16,9 +16,12 @@
>   */
>
>  #include <linux/kernel.h>
> +#include <linux/cpu.h>
> +#include <linux/cpuhotplug.h>
>  #include <linux/cpu_pm.h>
>  #include <linux/module.h>
>  #include <linux/notifier.h>
> +#include <linux/pm_runtime.h>
>  #include <linux/spinlock.h>
>  #include <linux/syscore_ops.h>
>
> @@ -99,6 +102,7 @@ int cpu_pm_enter(void)
>  {
>         int nr_calls;
>         int ret = 0;
> +       struct device *dev = get_cpu_device(smp_processor_id());
>
>         read_lock(&cpu_pm_notifier_lock);
>         ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
> @@ -110,6 +114,10 @@ int cpu_pm_enter(void)
>                 cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
>         read_unlock(&cpu_pm_notifier_lock);
>
> +       /* Notify Runtime PM that we are suspending the CPU */
> +       if (!ret && dev)
> +               RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
> +

I am trying to understand how the runtime PM usage count becomes
properly balanced.

I believe you could you end up first calling a
pm_runtime_put_sync_suspend(), without earlier having called
pm_runtime_get*(). I am not sure though, but perhaps you can
elaborate.

Anyway, in patch2/9, where you enable runtime PM there is only a call
to pm_runtime_set_active(), which doesn't increase the usage count. To
me, it seems like that change also needs a pm_runtime_get_noresume().

>         return ret;
>  }
>  EXPORT_SYMBOL_GPL(cpu_pm_enter);
> @@ -129,6 +137,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
>  int cpu_pm_exit(void)
>  {
>         int ret;
> +       struct device *dev = get_cpu_device(smp_processor_id());
> +
> +       /* Notify Runtime PM that we are resuming the CPU */
> +       if (dev)
> +               RCU_NONIDLE(pm_runtime_get_sync(dev));
>
>         read_lock(&cpu_pm_notifier_lock);
>         ret = cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
> @@ -200,6 +213,40 @@ int cpu_cluster_pm_exit(void)
>  }
>  EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit);
>
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int cpu_pm_cpu_dying(unsigned int cpu)
> +{
> +       struct device *dev = get_cpu_device(cpu);
> +
> +       if (dev)
> +               pm_runtime_put_sync_suspend(dev);
> +
> +       return 0;
> +}
> +
> +static int cpu_pm_cpu_starting(unsigned int cpu)
> +{
> +       struct device *dev = get_cpu_device(cpu);
> +
> +       if (dev)
> +               pm_runtime_get_sync(dev);

I assume that according to my comment above, you somehow need to
compensate for either of the cases when CONFIG_HOTPLUG_CPU is set or
unset. Right?

> +
> +       return 0;
> +}
> +
> +static int __init cpu_pm_hotplug_init(void)
> +{
> +       int ret;
> +
> +       ret = cpuhp_setup_state(CPUHP_AP_CPU_PM_STARTING,
> +                               "AP_CPU_PM_STARTING",
> +                               cpu_pm_cpu_starting, cpu_pm_cpu_dying);
> +
> +       return ret;
> +}
> +device_initcall(cpu_pm_hotplug_init);
> +#endif
> +
>  #ifdef CONFIG_PM
>  static int cpu_pm_suspend(void)
>  {
> --
> 2.7.4
>

Kind regards
Uffe

[PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Notify runtime PM when the CPU is going to be powered off in the idle
> state. This allows for runtime PM suspend/resume of the CPU as well as
> its PM domain.
>
> Cc: Kevin Hilman <khilman@kernel.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  include/linux/cpuhotplug.h |  1 +
>  kernel/cpu_pm.c            | 47 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 48 insertions(+)
>
> diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
> index 921acaa..448226a 100644
> --- a/include/linux/cpuhotplug.h
> +++ b/include/linux/cpuhotplug.h
> @@ -79,6 +79,7 @@ enum cpuhp_state {
>         CPUHP_AP_OFFLINE,
>         CPUHP_AP_SCHED_STARTING,
>         CPUHP_AP_RCUTREE_DYING,
> +       CPUHP_AP_CPU_PM_STARTING,
>         CPUHP_AP_IRQ_GIC_STARTING,
>         CPUHP_AP_IRQ_HIP04_STARTING,
>         CPUHP_AP_IRQ_ARMADA_XP_STARTING,
> diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
> index 009cc9a..db95ef3 100644
> --- a/kernel/cpu_pm.c
> +++ b/kernel/cpu_pm.c
> @@ -16,9 +16,12 @@
>   */
>
>  #include <linux/kernel.h>
> +#include <linux/cpu.h>
> +#include <linux/cpuhotplug.h>
>  #include <linux/cpu_pm.h>
>  #include <linux/module.h>
>  #include <linux/notifier.h>
> +#include <linux/pm_runtime.h>
>  #include <linux/spinlock.h>
>  #include <linux/syscore_ops.h>
>
> @@ -99,6 +102,7 @@ int cpu_pm_enter(void)
>  {
>         int nr_calls;
>         int ret = 0;
> +       struct device *dev = get_cpu_device(smp_processor_id());
>
>         read_lock(&cpu_pm_notifier_lock);
>         ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
> @@ -110,6 +114,10 @@ int cpu_pm_enter(void)
>                 cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
>         read_unlock(&cpu_pm_notifier_lock);
>
> +       /* Notify Runtime PM that we are suspending the CPU */
> +       if (!ret && dev)
> +               RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
> +

I am trying to understand how the runtime PM usage count becomes
properly balanced.

I believe you could you end up first calling a
pm_runtime_put_sync_suspend(), without earlier having called
pm_runtime_get*(). I am not sure though, but perhaps you can
elaborate.

Anyway, in patch2/9, where you enable runtime PM there is only a call
to pm_runtime_set_active(), which doesn't increase the usage count. To
me, it seems like that change also needs a pm_runtime_get_noresume().

>         return ret;
>  }
>  EXPORT_SYMBOL_GPL(cpu_pm_enter);
> @@ -129,6 +137,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
>  int cpu_pm_exit(void)
>  {
>         int ret;
> +       struct device *dev = get_cpu_device(smp_processor_id());
> +
> +       /* Notify Runtime PM that we are resuming the CPU */
> +       if (dev)
> +               RCU_NONIDLE(pm_runtime_get_sync(dev));
>
>         read_lock(&cpu_pm_notifier_lock);
>         ret = cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
> @@ -200,6 +213,40 @@ int cpu_cluster_pm_exit(void)
>  }
>  EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit);
>
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int cpu_pm_cpu_dying(unsigned int cpu)
> +{
> +       struct device *dev = get_cpu_device(cpu);
> +
> +       if (dev)
> +               pm_runtime_put_sync_suspend(dev);
> +
> +       return 0;
> +}
> +
> +static int cpu_pm_cpu_starting(unsigned int cpu)
> +{
> +       struct device *dev = get_cpu_device(cpu);
> +
> +       if (dev)
> +               pm_runtime_get_sync(dev);

I assume that according to my comment above, you somehow need to
compensate for either of the cases when CONFIG_HOTPLUG_CPU is set or
unset. Right?

> +
> +       return 0;
> +}
> +
> +static int __init cpu_pm_hotplug_init(void)
> +{
> +       int ret;
> +
> +       ret = cpuhp_setup_state(CPUHP_AP_CPU_PM_STARTING,
> +                               "AP_CPU_PM_STARTING",
> +                               cpu_pm_cpu_starting, cpu_pm_cpu_dying);
> +
> +       return ret;
> +}
> +device_initcall(cpu_pm_hotplug_init);
> +#endif
> +
>  #ifdef CONFIG_PM
>  static int cpu_pm_suspend(void)
>  {
> --
> 2.7.4
>

Kind regards
Uffe

Re: [PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Kevin Hilman]

Ulf Hansson <ulf.hansson@linaro.org> writes:

> On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
>> Notify runtime PM when the CPU is going to be powered off in the idle
>> state. This allows for runtime PM suspend/resume of the CPU as well as
>> its PM domain.
>>
>> Cc: Kevin Hilman <khilman@kernel.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>

[...]

>> @@ -99,6 +102,7 @@ int cpu_pm_enter(void)
>>  {
>>         int nr_calls;
>>         int ret = 0;
>> +       struct device *dev = get_cpu_device(smp_processor_id());
>>
>>         read_lock(&cpu_pm_notifier_lock);
>>         ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
>> @@ -110,6 +114,10 @@ int cpu_pm_enter(void)
>>                 cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
>>         read_unlock(&cpu_pm_notifier_lock);
>>
>> +       /* Notify Runtime PM that we are suspending the CPU */
>> +       if (!ret && dev)
>> +               RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
>> +
>
> I am trying to understand how the runtime PM usage count becomes
> properly balanced.
>
> I believe you could you end up first calling a
> pm_runtime_put_sync_suspend(), without earlier having called
> pm_runtime_get*(). I am not sure though, but perhaps you can
> elaborate.
>
> Anyway, in patch2/9, where you enable runtime PM there is only a call
> to pm_runtime_set_active(), which doesn't increase the usage count. To
> me, it seems like that change also needs a pm_runtime_get_noresume().

IIUC, the CPU hotplug callback below (cpu_pm_cpu_starting) will do the
first _get_sync(), and I'm assuming that will happen before any of the
CPU PM notifiers get called, so I think the usecount will always be at
least 1 by the time any CPU PM callbacks happen.

[...]

>> +#ifdef CONFIG_HOTPLUG_CPU
>> +static int cpu_pm_cpu_dying(unsigned int cpu)
>> +{
>> +       struct device *dev = get_cpu_device(cpu);
>> +
>> +       if (dev)
>> +               pm_runtime_put_sync_suspend(dev);
>> +
>> +       return 0;
>> +}
>> +
>> +static int cpu_pm_cpu_starting(unsigned int cpu)
>> +{
>> +       struct device *dev = get_cpu_device(cpu);
>> +
>> +       if (dev)
>> +               pm_runtime_get_sync(dev);
>
> I assume that according to my comment above, you somehow need to
> compensate for either of the cases when CONFIG_HOTPLUG_CPU is set or
> unset. Right?

Right, if for some reason CONFIG_HOTPLUG_CPU=n, we'll have a problem
where there is never an initial _get() so the usecount will be zero when
CPU PM notifiers get called the first time.

Kevin

[PATCH V5 3/9] kernel/cpu_pm: Add runtime PM support for CPUs

[Kevin Hilman]

Ulf Hansson <ulf.hansson@linaro.org> writes:

> On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
>> Notify runtime PM when the CPU is going to be powered off in the idle
>> state. This allows for runtime PM suspend/resume of the CPU as well as
>> its PM domain.
>>
>> Cc: Kevin Hilman <khilman@kernel.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>

[...]

>> @@ -99,6 +102,7 @@ int cpu_pm_enter(void)
>>  {
>>         int nr_calls;
>>         int ret = 0;
>> +       struct device *dev = get_cpu_device(smp_processor_id());
>>
>>         read_lock(&cpu_pm_notifier_lock);
>>         ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
>> @@ -110,6 +114,10 @@ int cpu_pm_enter(void)
>>                 cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
>>         read_unlock(&cpu_pm_notifier_lock);
>>
>> +       /* Notify Runtime PM that we are suspending the CPU */
>> +       if (!ret && dev)
>> +               RCU_NONIDLE(pm_runtime_put_sync_suspend(dev));
>> +
>
> I am trying to understand how the runtime PM usage count becomes
> properly balanced.
>
> I believe you could you end up first calling a
> pm_runtime_put_sync_suspend(), without earlier having called
> pm_runtime_get*(). I am not sure though, but perhaps you can
> elaborate.
>
> Anyway, in patch2/9, where you enable runtime PM there is only a call
> to pm_runtime_set_active(), which doesn't increase the usage count. To
> me, it seems like that change also needs a pm_runtime_get_noresume().

IIUC, the CPU hotplug callback below (cpu_pm_cpu_starting) will do the
first _get_sync(), and I'm assuming that will happen before any of the
CPU PM notifiers get called, so I think the usecount will always be at
least 1 by the time any CPU PM callbacks happen.

[...]

>> +#ifdef CONFIG_HOTPLUG_CPU
>> +static int cpu_pm_cpu_dying(unsigned int cpu)
>> +{
>> +       struct device *dev = get_cpu_device(cpu);
>> +
>> +       if (dev)
>> +               pm_runtime_put_sync_suspend(dev);
>> +
>> +       return 0;
>> +}
>> +
>> +static int cpu_pm_cpu_starting(unsigned int cpu)
>> +{
>> +       struct device *dev = get_cpu_device(cpu);
>> +
>> +       if (dev)
>> +               pm_runtime_get_sync(dev);
>
> I assume that according to my comment above, you somehow need to
> compensate for either of the cases when CONFIG_HOTPLUG_CPU is set or
> unset. Right?

Right, if for some reason CONFIG_HOTPLUG_CPU=n, we'll have a problem
where there is never an initial _get() so the usecount will be zero when
CPU PM notifiers get called the first time.

Kevin

[PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Lina Iyer]

Define and add Generic PM domains (genpd) for CPU clusters. Many new
SoCs group CPUs as clusters. Clusters share common resources like power
rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
idle, these shared resources may also be put in their idle state.

CPUs may be associated with their domain providers. The domains in
turn may be associated with their providers. This is clean way to model
the cluster hierarchy like that of ARM's big.little architecture.

Platform drivers may initialize generic PM domains and setup the CPU PM
domains for the genpd and attach CPUs to the domain. In the following
patches, the CPUs are hooked up to runtime PM framework which helps
power down the domain, when all the CPUs in the domain are idle.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Suggested-by: Kevin Hilman <khilman@kernel.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/Makefile      |   2 +-
 drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
 include/linux/cpu_domains.h      |  48 ++++++++++
 3 files changed, 241 insertions(+), 1 deletion(-)
 create mode 100644 drivers/base/power/cpu_domains.c
 create mode 100644 include/linux/cpu_domains.h

diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
index 5998c53..ee383f1 100644
--- a/drivers/base/power/Makefile
+++ b/drivers/base/power/Makefile
@@ -2,7 +2,7 @@ obj-$(CONFIG_PM)	+= sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
 obj-$(CONFIG_PM_SLEEP)	+= main.o wakeup.o
 obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
 obj-$(CONFIG_PM_OPP)	+= opp/
-obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
+obj-$(CONFIG_PM_GENERIC_DOMAINS)	+= domain.o domain_governor.o cpu_domains.o
 obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
 
 ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
new file mode 100644
index 0000000..04891dc
--- /dev/null
+++ b/drivers/base/power/cpu_domains.c
@@ -0,0 +1,192 @@
+/*
+ * drivers/base/power/cpu_domains.c - Helper functions to create CPU PM domains.
+ *
+ * Copyright (C) 2016 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpumask.h>
+#include <linux/cpu_domains.h>
+#include <linux/cpu_pm.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/pm_domain.h>
+#include <linux/rculist.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+
+#define CPU_PD_NAME_MAX 36
+
+struct cpu_pm_domain {
+	struct list_head link;
+	struct cpu_pd_ops ops;
+	struct generic_pm_domain *genpd;
+	struct cpu_pm_domain *parent;
+	cpumask_var_t cpus;
+};
+
+/* List of CPU PM domains we care about */
+static LIST_HEAD(of_cpu_pd_list);
+static DEFINE_MUTEX(cpu_pd_list_lock);
+
+static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
+{
+	struct cpu_pm_domain *pd;
+	struct cpu_pm_domain *res = NULL;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
+		if (pd->genpd == d) {
+			res = pd;
+			break;
+		}
+	rcu_read_unlock();
+
+	return res;
+}
+
+static int cpu_pd_power_on(struct generic_pm_domain *genpd)
+{
+	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
+
+	return pd->ops.power_on ? pd->ops.power_on() : 0;
+}
+
+static int cpu_pd_power_off(struct generic_pm_domain *genpd)
+{
+	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
+
+	return pd->ops.power_off ? pd->ops.power_off(genpd->state_idx,
+					genpd->states[genpd->state_idx].param,
+					pd->cpus) : 0;
+}
+
+/**
+ * cpu_pd_attach_domain:  Attach a child CPU PM to its parent
+ *
+ * @parent: The parent generic PM domain
+ * @child: The child generic PM domain
+ *
+ * Generally, the child PM domain is the one to which CPUs are attached.
+ */
+int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child)
+{
+	struct cpu_pm_domain *cpu_pd, *parent_cpu_pd;
+	int ret;
+
+	ret = pm_genpd_add_subdomain(parent, child);
+	if (ret) {
+		pr_err("%s: Unable to add sub-domain (%s) to %s.\n err=%d",
+				__func__, child->name, parent->name, ret);
+		return ret;
+	}
+
+	cpu_pd = to_cpu_pd(child);
+	parent_cpu_pd = to_cpu_pd(parent);
+
+	if (cpu_pd && parent_cpu_pd)
+		cpu_pd->parent = parent_cpu_pd;
+
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_attach_domain);
+
+/**
+ * cpu_pd_attach_cpu:  Attach a CPU to its CPU PM domain.
+ *
+ * @genpd: The parent generic PM domain
+ * @cpu: The CPU number
+ */
+int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
+{
+	int ret;
+	struct device *cpu_dev;
+	struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
+
+	cpu_dev = get_cpu_device(cpu);
+	if (!cpu_dev) {
+		pr_warn("%s: Unable to get device for CPU%d\n",
+				__func__, cpu);
+		return -ENODEV;
+	}
+
+	ret = genpd_dev_pm_attach(cpu_dev);
+	if (ret)
+		dev_warn(cpu_dev,
+			"%s: Unable to attach to power-domain: %d\n",
+			__func__, ret);
+	else
+		dev_dbg(cpu_dev, "Attached to domain\n");
+
+	while (!ret && cpu_pd) {
+		cpumask_set_cpu(cpu, cpu_pd->cpus);
+		cpu_pd = cpu_pd->parent;
+	};
+
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_attach_cpu);
+
+/**
+ * cpu_pd_init: Initialize a CPU PM domain for a genpd
+ *
+ * @genpd: The initialized generic PM domain object.
+ * @ops: The power_on/power_off ops for the domain controller.
+ *
+ * Initialize a CPU PM domain based on a generic PM domain. The platform driver
+ * is expected to setup the genpd object and the states associated with the
+ * generic PM domain, before calling this function.
+ */
+int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
+{
+	int ret = -ENOMEM;
+	struct cpu_pm_domain *pd;
+
+	if (IS_ERR_OR_NULL(genpd))
+		return -EINVAL;
+
+	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+	if (!pd)
+		goto fail;
+
+	if (!zalloc_cpumask_var(&pd->cpus, GFP_KERNEL))
+		goto fail;
+
+	genpd->power_off = cpu_pd_power_off;
+	genpd->power_on = cpu_pd_power_on;
+	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
+	pd->genpd = genpd;
+	if (ops) {
+		pd->ops.power_on = ops->power_on;
+		pd->ops.power_off = ops->power_off;
+	}
+
+	INIT_LIST_HEAD_RCU(&pd->link);
+	mutex_lock(&cpu_pd_list_lock);
+	list_add_rcu(&pd->link, &of_cpu_pd_list);
+	mutex_unlock(&cpu_pd_list_lock);
+
+	ret = pm_genpd_init(genpd, &simple_qos_governor, false);
+	if (ret) {
+		pr_err("Unable to initialize domain %s\n", genpd->name);
+		goto fail;
+	}
+
+	pr_debug("adding %s as CPU PM domain\n", pd->genpd->name);
+
+	return 0;
+fail:
+	kfree(genpd->name);
+	kfree(genpd);
+	if (pd)
+		kfree(pd->cpus);
+	kfree(pd);
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_init);
diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
new file mode 100644
index 0000000..7e71291
--- /dev/null
+++ b/include/linux/cpu_domains.h
@@ -0,0 +1,48 @@
+/*
+ * include/linux/cpu_domains.h
+ *
+ * Copyright (C) 2016 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __CPU_DOMAINS_H__
+#define __CPU_DOMAINS_H__
+
+#include <linux/types.h>
+
+struct generic_pm_domain;
+struct cpumask;
+
+struct cpu_pd_ops {
+	int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
+	int (*power_on)(void);
+};
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS
+
+int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops);
+
+int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child);
+
+int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu);
+
+#else
+
+static inline int cpu_pd_init(struct generic_pm_domain *genpd,
+				const struct cpu_pd_ops *ops)
+{ return ERR_PTR(-ENODEV); }
+
+static inline int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child)
+{ return -ENODEV; }
+
+static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
+{ return -ENODEV; }
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS */
+
+#endif /* __CPU_DOMAINS_H__ */
-- 
2.7.4

[PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Lina Iyer]

Define and add Generic PM domains (genpd) for CPU clusters. Many new
SoCs group CPUs as clusters. Clusters share common resources like power
rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
idle, these shared resources may also be put in their idle state.

CPUs may be associated with their domain providers. The domains in
turn may be associated with their providers. This is clean way to model
the cluster hierarchy like that of ARM's big.little architecture.

Platform drivers may initialize generic PM domains and setup the CPU PM
domains for the genpd and attach CPUs to the domain. In the following
patches, the CPUs are hooked up to runtime PM framework which helps
power down the domain, when all the CPUs in the domain are idle.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Suggested-by: Kevin Hilman <khilman@kernel.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/Makefile      |   2 +-
 drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
 include/linux/cpu_domains.h      |  48 ++++++++++
 3 files changed, 241 insertions(+), 1 deletion(-)
 create mode 100644 drivers/base/power/cpu_domains.c
 create mode 100644 include/linux/cpu_domains.h

diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
index 5998c53..ee383f1 100644
--- a/drivers/base/power/Makefile
+++ b/drivers/base/power/Makefile
@@ -2,7 +2,7 @@ obj-$(CONFIG_PM)	+= sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
 obj-$(CONFIG_PM_SLEEP)	+= main.o wakeup.o
 obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
 obj-$(CONFIG_PM_OPP)	+= opp/
-obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
+obj-$(CONFIG_PM_GENERIC_DOMAINS)	+= domain.o domain_governor.o cpu_domains.o
 obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
 
 ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
new file mode 100644
index 0000000..04891dc
--- /dev/null
+++ b/drivers/base/power/cpu_domains.c
@@ -0,0 +1,192 @@
+/*
+ * drivers/base/power/cpu_domains.c - Helper functions to create CPU PM domains.
+ *
+ * Copyright (C) 2016 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpumask.h>
+#include <linux/cpu_domains.h>
+#include <linux/cpu_pm.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/pm_domain.h>
+#include <linux/rculist.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+
+#define CPU_PD_NAME_MAX 36
+
+struct cpu_pm_domain {
+	struct list_head link;
+	struct cpu_pd_ops ops;
+	struct generic_pm_domain *genpd;
+	struct cpu_pm_domain *parent;
+	cpumask_var_t cpus;
+};
+
+/* List of CPU PM domains we care about */
+static LIST_HEAD(of_cpu_pd_list);
+static DEFINE_MUTEX(cpu_pd_list_lock);
+
+static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
+{
+	struct cpu_pm_domain *pd;
+	struct cpu_pm_domain *res = NULL;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
+		if (pd->genpd == d) {
+			res = pd;
+			break;
+		}
+	rcu_read_unlock();
+
+	return res;
+}
+
+static int cpu_pd_power_on(struct generic_pm_domain *genpd)
+{
+	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
+
+	return pd->ops.power_on ? pd->ops.power_on() : 0;
+}
+
+static int cpu_pd_power_off(struct generic_pm_domain *genpd)
+{
+	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
+
+	return pd->ops.power_off ? pd->ops.power_off(genpd->state_idx,
+					genpd->states[genpd->state_idx].param,
+					pd->cpus) : 0;
+}
+
+/**
+ * cpu_pd_attach_domain:  Attach a child CPU PM to its parent
+ *
+ * @parent: The parent generic PM domain
+ * @child: The child generic PM domain
+ *
+ * Generally, the child PM domain is the one to which CPUs are attached.
+ */
+int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child)
+{
+	struct cpu_pm_domain *cpu_pd, *parent_cpu_pd;
+	int ret;
+
+	ret = pm_genpd_add_subdomain(parent, child);
+	if (ret) {
+		pr_err("%s: Unable to add sub-domain (%s) to %s.\n err=%d",
+				__func__, child->name, parent->name, ret);
+		return ret;
+	}
+
+	cpu_pd = to_cpu_pd(child);
+	parent_cpu_pd = to_cpu_pd(parent);
+
+	if (cpu_pd && parent_cpu_pd)
+		cpu_pd->parent = parent_cpu_pd;
+
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_attach_domain);
+
+/**
+ * cpu_pd_attach_cpu:  Attach a CPU to its CPU PM domain.
+ *
+ * @genpd: The parent generic PM domain
+ * @cpu: The CPU number
+ */
+int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
+{
+	int ret;
+	struct device *cpu_dev;
+	struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
+
+	cpu_dev = get_cpu_device(cpu);
+	if (!cpu_dev) {
+		pr_warn("%s: Unable to get device for CPU%d\n",
+				__func__, cpu);
+		return -ENODEV;
+	}
+
+	ret = genpd_dev_pm_attach(cpu_dev);
+	if (ret)
+		dev_warn(cpu_dev,
+			"%s: Unable to attach to power-domain: %d\n",
+			__func__, ret);
+	else
+		dev_dbg(cpu_dev, "Attached to domain\n");
+
+	while (!ret && cpu_pd) {
+		cpumask_set_cpu(cpu, cpu_pd->cpus);
+		cpu_pd = cpu_pd->parent;
+	};
+
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_attach_cpu);
+
+/**
+ * cpu_pd_init: Initialize a CPU PM domain for a genpd
+ *
+ * @genpd: The initialized generic PM domain object.
+ * @ops: The power_on/power_off ops for the domain controller.
+ *
+ * Initialize a CPU PM domain based on a generic PM domain. The platform driver
+ * is expected to setup the genpd object and the states associated with the
+ * generic PM domain, before calling this function.
+ */
+int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
+{
+	int ret = -ENOMEM;
+	struct cpu_pm_domain *pd;
+
+	if (IS_ERR_OR_NULL(genpd))
+		return -EINVAL;
+
+	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+	if (!pd)
+		goto fail;
+
+	if (!zalloc_cpumask_var(&pd->cpus, GFP_KERNEL))
+		goto fail;
+
+	genpd->power_off = cpu_pd_power_off;
+	genpd->power_on = cpu_pd_power_on;
+	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
+	pd->genpd = genpd;
+	if (ops) {
+		pd->ops.power_on = ops->power_on;
+		pd->ops.power_off = ops->power_off;
+	}
+
+	INIT_LIST_HEAD_RCU(&pd->link);
+	mutex_lock(&cpu_pd_list_lock);
+	list_add_rcu(&pd->link, &of_cpu_pd_list);
+	mutex_unlock(&cpu_pd_list_lock);
+
+	ret = pm_genpd_init(genpd, &simple_qos_governor, false);
+	if (ret) {
+		pr_err("Unable to initialize domain %s\n", genpd->name);
+		goto fail;
+	}
+
+	pr_debug("adding %s as CPU PM domain\n", pd->genpd->name);
+
+	return 0;
+fail:
+	kfree(genpd->name);
+	kfree(genpd);
+	if (pd)
+		kfree(pd->cpus);
+	kfree(pd);
+	return ret;
+}
+EXPORT_SYMBOL(cpu_pd_init);
diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
new file mode 100644
index 0000000..7e71291
--- /dev/null
+++ b/include/linux/cpu_domains.h
@@ -0,0 +1,48 @@
+/*
+ * include/linux/cpu_domains.h
+ *
+ * Copyright (C) 2016 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __CPU_DOMAINS_H__
+#define __CPU_DOMAINS_H__
+
+#include <linux/types.h>
+
+struct generic_pm_domain;
+struct cpumask;
+
+struct cpu_pd_ops {
+	int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
+	int (*power_on)(void);
+};
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS
+
+int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops);
+
+int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child);
+
+int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu);
+
+#else
+
+static inline int cpu_pd_init(struct generic_pm_domain *genpd,
+				const struct cpu_pd_ops *ops)
+{ return ERR_PTR(-ENODEV); }
+
+static inline int cpu_pd_attach_domain(struct generic_pm_domain *parent,
+				struct generic_pm_domain *child)
+{ return -ENODEV; }
+
+static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
+{ return -ENODEV; }
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS */
+
+#endif /* __CPU_DOMAINS_H__ */
-- 
2.7.4

Re: [PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Define and add Generic PM domains (genpd) for CPU clusters. Many new
> SoCs group CPUs as clusters. Clusters share common resources like power
> rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
> idle, these shared resources may also be put in their idle state.
>
> CPUs may be associated with their domain providers. The domains in
> turn may be associated with their providers. This is clean way to model
> the cluster hierarchy like that of ARM's big.little architecture.

Perhaps mentions this needs representation in DT.

>
> Platform drivers may initialize generic PM domains and setup the CPU PM
> domains for the genpd and attach CPUs to the domain. In the following
> patches, the CPUs are hooked up to runtime PM framework which helps
> power down the domain, when all the CPUs in the domain are idle.

I think you could elaborate a bit more on the new APIs, as to provide
a better overview of what you add in this change.

>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Suggested-by: Kevin Hilman <khilman@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile      |   2 +-
>  drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
>  include/linux/cpu_domains.h      |  48 ++++++++++
>  3 files changed, 241 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/base/power/cpu_domains.c
>  create mode 100644 include/linux/cpu_domains.h

A couple of new files.

I assume Rafael appreciate some help to maintain this, so we should
update the MAINTAINERS as well. Feel free to add my name in there as
well, if you think it makes sense.

>
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 5998c53..ee383f1 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -2,7 +2,7 @@ obj-$(CONFIG_PM)        += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
>  obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o
>  obj-$(CONFIG_PM_TRACE_RTC)     += trace.o
>  obj-$(CONFIG_PM_OPP)   += opp/
> -obj-$(CONFIG_PM_GENERIC_DOMAINS)       +=  domain.o domain_governor.o
> +obj-$(CONFIG_PM_GENERIC_DOMAINS)       += domain.o domain_governor.o cpu_domains.o
>  obj-$(CONFIG_HAVE_CLK) += clock_ops.o
>
>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
> diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
> new file mode 100644
> index 0000000..04891dc
> --- /dev/null
> +++ b/drivers/base/power/cpu_domains.c
> @@ -0,0 +1,192 @@
> +/*
> + * drivers/base/power/cpu_domains.c - Helper functions to create CPU PM domains.
> + *
> + * Copyright (C) 2016 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/cpumask.h>
> +#include <linux/cpu_domains.h>
> +#include <linux/cpu_pm.h>
> +#include <linux/device.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/pm_domain.h>
> +#include <linux/rculist.h>
> +#include <linux/rcupdate.h>
> +#include <linux/slab.h>
> +
> +#define CPU_PD_NAME_MAX 36
> +
> +struct cpu_pm_domain {
> +       struct list_head link;
> +       struct cpu_pd_ops ops;
> +       struct generic_pm_domain *genpd;
> +       struct cpu_pm_domain *parent;
> +       cpumask_var_t cpus;
> +};
> +
> +/* List of CPU PM domains we care about */
> +static LIST_HEAD(of_cpu_pd_list);
> +static DEFINE_MUTEX(cpu_pd_list_lock);
> +
> +static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
> +{
> +       struct cpu_pm_domain *pd;
> +       struct cpu_pm_domain *res = NULL;
> +
> +       rcu_read_lock();
> +       list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
> +               if (pd->genpd == d) {
> +                       res = pd;
> +                       break;
> +               }
> +       rcu_read_unlock();
> +
> +       return res;
> +}
> +
> +static int cpu_pd_power_on(struct generic_pm_domain *genpd)
> +{
> +       struct cpu_pm_domain *pd = to_cpu_pd(genpd);

I don't get why you need to walk a list of cpu_pm_domain's to find the
correct handle?

Couldn't you just do:

pd = container_of(genpd, struct cpu_pm_domain, pd);

> +
> +       return pd->ops.power_on ? pd->ops.power_on() : 0;
> +}
> +
> +static int cpu_pd_power_off(struct generic_pm_domain *genpd)
> +{
> +       struct cpu_pm_domain *pd = to_cpu_pd(genpd);

Ditto.

> +
> +       return pd->ops.power_off ? pd->ops.power_off(genpd->state_idx,
> +                                       genpd->states[genpd->state_idx].param,
> +                                       pd->cpus) : 0;
> +}
> +
> +/**
> + * cpu_pd_attach_domain:  Attach a child CPU PM to its parent
> + *
> + * @parent: The parent generic PM domain
> + * @child: The child generic PM domain

The genpd terminology are rather "masters", "subdomains" and sometimes
"slaves". Let's try to stick to that to avoid confusion. Please try to
replace this for the entire file, as I think I have seen some more
places.

> + *
> + * Generally, the child PM domain is the one to which CPUs are attached.
> + */
> +int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child)

I think the name of this function is a bit confusing. We "attach"
devices to their PM domains. But for PM domains, we usually use
add/remove instead.

Perhaps rename this function to cpu_pd_add_subdomain()?

> +{
> +       struct cpu_pm_domain *cpu_pd, *parent_cpu_pd;
> +       int ret;
> +
> +       ret = pm_genpd_add_subdomain(parent, child);

Perhaps check whether it would make better sense to use
of_genpd_add_subdomain() instead.

> +       if (ret) {
> +               pr_err("%s: Unable to add sub-domain (%s) to %s.\n err=%d",
> +                               __func__, child->name, parent->name, ret);
> +               return ret;
> +       }
> +
> +       cpu_pd = to_cpu_pd(child);
> +       parent_cpu_pd = to_cpu_pd(parent);
> +
> +       if (cpu_pd && parent_cpu_pd)
> +               cpu_pd->parent = parent_cpu_pd;

This seems like duplicating the hierarchy information, which is
already being managed by genpd.

Why do you need this?

> +
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_attach_domain);
> +
> +/**
> + * cpu_pd_attach_cpu:  Attach a CPU to its CPU PM domain.
> + *
> + * @genpd: The parent generic PM domain

Parent?

> + * @cpu: The CPU number

Maybe elaborate a bit that this uses a PM domain specifier in DT to be
able to attach the cpu device?

> + */
> +int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
> +{
> +       int ret;
> +       struct device *cpu_dev;
> +       struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
> +
> +       cpu_dev = get_cpu_device(cpu);
> +       if (!cpu_dev) {
> +               pr_warn("%s: Unable to get device for CPU%d\n",
> +                               __func__, cpu);
> +               return -ENODEV;
> +       }
> +
> +       ret = genpd_dev_pm_attach(cpu_dev);

I wonder whether of_genpd_add_device() could be a better match for this case.

> +       if (ret)
> +               dev_warn(cpu_dev,
> +                       "%s: Unable to attach to power-domain: %d\n",
> +                       __func__, ret);
> +       else
> +               dev_dbg(cpu_dev, "Attached to domain\n");
> +
> +       while (!ret && cpu_pd) {
> +               cpumask_set_cpu(cpu, cpu_pd->cpus);
> +               cpu_pd = cpu_pd->parent;
> +       };
> +
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_attach_cpu);
> +
> +/**
> + * cpu_pd_init: Initialize a CPU PM domain for a genpd
> + *
> + * @genpd: The initialized generic PM domain object.
> + * @ops: The power_on/power_off ops for the domain controller.
> + *
> + * Initialize a CPU PM domain based on a generic PM domain. The platform driver
> + * is expected to setup the genpd object and the states associated with the
> + * generic PM domain, before calling this function.
> + */
> +int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
> +{
> +       int ret = -ENOMEM;
> +       struct cpu_pm_domain *pd;
> +
> +       if (IS_ERR_OR_NULL(genpd))
> +               return -EINVAL;
> +
> +       pd = kzalloc(sizeof(*pd), GFP_KERNEL);
> +       if (!pd)
> +               goto fail;
> +
> +       if (!zalloc_cpumask_var(&pd->cpus, GFP_KERNEL))
> +               goto fail;
> +
> +       genpd->power_off = cpu_pd_power_off;
> +       genpd->power_on = cpu_pd_power_on;
> +       genpd->flags |= GENPD_FLAG_IRQ_SAFE;
> +       pd->genpd = genpd;

Ah, now I get it. This is why you can't use the container_of() thing,
but instead need to walk the cpu_pd_list at some places.

Perhaps you can fix this by adding a pair of cpu_pd_alloc|free()
functions, which deals with the allocation, including what is needed
for client. A *void pointer can be added in the cpu_pm_domain struct
to allow the client to assign private data. Do you think that would
work?

Also, perhaps that removes the need for the cpu_pd_list?

> +       if (ops) {
> +               pd->ops.power_on = ops->power_on;
> +               pd->ops.power_off = ops->power_off;
> +       }
> +
> +       INIT_LIST_HEAD_RCU(&pd->link);
> +       mutex_lock(&cpu_pd_list_lock);
> +       list_add_rcu(&pd->link, &of_cpu_pd_list);
> +       mutex_unlock(&cpu_pd_list_lock);
> +
> +       ret = pm_genpd_init(genpd, &simple_qos_governor, false);
> +       if (ret) {
> +               pr_err("Unable to initialize domain %s\n", genpd->name);
> +               goto fail;
> +       }
> +
> +       pr_debug("adding %s as CPU PM domain\n", pd->genpd->name);
> +
> +       return 0;
> +fail:
> +       kfree(genpd->name);
> +       kfree(genpd);
> +       if (pd)
> +               kfree(pd->cpus);
> +       kfree(pd);
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_init);
> diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
> new file mode 100644
> index 0000000..7e71291
> --- /dev/null
> +++ b/include/linux/cpu_domains.h
> @@ -0,0 +1,48 @@
> +/*
> + * include/linux/cpu_domains.h
> + *
> + * Copyright (C) 2016 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#ifndef __CPU_DOMAINS_H__
> +#define __CPU_DOMAINS_H__
> +
> +#include <linux/types.h>
> +
> +struct generic_pm_domain;
> +struct cpumask;
> +
> +struct cpu_pd_ops {
> +       int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
> +       int (*power_on)(void);
> +};
> +
> +#ifdef CONFIG_PM_GENERIC_DOMAINS
> +
> +int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops);
> +
> +int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child);
> +
> +int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu);
> +
> +#else
> +
> +static inline int cpu_pd_init(struct generic_pm_domain *genpd,
> +                               const struct cpu_pd_ops *ops)
> +{ return ERR_PTR(-ENODEV); }
> +
> +static inline int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child)
> +{ return -ENODEV; }
> +
> +static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
> +{ return -ENODEV; }
> +
> +#endif /* CONFIG_PM_GENERIC_DOMAINS */
> +
> +#endif /* __CPU_DOMAINS_H__ */
> --
> 2.7.4
>

Kind regards
Uffe

[PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Define and add Generic PM domains (genpd) for CPU clusters. Many new
> SoCs group CPUs as clusters. Clusters share common resources like power
> rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
> idle, these shared resources may also be put in their idle state.
>
> CPUs may be associated with their domain providers. The domains in
> turn may be associated with their providers. This is clean way to model
> the cluster hierarchy like that of ARM's big.little architecture.

Perhaps mentions this needs representation in DT.

>
> Platform drivers may initialize generic PM domains and setup the CPU PM
> domains for the genpd and attach CPUs to the domain. In the following
> patches, the CPUs are hooked up to runtime PM framework which helps
> power down the domain, when all the CPUs in the domain are idle.

I think you could elaborate a bit more on the new APIs, as to provide
a better overview of what you add in this change.

>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Suggested-by: Kevin Hilman <khilman@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile      |   2 +-
>  drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
>  include/linux/cpu_domains.h      |  48 ++++++++++
>  3 files changed, 241 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/base/power/cpu_domains.c
>  create mode 100644 include/linux/cpu_domains.h

A couple of new files.

I assume Rafael appreciate some help to maintain this, so we should
update the MAINTAINERS as well. Feel free to add my name in there as
well, if you think it makes sense.

>
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 5998c53..ee383f1 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -2,7 +2,7 @@ obj-$(CONFIG_PM)        += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
>  obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o
>  obj-$(CONFIG_PM_TRACE_RTC)     += trace.o
>  obj-$(CONFIG_PM_OPP)   += opp/
> -obj-$(CONFIG_PM_GENERIC_DOMAINS)       +=  domain.o domain_governor.o
> +obj-$(CONFIG_PM_GENERIC_DOMAINS)       += domain.o domain_governor.o cpu_domains.o
>  obj-$(CONFIG_HAVE_CLK) += clock_ops.o
>
>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
> diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
> new file mode 100644
> index 0000000..04891dc
> --- /dev/null
> +++ b/drivers/base/power/cpu_domains.c
> @@ -0,0 +1,192 @@
> +/*
> + * drivers/base/power/cpu_domains.c - Helper functions to create CPU PM domains.
> + *
> + * Copyright (C) 2016 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/cpumask.h>
> +#include <linux/cpu_domains.h>
> +#include <linux/cpu_pm.h>
> +#include <linux/device.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/pm_domain.h>
> +#include <linux/rculist.h>
> +#include <linux/rcupdate.h>
> +#include <linux/slab.h>
> +
> +#define CPU_PD_NAME_MAX 36
> +
> +struct cpu_pm_domain {
> +       struct list_head link;
> +       struct cpu_pd_ops ops;
> +       struct generic_pm_domain *genpd;
> +       struct cpu_pm_domain *parent;
> +       cpumask_var_t cpus;
> +};
> +
> +/* List of CPU PM domains we care about */
> +static LIST_HEAD(of_cpu_pd_list);
> +static DEFINE_MUTEX(cpu_pd_list_lock);
> +
> +static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
> +{
> +       struct cpu_pm_domain *pd;
> +       struct cpu_pm_domain *res = NULL;
> +
> +       rcu_read_lock();
> +       list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
> +               if (pd->genpd == d) {
> +                       res = pd;
> +                       break;
> +               }
> +       rcu_read_unlock();
> +
> +       return res;
> +}
> +
> +static int cpu_pd_power_on(struct generic_pm_domain *genpd)
> +{
> +       struct cpu_pm_domain *pd = to_cpu_pd(genpd);

I don't get why you need to walk a list of cpu_pm_domain's to find the
correct handle?

Couldn't you just do:

pd = container_of(genpd, struct cpu_pm_domain, pd);

> +
> +       return pd->ops.power_on ? pd->ops.power_on() : 0;
> +}
> +
> +static int cpu_pd_power_off(struct generic_pm_domain *genpd)
> +{
> +       struct cpu_pm_domain *pd = to_cpu_pd(genpd);

Ditto.

> +
> +       return pd->ops.power_off ? pd->ops.power_off(genpd->state_idx,
> +                                       genpd->states[genpd->state_idx].param,
> +                                       pd->cpus) : 0;
> +}
> +
> +/**
> + * cpu_pd_attach_domain:  Attach a child CPU PM to its parent
> + *
> + * @parent: The parent generic PM domain
> + * @child: The child generic PM domain

The genpd terminology are rather "masters", "subdomains" and sometimes
"slaves". Let's try to stick to that to avoid confusion. Please try to
replace this for the entire file, as I think I have seen some more
places.

> + *
> + * Generally, the child PM domain is the one to which CPUs are attached.
> + */
> +int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child)

I think the name of this function is a bit confusing. We "attach"
devices to their PM domains. But for PM domains, we usually use
add/remove instead.

Perhaps rename this function to cpu_pd_add_subdomain()?

> +{
> +       struct cpu_pm_domain *cpu_pd, *parent_cpu_pd;
> +       int ret;
> +
> +       ret = pm_genpd_add_subdomain(parent, child);

Perhaps check whether it would make better sense to use
of_genpd_add_subdomain() instead.

> +       if (ret) {
> +               pr_err("%s: Unable to add sub-domain (%s) to %s.\n err=%d",
> +                               __func__, child->name, parent->name, ret);
> +               return ret;
> +       }
> +
> +       cpu_pd = to_cpu_pd(child);
> +       parent_cpu_pd = to_cpu_pd(parent);
> +
> +       if (cpu_pd && parent_cpu_pd)
> +               cpu_pd->parent = parent_cpu_pd;

This seems like duplicating the hierarchy information, which is
already being managed by genpd.

Why do you need this?

> +
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_attach_domain);
> +
> +/**
> + * cpu_pd_attach_cpu:  Attach a CPU to its CPU PM domain.
> + *
> + * @genpd: The parent generic PM domain

Parent?

> + * @cpu: The CPU number

Maybe elaborate a bit that this uses a PM domain specifier in DT to be
able to attach the cpu device?

> + */
> +int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
> +{
> +       int ret;
> +       struct device *cpu_dev;
> +       struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
> +
> +       cpu_dev = get_cpu_device(cpu);
> +       if (!cpu_dev) {
> +               pr_warn("%s: Unable to get device for CPU%d\n",
> +                               __func__, cpu);
> +               return -ENODEV;
> +       }
> +
> +       ret = genpd_dev_pm_attach(cpu_dev);

I wonder whether of_genpd_add_device() could be a better match for this case.

> +       if (ret)
> +               dev_warn(cpu_dev,
> +                       "%s: Unable to attach to power-domain: %d\n",
> +                       __func__, ret);
> +       else
> +               dev_dbg(cpu_dev, "Attached to domain\n");
> +
> +       while (!ret && cpu_pd) {
> +               cpumask_set_cpu(cpu, cpu_pd->cpus);
> +               cpu_pd = cpu_pd->parent;
> +       };
> +
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_attach_cpu);
> +
> +/**
> + * cpu_pd_init: Initialize a CPU PM domain for a genpd
> + *
> + * @genpd: The initialized generic PM domain object.
> + * @ops: The power_on/power_off ops for the domain controller.
> + *
> + * Initialize a CPU PM domain based on a generic PM domain. The platform driver
> + * is expected to setup the genpd object and the states associated with the
> + * generic PM domain, before calling this function.
> + */
> +int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
> +{
> +       int ret = -ENOMEM;
> +       struct cpu_pm_domain *pd;
> +
> +       if (IS_ERR_OR_NULL(genpd))
> +               return -EINVAL;
> +
> +       pd = kzalloc(sizeof(*pd), GFP_KERNEL);
> +       if (!pd)
> +               goto fail;
> +
> +       if (!zalloc_cpumask_var(&pd->cpus, GFP_KERNEL))
> +               goto fail;
> +
> +       genpd->power_off = cpu_pd_power_off;
> +       genpd->power_on = cpu_pd_power_on;
> +       genpd->flags |= GENPD_FLAG_IRQ_SAFE;
> +       pd->genpd = genpd;

Ah, now I get it. This is why you can't use the container_of() thing,
but instead need to walk the cpu_pd_list at some places.

Perhaps you can fix this by adding a pair of cpu_pd_alloc|free()
functions, which deals with the allocation, including what is needed
for client. A *void pointer can be added in the cpu_pm_domain struct
to allow the client to assign private data. Do you think that would
work?

Also, perhaps that removes the need for the cpu_pd_list?

> +       if (ops) {
> +               pd->ops.power_on = ops->power_on;
> +               pd->ops.power_off = ops->power_off;
> +       }
> +
> +       INIT_LIST_HEAD_RCU(&pd->link);
> +       mutex_lock(&cpu_pd_list_lock);
> +       list_add_rcu(&pd->link, &of_cpu_pd_list);
> +       mutex_unlock(&cpu_pd_list_lock);
> +
> +       ret = pm_genpd_init(genpd, &simple_qos_governor, false);
> +       if (ret) {
> +               pr_err("Unable to initialize domain %s\n", genpd->name);
> +               goto fail;
> +       }
> +
> +       pr_debug("adding %s as CPU PM domain\n", pd->genpd->name);
> +
> +       return 0;
> +fail:
> +       kfree(genpd->name);
> +       kfree(genpd);
> +       if (pd)
> +               kfree(pd->cpus);
> +       kfree(pd);
> +       return ret;
> +}
> +EXPORT_SYMBOL(cpu_pd_init);
> diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
> new file mode 100644
> index 0000000..7e71291
> --- /dev/null
> +++ b/include/linux/cpu_domains.h
> @@ -0,0 +1,48 @@
> +/*
> + * include/linux/cpu_domains.h
> + *
> + * Copyright (C) 2016 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#ifndef __CPU_DOMAINS_H__
> +#define __CPU_DOMAINS_H__
> +
> +#include <linux/types.h>
> +
> +struct generic_pm_domain;
> +struct cpumask;
> +
> +struct cpu_pd_ops {
> +       int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
> +       int (*power_on)(void);
> +};
> +
> +#ifdef CONFIG_PM_GENERIC_DOMAINS
> +
> +int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops);
> +
> +int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child);
> +
> +int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu);
> +
> +#else
> +
> +static inline int cpu_pd_init(struct generic_pm_domain *genpd,
> +                               const struct cpu_pd_ops *ops)
> +{ return ERR_PTR(-ENODEV); }
> +
> +static inline int cpu_pd_attach_domain(struct generic_pm_domain *parent,
> +                               struct generic_pm_domain *child)
> +{ return -ENODEV; }
> +
> +static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
> +{ return -ENODEV; }
> +
> +#endif /* CONFIG_PM_GENERIC_DOMAINS */
> +
> +#endif /* __CPU_DOMAINS_H__ */
> --
> 2.7.4
>

Kind regards
Uffe

Re: [PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Define and add Generic PM domains (genpd) for CPU clusters. Many new
> SoCs group CPUs as clusters. Clusters share common resources like power
> rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
> idle, these shared resources may also be put in their idle state.
>
> CPUs may be associated with their domain providers. The domains in
> turn may be associated with their providers. This is clean way to model
> the cluster hierarchy like that of ARM's big.little architecture.
>
> Platform drivers may initialize generic PM domains and setup the CPU PM
> domains for the genpd and attach CPUs to the domain. In the following
> patches, the CPUs are hooked up to runtime PM framework which helps
> power down the domain, when all the CPUs in the domain are idle.
>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Suggested-by: Kevin Hilman <khilman@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile      |   2 +-
>  drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
>  include/linux/cpu_domains.h      |  48 ++++++++++
>  3 files changed, 241 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/base/power/cpu_domains.c
>  create mode 100644 include/linux/cpu_domains.h
>
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 5998c53..ee383f1 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -2,7 +2,7 @@ obj-$(CONFIG_PM)        += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
>  obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o
>  obj-$(CONFIG_PM_TRACE_RTC)     += trace.o
>  obj-$(CONFIG_PM_OPP)   += opp/
> -obj-$(CONFIG_PM_GENERIC_DOMAINS)       +=  domain.o domain_governor.o
> +obj-$(CONFIG_PM_GENERIC_DOMAINS)       += domain.o domain_governor.o cpu_domains.o

One more thing that I wonder about.

Isn't cpu_domains relying on CONFIG_PM_GENERIC_DOMAINS_OF to work?
Perhaps it's just better to build for that case?

[...]

Kind regards
Uffe

[PATCH V5 4/9] PM / cpu_domains: Setup PM domains for CPUs/clusters

[Ulf Hansson]

On 3 March 2017 at 21:41, Lina Iyer <lina.iyer@linaro.org> wrote:
> Define and add Generic PM domains (genpd) for CPU clusters. Many new
> SoCs group CPUs as clusters. Clusters share common resources like power
> rails, caches, VFP, Coresight etc. When all CPUs in the cluster are
> idle, these shared resources may also be put in their idle state.
>
> CPUs may be associated with their domain providers. The domains in
> turn may be associated with their providers. This is clean way to model
> the cluster hierarchy like that of ARM's big.little architecture.
>
> Platform drivers may initialize generic PM domains and setup the CPU PM
> domains for the genpd and attach CPUs to the domain. In the following
> patches, the CPUs are hooked up to runtime PM framework which helps
> power down the domain, when all the CPUs in the domain are idle.
>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Suggested-by: Kevin Hilman <khilman@kernel.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile      |   2 +-
>  drivers/base/power/cpu_domains.c | 192 +++++++++++++++++++++++++++++++++++++++
>  include/linux/cpu_domains.h      |  48 ++++++++++
>  3 files changed, 241 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/base/power/cpu_domains.c
>  create mode 100644 include/linux/cpu_domains.h
>
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 5998c53..ee383f1 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -2,7 +2,7 @@ obj-$(CONFIG_PM)        += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o
>  obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o
>  obj-$(CONFIG_PM_TRACE_RTC)     += trace.o
>  obj-$(CONFIG_PM_OPP)   += opp/
> -obj-$(CONFIG_PM_GENERIC_DOMAINS)       +=  domain.o domain_governor.o
> +obj-$(CONFIG_PM_GENERIC_DOMAINS)       += domain.o domain_governor.o cpu_domains.o

One more thing that I wonder about.

Isn't cpu_domains relying on CONFIG_PM_GENERIC_DOMAINS_OF to work?
Perhaps it's just better to build for that case?

[...]

Kind regards
Uffe

[PATCH V5 5/9] timer: Export next wake up of a CPU

[Lina Iyer]

Knowing the sleep length of the CPU is useful for the power state
determination on idle. The value is relative to the time when the call
was invoked by the CPU. This doesn't work well when there is a need to
know when the actual wakeup is.

By reading the next wake up event of a CPU, governors can determine the
first CPU to wake up (due to timer) amongst a cluster of CPUs and the
sleep time available between the last CPU to idle and the first CPU to
resume. This information is useful to determine if the caches and other
common hardware blocks can also be put in idle during this common period
of inactivity.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/tick.h     | 10 ++++++++++
 kernel/time/tick-sched.c | 11 +++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index a04fea1..2416021 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -117,6 +117,7 @@ extern void tick_nohz_idle_enter(void);
 extern void tick_nohz_idle_exit(void);
 extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
+extern ktime_t tick_nohz_get_next_wakeup(int cpu);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #else /* !CONFIG_NO_HZ_COMMON */
@@ -129,6 +130,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
 {
 	return NSEC_PER_SEC / HZ;
 }
+
+static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	ktime_t len = NSEC_PER_SEC/HZ;
+
+	/* Next wake up is the tick period, assume it starts now */
+	return ktime_add(len, ktime_get());
+}
+
 static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
 #endif /* !CONFIG_NO_HZ_COMMON */
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2c115fd..03e7c17 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -989,6 +989,17 @@ ktime_t tick_nohz_get_sleep_length(void)
 	return ts->sleep_length;
 }
 
+/**
+ * tick_nohz_get_next_wakeup - return the next wake up of the CPU
+ */
+ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	struct clock_event_device *dev =
+			per_cpu(tick_cpu_device.evtdev, cpu);
+
+	return dev->next_event;
+}
+
 static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
 {
 #ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
-- 
2.7.4

[PATCH V5 5/9] timer: Export next wake up of a CPU

[Lina Iyer]

Knowing the sleep length of the CPU is useful for the power state
determination on idle. The value is relative to the time when the call
was invoked by the CPU. This doesn't work well when there is a need to
know when the actual wakeup is.

By reading the next wake up event of a CPU, governors can determine the
first CPU to wake up (due to timer) amongst a cluster of CPUs and the
sleep time available between the last CPU to idle and the first CPU to
resume. This information is useful to determine if the caches and other
common hardware blocks can also be put in idle during this common period
of inactivity.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/tick.h     | 10 ++++++++++
 kernel/time/tick-sched.c | 11 +++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index a04fea1..2416021 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -117,6 +117,7 @@ extern void tick_nohz_idle_enter(void);
 extern void tick_nohz_idle_exit(void);
 extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
+extern ktime_t tick_nohz_get_next_wakeup(int cpu);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #else /* !CONFIG_NO_HZ_COMMON */
@@ -129,6 +130,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
 {
 	return NSEC_PER_SEC / HZ;
 }
+
+static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	ktime_t len = NSEC_PER_SEC/HZ;
+
+	/* Next wake up is the tick period, assume it starts now */
+	return ktime_add(len, ktime_get());
+}
+
 static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
 #endif /* !CONFIG_NO_HZ_COMMON */
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2c115fd..03e7c17 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -989,6 +989,17 @@ ktime_t tick_nohz_get_sleep_length(void)
 	return ts->sleep_length;
 }
 
+/**
+ * tick_nohz_get_next_wakeup - return the next wake up of the CPU
+ */
+ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	struct clock_event_device *dev =
+			per_cpu(tick_cpu_device.evtdev, cpu);
+
+	return dev->next_event;
+}
+
 static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
 {
 #ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
-- 
2.7.4

[PATCH V5 6/9] PM / cpu_domains: Add PM Domain governor for CPUs

[Lina Iyer]

A PM domain comprising of CPUs may be powered off when all the CPUs in
the domain are powered down. Powering down a CPU domain is generally a
expensive operation and therefore the power performance trade offs
should be considered. The time between the last CPU powering down and
the first CPU powering up in a domain, is the time available for the
domain to sleep. Ideally, the sleep time of the domain should fulfill
the residency requirement of the domains' idle state.

To do this effectively, read the time before the wakeup of the cluster's
CPUs and ensure that the domain's idle state sleep time guarantees the
QoS requirements of each of the CPU, the PM QoS CPU_DMA_LATENCY and the
state's residency.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/cpu_domains.c | 80 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)

diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
index 04891dc..4f2a40e 100644
--- a/drivers/base/power/cpu_domains.c
+++ b/drivers/base/power/cpu_domains.c
@@ -16,9 +16,12 @@
 #include <linux/kernel.h>
 #include <linux/list.h>
 #include <linux/pm_domain.h>
+#include <linux/pm_qos.h>
+#include <linux/pm_runtime.h>
 #include <linux/rculist.h>
 #include <linux/rcupdate.h>
 #include <linux/slab.h>
+#include <linux/tick.h>
 
 #define CPU_PD_NAME_MAX 36
 
@@ -50,6 +53,81 @@ static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
 	return res;
 }
 
+static bool cpu_pd_down_ok(struct dev_pm_domain *pd)
+{
+	struct generic_pm_domain *genpd = pd_to_genpd(pd);
+	struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
+	int qos_ns = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
+	u64 sleep_ns;
+	ktime_t earliest, next_wakeup;
+	int cpu;
+	int i;
+
+	/* Reset the last set genpd state, default to index 0 */
+	genpd->state_idx = 0;
+
+	/* We don't want to power down, if QoS is 0 */
+	if (!qos_ns)
+		return false;
+
+	/*
+	 * Find the sleep time for the cluster.
+	 * The time between now and the first wake up of any CPU that
+	 * are in this domain hierarchy is the time available for the
+	 * domain to be idle.
+	 *
+	 * We only care about the next wakeup for any online CPU in that
+	 * cluster. Hotplug off any of the CPUs that we care about will
+	 * wait on the genpd lock, until we are done. Any other CPU hotplug
+	 * is not of consequence to our sleep time.
+	 */
+	earliest = ktime_set(KTIME_SEC_MAX, 0);
+	for_each_cpu_and(cpu, cpu_pd->cpus, cpu_online_mask) {
+		next_wakeup = tick_nohz_get_next_wakeup(cpu);
+		if (earliest > next_wakeup)
+			earliest = next_wakeup;
+	}
+
+	sleep_ns = ktime_to_ns(ktime_sub(earliest, ktime_get()));
+	if (sleep_ns <= 0)
+		return false;
+
+	/*
+	 * Find the deepest sleep state that satisfies the residency
+	 * requirement and the QoS constraint
+	 */
+	for (i = genpd->state_count - 1; i >= 0; i--) {
+		u64 state_sleep_ns;
+
+		state_sleep_ns = genpd->states[i].power_off_latency_ns +
+			genpd->states[i].power_on_latency_ns +
+			genpd->states[i].residency_ns;
+
+		/*
+		 * If we can't sleep to save power in the state, move on
+		 * to the next lower idle state.
+		 */
+		if (state_sleep_ns > sleep_ns)
+			continue;
+
+		/*
+		 * We also don't want to sleep more than we should to
+		 * gaurantee QoS.
+		 */
+		if (state_sleep_ns < (qos_ns * NSEC_PER_USEC))
+			break;
+	}
+
+	if (i >= 0)
+		genpd->state_idx = i;
+
+	return (i >= 0);
+}
+
+static struct dev_power_governor cpu_pd_gov = {
+	.power_down_ok = cpu_pd_down_ok,
+};
+
 static int cpu_pd_power_on(struct generic_pm_domain *genpd)
 {
 	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
@@ -172,7 +250,7 @@ int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
 	list_add_rcu(&pd->link, &of_cpu_pd_list);
 	mutex_unlock(&cpu_pd_list_lock);
 
-	ret = pm_genpd_init(genpd, &simple_qos_governor, false);
+	ret = pm_genpd_init(genpd, &cpu_pd_gov, false);
 	if (ret) {
 		pr_err("Unable to initialize domain %s\n", genpd->name);
 		goto fail;
-- 
2.7.4

[PATCH V5 6/9] PM / cpu_domains: Add PM Domain governor for CPUs

[Lina Iyer]

A PM domain comprising of CPUs may be powered off when all the CPUs in
the domain are powered down. Powering down a CPU domain is generally a
expensive operation and therefore the power performance trade offs
should be considered. The time between the last CPU powering down and
the first CPU powering up in a domain, is the time available for the
domain to sleep. Ideally, the sleep time of the domain should fulfill
the residency requirement of the domains' idle state.

To do this effectively, read the time before the wakeup of the cluster's
CPUs and ensure that the domain's idle state sleep time guarantees the
QoS requirements of each of the CPU, the PM QoS CPU_DMA_LATENCY and the
state's residency.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/cpu_domains.c | 80 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)

diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
index 04891dc..4f2a40e 100644
--- a/drivers/base/power/cpu_domains.c
+++ b/drivers/base/power/cpu_domains.c
@@ -16,9 +16,12 @@
 #include <linux/kernel.h>
 #include <linux/list.h>
 #include <linux/pm_domain.h>
+#include <linux/pm_qos.h>
+#include <linux/pm_runtime.h>
 #include <linux/rculist.h>
 #include <linux/rcupdate.h>
 #include <linux/slab.h>
+#include <linux/tick.h>
 
 #define CPU_PD_NAME_MAX 36
 
@@ -50,6 +53,81 @@ static inline struct cpu_pm_domain *to_cpu_pd(struct generic_pm_domain *d)
 	return res;
 }
 
+static bool cpu_pd_down_ok(struct dev_pm_domain *pd)
+{
+	struct generic_pm_domain *genpd = pd_to_genpd(pd);
+	struct cpu_pm_domain *cpu_pd = to_cpu_pd(genpd);
+	int qos_ns = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
+	u64 sleep_ns;
+	ktime_t earliest, next_wakeup;
+	int cpu;
+	int i;
+
+	/* Reset the last set genpd state, default to index 0 */
+	genpd->state_idx = 0;
+
+	/* We don't want to power down, if QoS is 0 */
+	if (!qos_ns)
+		return false;
+
+	/*
+	 * Find the sleep time for the cluster.
+	 * The time between now and the first wake up of any CPU that
+	 * are in this domain hierarchy is the time available for the
+	 * domain to be idle.
+	 *
+	 * We only care about the next wakeup for any online CPU in that
+	 * cluster. Hotplug off any of the CPUs that we care about will
+	 * wait on the genpd lock, until we are done. Any other CPU hotplug
+	 * is not of consequence to our sleep time.
+	 */
+	earliest = ktime_set(KTIME_SEC_MAX, 0);
+	for_each_cpu_and(cpu, cpu_pd->cpus, cpu_online_mask) {
+		next_wakeup = tick_nohz_get_next_wakeup(cpu);
+		if (earliest > next_wakeup)
+			earliest = next_wakeup;
+	}
+
+	sleep_ns = ktime_to_ns(ktime_sub(earliest, ktime_get()));
+	if (sleep_ns <= 0)
+		return false;
+
+	/*
+	 * Find the deepest sleep state that satisfies the residency
+	 * requirement and the QoS constraint
+	 */
+	for (i = genpd->state_count - 1; i >= 0; i--) {
+		u64 state_sleep_ns;
+
+		state_sleep_ns = genpd->states[i].power_off_latency_ns +
+			genpd->states[i].power_on_latency_ns +
+			genpd->states[i].residency_ns;
+
+		/*
+		 * If we can't sleep to save power in the state, move on
+		 * to the next lower idle state.
+		 */
+		if (state_sleep_ns > sleep_ns)
+			continue;
+
+		/*
+		 * We also don't want to sleep more than we should to
+		 * gaurantee QoS.
+		 */
+		if (state_sleep_ns < (qos_ns * NSEC_PER_USEC))
+			break;
+	}
+
+	if (i >= 0)
+		genpd->state_idx = i;
+
+	return (i >= 0);
+}
+
+static struct dev_power_governor cpu_pd_gov = {
+	.power_down_ok = cpu_pd_down_ok,
+};
+
 static int cpu_pd_power_on(struct generic_pm_domain *genpd)
 {
 	struct cpu_pm_domain *pd = to_cpu_pd(genpd);
@@ -172,7 +250,7 @@ int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
 	list_add_rcu(&pd->link, &of_cpu_pd_list);
 	mutex_unlock(&cpu_pd_list_lock);
 
-	ret = pm_genpd_init(genpd, &simple_qos_governor, false);
+	ret = pm_genpd_init(genpd, &cpu_pd_gov, false);
 	if (ret) {
 		pr_err("Unable to initialize domain %s\n", genpd->name);
 		goto fail;
-- 
2.7.4

[PATCH V5 7/9] PM / Domains: allow platform specific data for genpd states

[Lina Iyer]

Allow platforms to save a state specific parameter for the genpd state.
The parameter would be used to identify the state when the platform is
called to power off the domain.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/pm_domain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 5339ed5..183995a 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -42,6 +42,7 @@ struct genpd_power_state {
 	s64 power_on_latency_ns;
 	s64 residency_ns;
 	struct fwnode_handle *fwnode;
+	u32 param;
 };
 
 struct genpd_lock_ops;
-- 
2.7.4

[PATCH V5 7/9] PM / Domains: allow platform specific data for genpd states

[Lina Iyer]

Allow platforms to save a state specific parameter for the genpd state.
The parameter would be used to identify the state when the platform is
called to power off the domain.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 include/linux/pm_domain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 5339ed5..183995a 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -42,6 +42,7 @@ struct genpd_power_state {
 	s64 power_on_latency_ns;
 	s64 residency_ns;
 	struct fwnode_handle *fwnode;
+	u32 param;
 };
 
 struct genpd_lock_ops;
-- 
2.7.4

[PATCH V5 8/9] PM / cpu_domains: Initialize CPU PM domains from DT

[Lina Iyer]

Add helper functions to parse DT and initialize the CPU PM domains and
attach CPU to their respective domains using information provided in the
DT.

For each CPU in the DT, we identify the domain provider; initialize and
register the PM domain if isn't already registered and attach all the
CPU devices to the domain. Usually, when there are multiple clusters of
CPUs, there is a top level coherency domain that is dependent on these
individual domains. All domains thus created are marked IRQ safe
automatically and therefore may be powered down when the CPUs in the
domain are powered down by cpuidle.

Cc: Kevin Hilman <khilman@kernel.org>
Suggested-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/cpu_domains.c | 210 +++++++++++++++++++++++++++++++++++++++
 include/linux/cpu_domains.h      |  20 ++++
 2 files changed, 230 insertions(+)

diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
index 4f2a40e..a566d84 100644
--- a/drivers/base/power/cpu_domains.c
+++ b/drivers/base/power/cpu_domains.c
@@ -13,8 +13,10 @@
 #include <linux/cpu_domains.h>
 #include <linux/cpu_pm.h>
 #include <linux/device.h>
+#include <linux/fwnode.h>
 #include <linux/kernel.h>
 #include <linux/list.h>
+#include <linux/of.h>
 #include <linux/pm_domain.h>
 #include <linux/pm_qos.h>
 #include <linux/pm_runtime.h>
@@ -268,3 +270,211 @@ int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
 	return ret;
 }
 EXPORT_SYMBOL(cpu_pd_init);
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+
+static struct generic_pm_domain *of_get_genpd(struct device_node *dn)
+{
+	struct cpu_pm_domain *pd;
+	struct generic_pm_domain *genpd = NULL;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
+		if (pd->genpd->provider == &dn->fwnode) {
+			genpd = pd->genpd;
+			break;
+		}
+	rcu_read_unlock();
+
+	return genpd;
+}
+
+static struct generic_pm_domain *alloc_genpd(struct device_node *dn)
+{
+	struct generic_pm_domain *genpd;
+
+	genpd = kzalloc(sizeof(*genpd), GFP_KERNEL);
+	if (!genpd)
+		return ERR_PTR(-ENOMEM);
+
+	genpd->name = kstrndup(dn->full_name, CPU_PD_NAME_MAX, GFP_KERNEL);
+	if (!genpd->name) {
+		kfree(genpd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	return genpd;
+}
+
+/**
+ * of_init_cpu_pm_domain() - Initialize a CPU PM domain from a device node
+ *
+ * @dn: The domain provider's device node
+ * @ops: The power_on/_off callbacks for the domain
+ *
+ * Returns the generic_pm_domain (genpd) pointer to the domain on success
+ */
+static struct generic_pm_domain *of_init_cpu_pm_domain(struct device_node *dn,
+				const struct cpu_pd_ops *plat_ops)
+{
+	struct cpu_pm_domain *pd = NULL;
+	struct generic_pm_domain *genpd = NULL;
+	int ret = -ENOMEM;
+	struct genpd_power_state *states = NULL;
+	const struct cpu_pd_ops *ops = plat_ops;
+	int count;
+	int i;
+
+	if (!of_device_is_available(dn))
+		return ERR_PTR(-ENODEV);
+
+	/* If we already have the PM domain return that */
+	genpd = of_get_genpd(dn);
+	if (genpd)
+		return genpd;
+
+	/* Initialize a new PM Domains */
+	genpd = alloc_genpd(dn);
+	if (IS_ERR(genpd))
+		return genpd;
+
+	ret = of_genpd_parse_idle_states(dn, &states, &count);
+	if (ret)
+		goto fail;
+	if (!count) {
+		ops = NULL;
+		goto skip_states;
+	}
+
+	/* Populate platform specific states from DT */
+	for (i = 0; ops->populate_state_data && i < count; i++) {
+		ret = ops->populate_state_data(to_of_node(states[i].fwnode),
+						&states[i].param);
+		if (ret)
+			goto fail;
+	}
+
+	genpd->states = states;
+	genpd->state_count = count;
+
+skip_states:
+	ret = cpu_pd_init(genpd, ops);
+	if (ret)
+		goto fail;
+
+	ret = of_genpd_add_provider_simple(dn, genpd);
+	if (ret)
+		pr_warn("Unable to add genpd %s as provider\n",
+				pd->genpd->name);
+
+	return genpd;
+fail:
+	kfree(genpd->name);
+	kfree(genpd);
+	if (pd)
+		kfree(pd->cpus);
+	kfree(pd);
+	return ERR_PTR(ret);
+}
+
+static struct generic_pm_domain *of_get_cpu_domain(struct device_node *dn,
+		const struct cpu_pd_ops *ops, int cpu)
+{
+	struct of_phandle_args args;
+	struct generic_pm_domain *genpd, *parent;
+	int ret;
+
+	genpd = of_init_cpu_pm_domain(dn, ops);
+	if (IS_ERR(genpd))
+		return genpd;
+
+	/* Is there a domain provider for this domain? */
+	ret = of_parse_phandle_with_args(dn, "power-domains",
+					"#power-domain-cells", 0, &args);
+	if (ret < 0)
+		goto skip_parent;
+
+	/* Find its parent and attach this domain to it, recursively */
+	parent = of_get_cpu_domain(args.np, ops, cpu);
+	if (IS_ERR(parent))
+		goto skip_parent;
+
+	ret = cpu_pd_attach_domain(parent, genpd);
+	if (ret)
+		pr_err("Unable to attach domain %s to parent %s\n",
+				genpd->name, parent->name);
+
+skip_parent:
+	of_node_put(dn);
+	return genpd;
+}
+
+/**
+ * of_setup_cpu_pd_single() - Setup the PM domains for a CPU
+ *
+ * @cpu: The CPU for which the PM domain is to be set up.
+ * @ops: The PM domain suspend/resume ops for the CPU's domain
+ *
+ * If the CPU PM domain exists already, then the CPU is attached to
+ * that CPU PD. If it doesn't, the domain is created, the @ops are
+ * set for power_on/power_off callbacks and then the CPU is attached
+ * to that domain. If the domain was created outside this framework,
+ * then we do not attach the CPU to the domain.
+ */
+int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops)
+{
+
+	struct device_node *dn, *np;
+	struct generic_pm_domain *genpd;
+	struct cpu_pm_domain *cpu_pd;
+
+	np = of_get_cpu_node(cpu, NULL);
+	if (!np)
+		return -ENODEV;
+
+	dn = of_parse_phandle(np, "power-domains", 0);
+	of_node_put(np);
+	if (!dn)
+		return -ENODEV;
+
+	/* Find the genpd for this CPU, create if not found */
+	genpd = of_get_cpu_domain(dn, ops, cpu);
+	of_node_put(dn);
+	if (IS_ERR(genpd))
+		return PTR_ERR(genpd);
+
+	cpu_pd = to_cpu_pd(genpd);
+	if (!cpu_pd) {
+		pr_err("%s: Genpd was created outside CPU PM domains\n",
+				__func__);
+		return -ENOENT;
+	}
+
+	return cpu_pd_attach_cpu(genpd, cpu);
+}
+EXPORT_SYMBOL(of_setup_cpu_pd_single);
+
+/**
+ * of_setup_cpu_pd() - Setup the PM domains for all CPUs
+ *
+ * @ops: The PM domain suspend/resume ops for all the domains
+ *
+ * Setup the CPU PM domain and attach all possible CPUs to their respective
+ * domains. The domains are created if not already and then attached.
+ */
+int of_setup_cpu_pd(const struct cpu_pd_ops *ops)
+{
+	int cpu;
+	int ret;
+
+	for_each_possible_cpu(cpu) {
+		ret = of_setup_cpu_pd_single(cpu, ops);
+		if (ret)
+			break;
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(of_setup_cpu_pd);
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS_OF */
diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
index 7e71291..251fbc2 100644
--- a/include/linux/cpu_domains.h
+++ b/include/linux/cpu_domains.h
@@ -15,8 +15,12 @@
 
 struct generic_pm_domain;
 struct cpumask;
+struct device_node;
 
 struct cpu_pd_ops {
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+	int (*populate_state_data)(struct device_node *n, u32 *param);
+#endif
 	int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
 	int (*power_on)(void);
 };
@@ -45,4 +49,20 @@ static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
 
 #endif /* CONFIG_PM_GENERIC_DOMAINS */
 
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+
+int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops);
+
+int of_setup_cpu_pd(const struct cpu_pd_ops *ops);
+
+#else
+
+static inline int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops)
+{ return -ENODEV; }
+
+static inline int of_setup_cpu_pd(const struct cpu_pd_ops *ops)
+{ return -ENODEV; }
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS_OF */
+
 #endif /* __CPU_DOMAINS_H__ */
-- 
2.7.4

[PATCH V5 8/9] PM / cpu_domains: Initialize CPU PM domains from DT

[Lina Iyer]

Add helper functions to parse DT and initialize the CPU PM domains and
attach CPU to their respective domains using information provided in the
DT.

For each CPU in the DT, we identify the domain provider; initialize and
register the PM domain if isn't already registered and attach all the
CPU devices to the domain. Usually, when there are multiple clusters of
CPUs, there is a top level coherency domain that is dependent on these
individual domains. All domains thus created are marked IRQ safe
automatically and therefore may be powered down when the CPUs in the
domain are powered down by cpuidle.

Cc: Kevin Hilman <khilman@kernel.org>
Suggested-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/cpu_domains.c | 210 +++++++++++++++++++++++++++++++++++++++
 include/linux/cpu_domains.h      |  20 ++++
 2 files changed, 230 insertions(+)

diff --git a/drivers/base/power/cpu_domains.c b/drivers/base/power/cpu_domains.c
index 4f2a40e..a566d84 100644
--- a/drivers/base/power/cpu_domains.c
+++ b/drivers/base/power/cpu_domains.c
@@ -13,8 +13,10 @@
 #include <linux/cpu_domains.h>
 #include <linux/cpu_pm.h>
 #include <linux/device.h>
+#include <linux/fwnode.h>
 #include <linux/kernel.h>
 #include <linux/list.h>
+#include <linux/of.h>
 #include <linux/pm_domain.h>
 #include <linux/pm_qos.h>
 #include <linux/pm_runtime.h>
@@ -268,3 +270,211 @@ int cpu_pd_init(struct generic_pm_domain *genpd, const struct cpu_pd_ops *ops)
 	return ret;
 }
 EXPORT_SYMBOL(cpu_pd_init);
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+
+static struct generic_pm_domain *of_get_genpd(struct device_node *dn)
+{
+	struct cpu_pm_domain *pd;
+	struct generic_pm_domain *genpd = NULL;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(pd, &of_cpu_pd_list, link)
+		if (pd->genpd->provider == &dn->fwnode) {
+			genpd = pd->genpd;
+			break;
+		}
+	rcu_read_unlock();
+
+	return genpd;
+}
+
+static struct generic_pm_domain *alloc_genpd(struct device_node *dn)
+{
+	struct generic_pm_domain *genpd;
+
+	genpd = kzalloc(sizeof(*genpd), GFP_KERNEL);
+	if (!genpd)
+		return ERR_PTR(-ENOMEM);
+
+	genpd->name = kstrndup(dn->full_name, CPU_PD_NAME_MAX, GFP_KERNEL);
+	if (!genpd->name) {
+		kfree(genpd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	return genpd;
+}
+
+/**
+ * of_init_cpu_pm_domain() - Initialize a CPU PM domain from a device node
+ *
+ * @dn: The domain provider's device node
+ * @ops: The power_on/_off callbacks for the domain
+ *
+ * Returns the generic_pm_domain (genpd) pointer to the domain on success
+ */
+static struct generic_pm_domain *of_init_cpu_pm_domain(struct device_node *dn,
+				const struct cpu_pd_ops *plat_ops)
+{
+	struct cpu_pm_domain *pd = NULL;
+	struct generic_pm_domain *genpd = NULL;
+	int ret = -ENOMEM;
+	struct genpd_power_state *states = NULL;
+	const struct cpu_pd_ops *ops = plat_ops;
+	int count;
+	int i;
+
+	if (!of_device_is_available(dn))
+		return ERR_PTR(-ENODEV);
+
+	/* If we already have the PM domain return that */
+	genpd = of_get_genpd(dn);
+	if (genpd)
+		return genpd;
+
+	/* Initialize a new PM Domains */
+	genpd = alloc_genpd(dn);
+	if (IS_ERR(genpd))
+		return genpd;
+
+	ret = of_genpd_parse_idle_states(dn, &states, &count);
+	if (ret)
+		goto fail;
+	if (!count) {
+		ops = NULL;
+		goto skip_states;
+	}
+
+	/* Populate platform specific states from DT */
+	for (i = 0; ops->populate_state_data && i < count; i++) {
+		ret = ops->populate_state_data(to_of_node(states[i].fwnode),
+						&states[i].param);
+		if (ret)
+			goto fail;
+	}
+
+	genpd->states = states;
+	genpd->state_count = count;
+
+skip_states:
+	ret = cpu_pd_init(genpd, ops);
+	if (ret)
+		goto fail;
+
+	ret = of_genpd_add_provider_simple(dn, genpd);
+	if (ret)
+		pr_warn("Unable to add genpd %s as provider\n",
+				pd->genpd->name);
+
+	return genpd;
+fail:
+	kfree(genpd->name);
+	kfree(genpd);
+	if (pd)
+		kfree(pd->cpus);
+	kfree(pd);
+	return ERR_PTR(ret);
+}
+
+static struct generic_pm_domain *of_get_cpu_domain(struct device_node *dn,
+		const struct cpu_pd_ops *ops, int cpu)
+{
+	struct of_phandle_args args;
+	struct generic_pm_domain *genpd, *parent;
+	int ret;
+
+	genpd = of_init_cpu_pm_domain(dn, ops);
+	if (IS_ERR(genpd))
+		return genpd;
+
+	/* Is there a domain provider for this domain? */
+	ret = of_parse_phandle_with_args(dn, "power-domains",
+					"#power-domain-cells", 0, &args);
+	if (ret < 0)
+		goto skip_parent;
+
+	/* Find its parent and attach this domain to it, recursively */
+	parent = of_get_cpu_domain(args.np, ops, cpu);
+	if (IS_ERR(parent))
+		goto skip_parent;
+
+	ret = cpu_pd_attach_domain(parent, genpd);
+	if (ret)
+		pr_err("Unable to attach domain %s to parent %s\n",
+				genpd->name, parent->name);
+
+skip_parent:
+	of_node_put(dn);
+	return genpd;
+}
+
+/**
+ * of_setup_cpu_pd_single() - Setup the PM domains for a CPU
+ *
+ * @cpu: The CPU for which the PM domain is to be set up.
+ * @ops: The PM domain suspend/resume ops for the CPU's domain
+ *
+ * If the CPU PM domain exists already, then the CPU is attached to
+ * that CPU PD. If it doesn't, the domain is created, the @ops are
+ * set for power_on/power_off callbacks and then the CPU is attached
+ * to that domain. If the domain was created outside this framework,
+ * then we do not attach the CPU to the domain.
+ */
+int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops)
+{
+
+	struct device_node *dn, *np;
+	struct generic_pm_domain *genpd;
+	struct cpu_pm_domain *cpu_pd;
+
+	np = of_get_cpu_node(cpu, NULL);
+	if (!np)
+		return -ENODEV;
+
+	dn = of_parse_phandle(np, "power-domains", 0);
+	of_node_put(np);
+	if (!dn)
+		return -ENODEV;
+
+	/* Find the genpd for this CPU, create if not found */
+	genpd = of_get_cpu_domain(dn, ops, cpu);
+	of_node_put(dn);
+	if (IS_ERR(genpd))
+		return PTR_ERR(genpd);
+
+	cpu_pd = to_cpu_pd(genpd);
+	if (!cpu_pd) {
+		pr_err("%s: Genpd was created outside CPU PM domains\n",
+				__func__);
+		return -ENOENT;
+	}
+
+	return cpu_pd_attach_cpu(genpd, cpu);
+}
+EXPORT_SYMBOL(of_setup_cpu_pd_single);
+
+/**
+ * of_setup_cpu_pd() - Setup the PM domains for all CPUs
+ *
+ * @ops: The PM domain suspend/resume ops for all the domains
+ *
+ * Setup the CPU PM domain and attach all possible CPUs to their respective
+ * domains. The domains are created if not already and then attached.
+ */
+int of_setup_cpu_pd(const struct cpu_pd_ops *ops)
+{
+	int cpu;
+	int ret;
+
+	for_each_possible_cpu(cpu) {
+		ret = of_setup_cpu_pd_single(cpu, ops);
+		if (ret)
+			break;
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(of_setup_cpu_pd);
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS_OF */
diff --git a/include/linux/cpu_domains.h b/include/linux/cpu_domains.h
index 7e71291..251fbc2 100644
--- a/include/linux/cpu_domains.h
+++ b/include/linux/cpu_domains.h
@@ -15,8 +15,12 @@
 
 struct generic_pm_domain;
 struct cpumask;
+struct device_node;
 
 struct cpu_pd_ops {
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+	int (*populate_state_data)(struct device_node *n, u32 *param);
+#endif
 	int (*power_off)(u32 state_idx, u32 param, const struct cpumask *mask);
 	int (*power_on)(void);
 };
@@ -45,4 +49,20 @@ static inline int cpu_pd_attach_cpu(struct generic_pm_domain *genpd, int cpu)
 
 #endif /* CONFIG_PM_GENERIC_DOMAINS */
 
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+
+int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops);
+
+int of_setup_cpu_pd(const struct cpu_pd_ops *ops);
+
+#else
+
+static inline int of_setup_cpu_pd_single(int cpu, const struct cpu_pd_ops *ops)
+{ return -ENODEV; }
+
+static inline int of_setup_cpu_pd(const struct cpu_pd_ops *ops)
+{ return -ENODEV; }
+
+#endif /* CONFIG_PM_GENERIC_DOMAINS_OF */
+
 #endif /* __CPU_DOMAINS_H__ */
-- 
2.7.4

[PATCH V5 9/9] doc / cpu_domains: Describe CPU PM domains setup and governor

[Lina Iyer]

A generic CPU PM domain functionality is provided by
drivers/base/power/cpu_domains.c. This document describes the generic
usecase of CPU's PM domains, the setup of such domains and a CPU
specific genpd governor.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 Documentation/power/cpu_domains.txt | 109 ++++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)
 create mode 100644 Documentation/power/cpu_domains.txt

diff --git a/Documentation/power/cpu_domains.txt b/Documentation/power/cpu_domains.txt
new file mode 100644
index 0000000..6a39a64
--- /dev/null
+++ b/Documentation/power/cpu_domains.txt
@@ -0,0 +1,109 @@
+CPU PM domains
+==============
+
+Newer CPUs are grouped in SoCs as clusters. A cluster in addition to the CPUs
+may have caches, floating point units and other architecture specific power
+controller that share resources when any of the CPUs are active. When the CPUs
+are in idle, some of these cluster components may also idle. A cluster may
+also be nested inside another cluster that provides common coherency
+interfaces to share data between the clusters. The organization of such
+clusters and CPU may be descibed in DT, since they are SoC specific.
+
+CPUIdle framework enables the CPUs to determine the sleep time and enter low
+power state to save power during periods of idle. CPUs in a cluster may enter
+and exit idle state independently of each other. During the time when all the
+CPUs are in idle state, the cluster may safely put some of the shared
+resources in their idle state. The time between the last CPU to enter idle and
+the first CPU to wake up is the time available for the cluster to enter its
+idle state.
+
+When SoCs power down the CPU during cpuidle, they generally have supplemental
+hardware that can handshake with the CPU with a signal that indicates that the
+CPU has stopped execution. The hardware is also responsible for warm booting
+the CPU on receiving an interrupt. In a cluster architecture, common resources
+that are shared by a cluster may also be powered down by an external
+microcontroller or a processor. The microcontroller may be programmed in
+advance to put the hardware blocks in a low power state, when the last active
+CPU sends the idle signal. When the signal is received, the microcontroller
+may trigger the hardware blocks to enter their low power state. When an
+interrupt to wakeup the processor is received, the microcontroller is
+responsible for bringing up the hardware blocks to its active state, before
+waking up the CPU. The timelines for such operations should be in the
+acceptable range for for CPU idle to get power benefits.
+
+CPU PM Domain Setup
+-------------------
+
+PM domains are represented in the DT as domain consumers and providers. A
+device may have a domain provider and a domain provider may support multiple
+domain consumers. Domains like clusters, may also be nested inside one
+another. A domain that has no active consumer, may be powered off and any
+resuming consumer would trigger the domain back to active. Parent domains may
+be powered off when the child domains are powered off. The CPU cluster can be
+fashioned as a PM domain. When the CPU devices are powered off, the PM domain
+may be powered off.
+
+Device idle is reference counted by runtime PM. When there is no active need
+for the device, runtime PM invokes callbacks to suspend the parent domain.
+Generic PM domain (genpd) handles the hierarchy of devices, domains and the
+reference counting of objects leading to last man down and first man up in the
+domain. The CPU domains helper functions defines PM domains for each CPU
+cluster and attaches the CPU devices to the respective PM domains.
+
+Platform drivers may use the following API to register their CPU PM domains.
+
+of_setup_cpu_pd() -
+Provides a single step registration of the CPU PM domain and attach CPUs to
+the genpd. Platform drivers may additionally register callbacks for power_on
+and power_off operations for the PM domain.
+
+of_setup_cpu_pd_single() -
+Define PM domain for a single CPU and attach the CPU to its domain.
+
+
+CPU PM Domain governor
+----------------------
+
+CPUs have a unique ability to determine their next wakeup. CPUs may wake up
+for known timer interrupts and unknown interrupts from idle. Prediction
+algorithms and heuristic based algorithms like the Menu governor for cpuidle
+can determine the next wakeup of the CPU. However, determining the wakeup
+across a group of CPUs is a tough problem to solve.
+
+A simplistic approach would be to resort to known wakeups of the CPUs in
+determining the next wakeup of any CPU in the cluster. The CPU PM domain
+governor does just that. By looking into the tick device of the CPUs, the
+governor can determine the sleep time between the last CPU and the first
+scheduled wakeup of any CPU in that domain. This combined with the PM QoS
+requirement for CPU_DMA_LATENCY can be used to determine the deepest possible
+idle state of the CPU domain.
+
+
+PSCI based CPU PM Domains
+-------------------------
+
+ARM PSCI v1.0 supports PM domains for CPU clusters like in big.Little
+architecture. It is supported as part of the OS-Initiated (OSI) mode of the
+PSCI firmware. Since the control of domains is abstracted in the firmware,
+Linux does not need even a driver to control these domains. The complexity of
+determining the idle state of the PM domain is handled by the CPU PM domains.
+
+Every PSCI CPU PM domain idle state has a unique PSCI state id. The state id
+is read from the DT and specified using the arm,psci-suspend-param property.
+This makes it easy for big.Little SoCs to just specify the PM domain idle
+states for the CPU along with the psci-suspend-param and everything else is
+handled by the PSCI fimrware drivers and the firmware.
+
+
+DT definitions for PSCI CPU PM Domains
+--------------------------------------
+
+A PM domain's idle state can be defined in DT, the description of which is
+available in [1]. PSCI based CPU PM domains may define their idle states as
+part of the psci node. The additional parameter arm,psci-suspend-param is used
+to indicate to the firmwware the addition cluster state that would be achieved
+after the last CPU makes the PSCI call to suspend the CPU. The description of
+PSCI domain states is available in [2].
+
+[1]. Documentation/devicetree/bindings/arm/idle-states.txt
+[2]. Documentation/devicetree/bindings/arm/psci.txt
-- 
2.7.4

[PATCH V5 9/9] doc / cpu_domains: Describe CPU PM domains setup and governor

[Lina Iyer]

A generic CPU PM domain functionality is provided by
drivers/base/power/cpu_domains.c. This document describes the generic
usecase of CPU's PM domains, the setup of such domains and a CPU
specific genpd governor.

Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 Documentation/power/cpu_domains.txt | 109 ++++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)
 create mode 100644 Documentation/power/cpu_domains.txt

diff --git a/Documentation/power/cpu_domains.txt b/Documentation/power/cpu_domains.txt
new file mode 100644
index 0000000..6a39a64
--- /dev/null
+++ b/Documentation/power/cpu_domains.txt
@@ -0,0 +1,109 @@
+CPU PM domains
+==============
+
+Newer CPUs are grouped in SoCs as clusters. A cluster in addition to the CPUs
+may have caches, floating point units and other architecture specific power
+controller that share resources when any of the CPUs are active. When the CPUs
+are in idle, some of these cluster components may also idle. A cluster may
+also be nested inside another cluster that provides common coherency
+interfaces to share data between the clusters. The organization of such
+clusters and CPU may be descibed in DT, since they are SoC specific.
+
+CPUIdle framework enables the CPUs to determine the sleep time and enter low
+power state to save power during periods of idle. CPUs in a cluster may enter
+and exit idle state independently of each other. During the time when all the
+CPUs are in idle state, the cluster may safely put some of the shared
+resources in their idle state. The time between the last CPU to enter idle and
+the first CPU to wake up is the time available for the cluster to enter its
+idle state.
+
+When SoCs power down the CPU during cpuidle, they generally have supplemental
+hardware that can handshake with the CPU with a signal that indicates that the
+CPU has stopped execution. The hardware is also responsible for warm booting
+the CPU on receiving an interrupt. In a cluster architecture, common resources
+that are shared by a cluster may also be powered down by an external
+microcontroller or a processor. The microcontroller may be programmed in
+advance to put the hardware blocks in a low power state, when the last active
+CPU sends the idle signal. When the signal is received, the microcontroller
+may trigger the hardware blocks to enter their low power state. When an
+interrupt to wakeup the processor is received, the microcontroller is
+responsible for bringing up the hardware blocks to its active state, before
+waking up the CPU. The timelines for such operations should be in the
+acceptable range for for CPU idle to get power benefits.
+
+CPU PM Domain Setup
+-------------------
+
+PM domains are represented in the DT as domain consumers and providers. A
+device may have a domain provider and a domain provider may support multiple
+domain consumers. Domains like clusters, may also be nested inside one
+another. A domain that has no active consumer, may be powered off and any
+resuming consumer would trigger the domain back to active. Parent domains may
+be powered off when the child domains are powered off. The CPU cluster can be
+fashioned as a PM domain. When the CPU devices are powered off, the PM domain
+may be powered off.
+
+Device idle is reference counted by runtime PM. When there is no active need
+for the device, runtime PM invokes callbacks to suspend the parent domain.
+Generic PM domain (genpd) handles the hierarchy of devices, domains and the
+reference counting of objects leading to last man down and first man up in the
+domain. The CPU domains helper functions defines PM domains for each CPU
+cluster and attaches the CPU devices to the respective PM domains.
+
+Platform drivers may use the following API to register their CPU PM domains.
+
+of_setup_cpu_pd() -
+Provides a single step registration of the CPU PM domain and attach CPUs to
+the genpd. Platform drivers may additionally register callbacks for power_on
+and power_off operations for the PM domain.
+
+of_setup_cpu_pd_single() -
+Define PM domain for a single CPU and attach the CPU to its domain.
+
+
+CPU PM Domain governor
+----------------------
+
+CPUs have a unique ability to determine their next wakeup. CPUs may wake up
+for known timer interrupts and unknown interrupts from idle. Prediction
+algorithms and heuristic based algorithms like the Menu governor for cpuidle
+can determine the next wakeup of the CPU. However, determining the wakeup
+across a group of CPUs is a tough problem to solve.
+
+A simplistic approach would be to resort to known wakeups of the CPUs in
+determining the next wakeup of any CPU in the cluster. The CPU PM domain
+governor does just that. By looking into the tick device of the CPUs, the
+governor can determine the sleep time between the last CPU and the first
+scheduled wakeup of any CPU in that domain. This combined with the PM QoS
+requirement for CPU_DMA_LATENCY can be used to determine the deepest possible
+idle state of the CPU domain.
+
+
+PSCI based CPU PM Domains
+-------------------------
+
+ARM PSCI v1.0 supports PM domains for CPU clusters like in big.Little
+architecture. It is supported as part of the OS-Initiated (OSI) mode of the
+PSCI firmware. Since the control of domains is abstracted in the firmware,
+Linux does not need even a driver to control these domains. The complexity of
+determining the idle state of the PM domain is handled by the CPU PM domains.
+
+Every PSCI CPU PM domain idle state has a unique PSCI state id. The state id
+is read from the DT and specified using the arm,psci-suspend-param property.
+This makes it easy for big.Little SoCs to just specify the PM domain idle
+states for the CPU along with the psci-suspend-param and everything else is
+handled by the PSCI fimrware drivers and the firmware.
+
+
+DT definitions for PSCI CPU PM Domains
+--------------------------------------
+
+A PM domain's idle state can be defined in DT, the description of which is
+available in [1]. PSCI based CPU PM domains may define their idle states as
+part of the psci node. The additional parameter arm,psci-suspend-param is used
+to indicate to the firmwware the addition cluster state that would be achieved
+after the last CPU makes the PSCI call to suspend the CPU. The description of
+PSCI domain states is available in [2].
+
+[1]. Documentation/devicetree/bindings/arm/idle-states.txt
+[2]. Documentation/devicetree/bindings/arm/psci.txt
-- 
2.7.4