[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

This series is a re-worked version from Lina Iyer's two series [1] that got
posted more than a year ago by now. I have picked up the series and done a
significant re-work of it and here's the result. All patches have been changed,
some have been dropped, some are entirely new. For this reason I decided to not
include a version history, as I think people need a fresh start anyway.

For those few changes that are kind of similar, I have kept Lina's authorship,
the rest are now authored by me.

 - Lina (or anybody else for that matter) please tell if you think I should
   change, in regards to what tag/authorship should be for each commit!

Some background:

Overall this series have been discussed over years at various Linux conferences
and LKML, although let me give a brief introduction and then the rest can be
read in each changelog.

For ARM, the PSCI firmware interface may be managing the power to the CPUs.
Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
could add another level of complexity from a CPU idle management point of view.

PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
enables a more fine grained method, allowing Linux to get more control, in
regards to being energy efficient. This is typically useful for these kind of
complex battery driven platforms.

Now, in principle what is missing today around CPU idle management for these
SoCs arranging CPUs in a hierarchical manner, that is what this series intends
to address.

I could potentially have split the series into two parts, but for completeness
I decided to post it as one. Please tell if want me to change!

 - The first part are changes to generic PM code (genpd, cpu_pm and timers):
   Patch 1 -> Patch 9.

 - The second part, which depends on the first, are changes specific to
   PSCI/ARM64 (psci, of, arm64): Patch 10 -> Patch 25.

The series is based on v4.16-rc5, but applies also as of today to Rafael's
linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
and I except that test coverage should be increased by some more boards
shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).
You may find the code at:

git.linaro.org/people/ulf.hansson/linux-pm.git next


Kind regards
Ulf Hansson

[1]
https://lwn.net/Articles/716300/
https://www.spinics.net/lists/arm-kernel/msg566200.html


Lina Iyer (6):
  PM / Domains: Add generic data pointer to genpd_power_state struct
  timer: Export next wakeup time of a CPU
  dt: psci: Update DT bindings to support hierarchical PSCI states
  cpuidle: dt: Support hierarchical CPU idle states
  drivers: firmware: psci: Support hierarchical CPU idle states
  arm64: dts: Convert to the hierarchical CPU topology layout for
    MSM8916

Ulf Hansson (19):
  PM / Domains: Don't treat zero found compatible idle states as an
    error
  PM / Domains: Deal with multiple states but no governor in genpd
  PM / Domains: Add support for CPU devices to genpd
  PM / Domains: Add helper functions to attach/detach CPUs to/from genpd
  PM / Domains: Add genpd governor for CPUs
  PM / Domains: Extend genpd CPU governor to cope with QoS constraints
  kernel/cpu_pm: Manage runtime PM in the idle path for CPUs
  of: base: Add of_get_cpu_state_node() to get idle states for a CPU
    node
  drivers: firmware: psci: Split psci_dt_cpu_init_idle()
  drivers: firmware: psci: Simplify error path of psci_dt_init()
  drivers: firmware: psci: Announce support for OS initiated suspend
    mode
  drivers: firmware: psci: Prepare to use OS initiated suspend mode
  drivers: firmware: psci: Share a few internal PSCI functions
  drivers: firmware: psci: Add support for PM domains using genpd
  drivers: firmware: psci: Introduce psci_dt_topology_init()
  drivers: firmware: psci: Try to attach CPU devices to their PM domains
  drivers: firmware: psci: Deal with CPU hotplug when using OSI mode
  MAINTAINERS: Update files for PSCI
  arm64: kernel: Respect the hierarchical CPU topology in DT for PSCI

 Documentation/devicetree/bindings/arm/psci.txt | 156 +++++++++++++++++++++
 MAINTAINERS                                    |   2 +-
 arch/arm64/boot/dts/qcom/msm8916.dtsi          |  53 +++++++-
 arch/arm64/kernel/setup.c                      |   3 +
 drivers/base/power/domain.c                    | 156 ++++++++++++++++++++-
 drivers/base/power/domain_governor.c           |  65 ++++++++-
 drivers/cpuidle/dt_idle_states.c               |   5 +-
 drivers/firmware/Makefile                      |   2 +-
 drivers/firmware/psci.c                        | 169 +++++++++++++++++------
 drivers/firmware/psci.h                        |  19 +++
 drivers/firmware/psci_pm_domain.c              | 180 +++++++++++++++++++++++++
 drivers/of/base.c                              |  35 +++++
 include/linux/of.h                             |   8 ++
 include/linux/pm_domain.h                      |  16 +++
 include/linux/psci.h                           |   2 +
 include/linux/tick.h                           |  10 ++
 include/uapi/linux/psci.h                      |   5 +
 kernel/cpu_pm.c                                |  11 ++
 kernel/time/tick-sched.c                       |  11 ++
 19 files changed, 853 insertions(+), 55 deletions(-)
 create mode 100644 drivers/firmware/psci.h
 create mode 100644 drivers/firmware/psci_pm_domain.c

-- 
2.7.4

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

This series is a re-worked version from Lina Iyer's two series [1] that got
posted more than a year ago by now. I have picked up the series and done a
significant re-work of it and here's the result. All patches have been changed,
some have been dropped, some are entirely new. For this reason I decided to not
include a version history, as I think people need a fresh start anyway.

For those few changes that are kind of similar, I have kept Lina's authorship,
the rest are now authored by me.

 - Lina (or anybody else for that matter) please tell if you think I should
   change, in regards to what tag/authorship should be for each commit!

Some background:

Overall this series have been discussed over years at various Linux conferences
and LKML, although let me give a brief introduction and then the rest can be
read in each changelog.

For ARM, the PSCI firmware interface may be managing the power to the CPUs.
Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
could add another level of complexity from a CPU idle management point of view.

PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
enables a more fine grained method, allowing Linux to get more control, in
regards to being energy efficient. This is typically useful for these kind of
complex battery driven platforms.

Now, in principle what is missing today around CPU idle management for these
SoCs arranging CPUs in a hierarchical manner, that is what this series intends
to address.

I could potentially have split the series into two parts, but for completeness
I decided to post it as one. Please tell if want me to change!

 - The first part are changes to generic PM code (genpd, cpu_pm and timers):
   Patch 1 -> Patch 9.

 - The second part, which depends on the first, are changes specific to
   PSCI/ARM64 (psci, of, arm64): Patch 10 -> Patch 25.

The series is based on v4.16-rc5, but applies also as of today to Rafael's
linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
and I except that test coverage should be increased by some more boards
shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).
You may find the code at:

git.linaro.org/people/ulf.hansson/linux-pm.git next


Kind regards
Ulf Hansson

[1]
https://lwn.net/Articles/716300/
https://www.spinics.net/lists/arm-kernel/msg566200.html


Lina Iyer (6):
  PM / Domains: Add generic data pointer to genpd_power_state struct
  timer: Export next wakeup time of a CPU
  dt: psci: Update DT bindings to support hierarchical PSCI states
  cpuidle: dt: Support hierarchical CPU idle states
  drivers: firmware: psci: Support hierarchical CPU idle states
  arm64: dts: Convert to the hierarchical CPU topology layout for
    MSM8916

Ulf Hansson (19):
  PM / Domains: Don't treat zero found compatible idle states as an
    error
  PM / Domains: Deal with multiple states but no governor in genpd
  PM / Domains: Add support for CPU devices to genpd
  PM / Domains: Add helper functions to attach/detach CPUs to/from genpd
  PM / Domains: Add genpd governor for CPUs
  PM / Domains: Extend genpd CPU governor to cope with QoS constraints
  kernel/cpu_pm: Manage runtime PM in the idle path for CPUs
  of: base: Add of_get_cpu_state_node() to get idle states for a CPU
    node
  drivers: firmware: psci: Split psci_dt_cpu_init_idle()
  drivers: firmware: psci: Simplify error path of psci_dt_init()
  drivers: firmware: psci: Announce support for OS initiated suspend
    mode
  drivers: firmware: psci: Prepare to use OS initiated suspend mode
  drivers: firmware: psci: Share a few internal PSCI functions
  drivers: firmware: psci: Add support for PM domains using genpd
  drivers: firmware: psci: Introduce psci_dt_topology_init()
  drivers: firmware: psci: Try to attach CPU devices to their PM domains
  drivers: firmware: psci: Deal with CPU hotplug when using OSI mode
  MAINTAINERS: Update files for PSCI
  arm64: kernel: Respect the hierarchical CPU topology in DT for PSCI

 Documentation/devicetree/bindings/arm/psci.txt | 156 +++++++++++++++++++++
 MAINTAINERS                                    |   2 +-
 arch/arm64/boot/dts/qcom/msm8916.dtsi          |  53 +++++++-
 arch/arm64/kernel/setup.c                      |   3 +
 drivers/base/power/domain.c                    | 156 ++++++++++++++++++++-
 drivers/base/power/domain_governor.c           |  65 ++++++++-
 drivers/cpuidle/dt_idle_states.c               |   5 +-
 drivers/firmware/Makefile                      |   2 +-
 drivers/firmware/psci.c                        | 169 +++++++++++++++++------
 drivers/firmware/psci.h                        |  19 +++
 drivers/firmware/psci_pm_domain.c              | 180 +++++++++++++++++++++++++
 drivers/of/base.c                              |  35 +++++
 include/linux/of.h                             |   8 ++
 include/linux/pm_domain.h                      |  16 +++
 include/linux/psci.h                           |   2 +
 include/linux/tick.h                           |  10 ++
 include/uapi/linux/psci.h                      |   5 +
 kernel/cpu_pm.c                                |  11 ++
 kernel/time/tick-sched.c                       |  11 ++
 19 files changed, 853 insertions(+), 55 deletions(-)
 create mode 100644 drivers/firmware/psci.h
 create mode 100644 drivers/firmware/psci_pm_domain.c

-- 
2.7.4

[PATCH v6 01/25] PM / Domains: Don't treat zero found compatible idle states as an error

[Ulf Hansson]

Instead of returning -EINVAL from of_genpd_parse_idle_states() in case none
compatible states was found, let's return 0 to indicate success. Assign
also the out-parameter *states to NULL and *n to 0, to indicate to the
caller that zero states have been found/allocated.

This enables the caller of of_genpd_parse_idle_states() to easier act on
the returned error code.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 1ea0e25..c2ef944 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2331,8 +2331,8 @@ static int genpd_iterate_idle_states(struct device_node *dn,
  *
  * Returns the device states parsed from the OF node. The memory for the states
  * is allocated by this function and is the responsibility of the caller to
- * free the memory after use. If no domain idle states is found it returns
- * -EINVAL and in case of errors, a negative error code.
+ * free the memory after use. If any or zero compatible domain idle states is
+ * found it returns 0 and in case of errors, a negative error code is returned.
  */
 int of_genpd_parse_idle_states(struct device_node *dn,
 			struct genpd_power_state **states, int *n)
@@ -2341,8 +2341,14 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	int ret;
 
 	ret = genpd_iterate_idle_states(dn, NULL);
-	if (ret <= 0)
-		return ret < 0 ? ret : -EINVAL;
+	if (ret < 0)
+		return ret;
+
+	if (!ret) {
+		*states = NULL;
+		*n = 0;
+		return 0;
+	}
 
 	st = kcalloc(ret, sizeof(*st), GFP_KERNEL);
 	if (!st)
-- 
2.7.4

[PATCH v6 01/25] PM / Domains: Don't treat zero found compatible idle states as an error

[Ulf Hansson]

Instead of returning -EINVAL from of_genpd_parse_idle_states() in case none
compatible states was found, let's return 0 to indicate success. Assign
also the out-parameter *states to NULL and *n to 0, to indicate to the
caller that zero states have been found/allocated.

This enables the caller of of_genpd_parse_idle_states() to easier act on
the returned error code.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 1ea0e25..c2ef944 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2331,8 +2331,8 @@ static int genpd_iterate_idle_states(struct device_node *dn,
  *
  * Returns the device states parsed from the OF node. The memory for the states
  * is allocated by this function and is the responsibility of the caller to
- * free the memory after use. If no domain idle states is found it returns
- * -EINVAL and in case of errors, a negative error code.
+ * free the memory after use. If any or zero compatible domain idle states is
+ * found it returns 0 and in case of errors, a negative error code is returned.
  */
 int of_genpd_parse_idle_states(struct device_node *dn,
 			struct genpd_power_state **states, int *n)
@@ -2341,8 +2341,14 @@ int of_genpd_parse_idle_states(struct device_node *dn,
 	int ret;
 
 	ret = genpd_iterate_idle_states(dn, NULL);
-	if (ret <= 0)
-		return ret < 0 ? ret : -EINVAL;
+	if (ret < 0)
+		return ret;
+
+	if (!ret) {
+		*states = NULL;
+		*n = 0;
+		return 0;
+	}
 
 	st = kcalloc(ret, sizeof(*st), GFP_KERNEL);
 	if (!st)
-- 
2.7.4

[PATCH v6 02/25] PM / Domains: Deal with multiple states but no governor in genpd

[Ulf Hansson]

A caller of pm_genpd_init() that provides some states for the genpd via the
->states pointer in the struct generic_pm_domain, should also provide a
governor. This because it's the job of the governor to pick a state that
satisfies the constraints.

Therefore, let's print a warning to inform the user about such bogus
configuration and avoid to bail out, by instead picking the shallowest
state before genpd invokes the ->power_off() callback.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index c2ef944..9aff79d 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -466,6 +466,10 @@ static int genpd_power_off(struct generic_pm_domain *genpd, bool one_dev_on,
 			return -EAGAIN;
 	}
 
+	/* Default to shallowest state. */
+	if (!genpd->gov)
+		genpd->state_idx = 0;
+
 	if (genpd->power_off) {
 		int ret;
 
@@ -1694,6 +1698,8 @@ int pm_genpd_init(struct generic_pm_domain *genpd,
 		ret = genpd_set_default_power_state(genpd);
 		if (ret)
 			return ret;
+	} else if (!gov) {
+		pr_warn("%s : no governor for states\n", genpd->name);
 	}
 
 	mutex_lock(&gpd_list_lock);
-- 
2.7.4

[PATCH v6 02/25] PM / Domains: Deal with multiple states but no governor in genpd

[Ulf Hansson]

A caller of pm_genpd_init() that provides some states for the genpd via the
->states pointer in the struct generic_pm_domain, should also provide a
governor. This because it's the job of the governor to pick a state that
satisfies the constraints.

Therefore, let's print a warning to inform the user about such bogus
configuration and avoid to bail out, by instead picking the shallowest
state before genpd invokes the ->power_off() callback.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index c2ef944..9aff79d 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -466,6 +466,10 @@ static int genpd_power_off(struct generic_pm_domain *genpd, bool one_dev_on,
 			return -EAGAIN;
 	}
 
+	/* Default to shallowest state. */
+	if (!genpd->gov)
+		genpd->state_idx = 0;
+
 	if (genpd->power_off) {
 		int ret;
 
@@ -1694,6 +1698,8 @@ int pm_genpd_init(struct generic_pm_domain *genpd,
 		ret = genpd_set_default_power_state(genpd);
 		if (ret)
 			return ret;
+	} else if (!gov) {
+		pr_warn("%s : no governor for states\n", genpd->name);
 	}
 
 	mutex_lock(&gpd_list_lock);
-- 
2.7.4

[PATCH v6 03/25] PM / Domains: Add generic data pointer to genpd_power_state struct

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Let's add a data pointer to the genpd_power_state struct, to allow
platforms to store per state specific data.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 include/linux/pm_domain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 04dbef9..55ad34d 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -44,6 +44,7 @@ struct genpd_power_state {
 	s64 residency_ns;
 	struct fwnode_handle *fwnode;
 	ktime_t idle_time;
+	void *data;
 };
 
 struct genpd_lock_ops;
-- 
2.7.4

[PATCH v6 03/25] PM / Domains: Add generic data pointer to genpd_power_state struct

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Let's add a data pointer to the genpd_power_state struct, to allow
platforms to store per state specific data.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 include/linux/pm_domain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 04dbef9..55ad34d 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -44,6 +44,7 @@ struct genpd_power_state {
 	s64 residency_ns;
 	struct fwnode_handle *fwnode;
 	ktime_t idle_time;
+	void *data;
 };
 
 struct genpd_lock_ops;
-- 
2.7.4

[PATCH v6 04/25] PM / Domains: Add support for CPU devices to genpd

[Ulf Hansson]

To enable a device belonging to a CPU to be attached to a PM domain managed
by genpd, let's do a few changes to genpd as to make it convenient to
manage the specifics around CPUs.

First, as to be able to quickly find out what CPUs that are attached to a
genpd, which typically becomes useful from a genpd governor as following
changes is about to show, let's add a cpumask 'cpus' to the struct
generic_pm_domain.

At the point when a device that belongs to a CPU, is attached/detached to
its corresponding PM domain via genpd_add_device(), let's update the
cpumask in genpd->cpus. Moreover, propagate the update of the cpumask to
the master domains, which makes the genpd->cpus to contain a cpumask that
hierarchically reflect all CPUs for a genpd, including CPUs attached to
subdomains.

Second, to unconditionally manage CPUs and the cpumask in genpd->cpus, is
unnecessary for cases when only non-CPU devices are parts of a genpd.
Let's avoid this by adding a new configuration bit, GENPD_FLAG_CPU_DOMAIN.
Clients must set the bit before they call pm_genpd_init(), as to instruct
genpd that it shall deal with CPUs and thus manage the cpumask in
genpd->cpus.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 69 ++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/pm_domain.h   |  3 ++
 2 files changed, 71 insertions(+), 1 deletion(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 9aff79d..e178521 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -19,6 +19,7 @@
 #include <linux/sched.h>
 #include <linux/suspend.h>
 #include <linux/export.h>
+#include <linux/cpu.h>
 
 #include "power.h"
 
@@ -125,6 +126,7 @@ static const struct genpd_lock_ops genpd_spin_ops = {
 #define genpd_is_irq_safe(genpd)	(genpd->flags & GENPD_FLAG_IRQ_SAFE)
 #define genpd_is_always_on(genpd)	(genpd->flags & GENPD_FLAG_ALWAYS_ON)
 #define genpd_is_active_wakeup(genpd)	(genpd->flags & GENPD_FLAG_ACTIVE_WAKEUP)
+#define genpd_is_cpu_domain(genpd)	(genpd->flags & GENPD_FLAG_CPU_DOMAIN)
 
 static inline bool irq_safe_dev_in_no_sleep_domain(struct device *dev,
 		const struct generic_pm_domain *genpd)
@@ -1377,6 +1379,62 @@ static void genpd_free_dev_data(struct device *dev,
 	dev_pm_put_subsys_data(dev);
 }
 
+static void __genpd_update_cpumask(struct generic_pm_domain *genpd,
+				   int cpu, bool set, unsigned int depth)
+{
+	struct gpd_link *link;
+
+	if (!genpd_is_cpu_domain(genpd))
+		return;
+
+	list_for_each_entry(link, &genpd->slave_links, slave_node) {
+		struct generic_pm_domain *master = link->master;
+
+		genpd_lock_nested(master, depth + 1);
+		__genpd_update_cpumask(master, cpu, set, depth + 1);
+		genpd_unlock(master);
+	}
+
+	if (set)
+		cpumask_set_cpu(cpu, genpd->cpus);
+	else
+		cpumask_clear_cpu(cpu, genpd->cpus);
+}
+
+static void genpd_update_cpumask(struct generic_pm_domain *genpd,
+				 struct device *dev, bool set)
+{
+	bool is_cpu = false;
+	int cpu;
+
+	if (!genpd_is_cpu_domain(genpd))
+		return;
+
+	for_each_possible_cpu(cpu) {
+		if (get_cpu_device(cpu) == dev) {
+			is_cpu = true;
+			break;
+		}
+	}
+
+	if (!is_cpu)
+		return;
+
+	__genpd_update_cpumask(genpd, cpu, set, 0);
+}
+
+static void genpd_set_cpumask(struct generic_pm_domain *genpd,
+			      struct device *dev)
+{
+	genpd_update_cpumask(genpd, dev, true);
+}
+
+static void genpd_clear_cpumask(struct generic_pm_domain *genpd,
+				struct device *dev)
+{
+	genpd_update_cpumask(genpd, dev, false);
+}
+
 static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 			    struct gpd_timing_data *td)
 {
@@ -1403,6 +1461,8 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	if (ret)
 		goto out;
 
+	genpd_set_cpumask(genpd, dev);
+
 	dev_pm_domain_set(dev, &genpd->domain);
 
 	genpd->device_count++;
@@ -1466,6 +1526,7 @@ static int genpd_remove_device(struct generic_pm_domain *genpd,
 	if (genpd->detach_dev)
 		genpd->detach_dev(genpd, dev);
 
+	genpd_clear_cpumask(genpd, dev);
 	dev_pm_domain_set(dev, NULL);
 
 	list_del_init(&pdd->list_node);
@@ -1693,11 +1754,16 @@ int pm_genpd_init(struct generic_pm_domain *genpd,
 	if (genpd_is_always_on(genpd) && !genpd_status_on(genpd))
 		return -EINVAL;
 
+	if (!zalloc_cpumask_var(&genpd->cpus, GFP_KERNEL))
+		return -ENOMEM;
+
 	/* Use only one "off" state if there were no states declared */
 	if (genpd->state_count == 0) {
 		ret = genpd_set_default_power_state(genpd);
-		if (ret)
+		if (ret) {
+			free_cpumask_var(genpd->cpus);
 			return ret;
+		}
 	} else if (!gov) {
 		pr_warn("%s : no governor for states\n", genpd->name);
 	}
@@ -1740,6 +1806,7 @@ static int genpd_remove(struct generic_pm_domain *genpd)
 	list_del(&genpd->gpd_list_node);
 	genpd_unlock(genpd);
 	cancel_work_sync(&genpd->power_off_work);
+	free_cpumask_var(genpd->cpus);
 	kfree(genpd->free);
 	pr_debug("%s: removed %s\n", __func__, genpd->name);
 
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 55ad34d..29ab00c 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -16,12 +16,14 @@
 #include <linux/of.h>
 #include <linux/notifier.h>
 #include <linux/spinlock.h>
+#include <linux/cpumask.h>
 
 /* Defines used for the flags field in the struct generic_pm_domain */
 #define GENPD_FLAG_PM_CLK	 (1U << 0) /* PM domain uses PM clk */
 #define GENPD_FLAG_IRQ_SAFE	 (1U << 1) /* PM domain operates in atomic */
 #define GENPD_FLAG_ALWAYS_ON	 (1U << 2) /* PM domain is always powered on */
 #define GENPD_FLAG_ACTIVE_WAKEUP (1U << 3) /* Keep devices active if wakeup */
+#define GENPD_FLAG_CPU_DOMAIN	 (1U << 4) /* PM domain manages CPUs */
 
 enum gpd_status {
 	GPD_STATE_ACTIVE = 0,	/* PM domain is active */
@@ -66,6 +68,7 @@ struct generic_pm_domain {
 	unsigned int suspended_count;	/* System suspend device counter */
 	unsigned int prepared_count;	/* Suspend counter of prepared devices */
 	unsigned int performance_state;	/* Aggregated max performance state */
+	cpumask_var_t cpus;		/* A cpumask of the attached CPUs */
 	int (*power_off)(struct generic_pm_domain *domain);
 	int (*power_on)(struct generic_pm_domain *domain);
 	int (*set_performance_state)(struct generic_pm_domain *genpd,
-- 
2.7.4

[PATCH v6 04/25] PM / Domains: Add support for CPU devices to genpd

[Ulf Hansson]

To enable a device belonging to a CPU to be attached to a PM domain managed
by genpd, let's do a few changes to genpd as to make it convenient to
manage the specifics around CPUs.

First, as to be able to quickly find out what CPUs that are attached to a
genpd, which typically becomes useful from a genpd governor as following
changes is about to show, let's add a cpumask 'cpus' to the struct
generic_pm_domain.

At the point when a device that belongs to a CPU, is attached/detached to
its corresponding PM domain via genpd_add_device(), let's update the
cpumask in genpd->cpus. Moreover, propagate the update of the cpumask to
the master domains, which makes the genpd->cpus to contain a cpumask that
hierarchically reflect all CPUs for a genpd, including CPUs attached to
subdomains.

Second, to unconditionally manage CPUs and the cpumask in genpd->cpus, is
unnecessary for cases when only non-CPU devices are parts of a genpd.
Let's avoid this by adding a new configuration bit, GENPD_FLAG_CPU_DOMAIN.
Clients must set the bit before they call pm_genpd_init(), as to instruct
genpd that it shall deal with CPUs and thus manage the cpumask in
genpd->cpus.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 69 ++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/pm_domain.h   |  3 ++
 2 files changed, 71 insertions(+), 1 deletion(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 9aff79d..e178521 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -19,6 +19,7 @@
 #include <linux/sched.h>
 #include <linux/suspend.h>
 #include <linux/export.h>
+#include <linux/cpu.h>
 
 #include "power.h"
 
@@ -125,6 +126,7 @@ static const struct genpd_lock_ops genpd_spin_ops = {
 #define genpd_is_irq_safe(genpd)	(genpd->flags & GENPD_FLAG_IRQ_SAFE)
 #define genpd_is_always_on(genpd)	(genpd->flags & GENPD_FLAG_ALWAYS_ON)
 #define genpd_is_active_wakeup(genpd)	(genpd->flags & GENPD_FLAG_ACTIVE_WAKEUP)
+#define genpd_is_cpu_domain(genpd)	(genpd->flags & GENPD_FLAG_CPU_DOMAIN)
 
 static inline bool irq_safe_dev_in_no_sleep_domain(struct device *dev,
 		const struct generic_pm_domain *genpd)
@@ -1377,6 +1379,62 @@ static void genpd_free_dev_data(struct device *dev,
 	dev_pm_put_subsys_data(dev);
 }
 
+static void __genpd_update_cpumask(struct generic_pm_domain *genpd,
+				   int cpu, bool set, unsigned int depth)
+{
+	struct gpd_link *link;
+
+	if (!genpd_is_cpu_domain(genpd))
+		return;
+
+	list_for_each_entry(link, &genpd->slave_links, slave_node) {
+		struct generic_pm_domain *master = link->master;
+
+		genpd_lock_nested(master, depth + 1);
+		__genpd_update_cpumask(master, cpu, set, depth + 1);
+		genpd_unlock(master);
+	}
+
+	if (set)
+		cpumask_set_cpu(cpu, genpd->cpus);
+	else
+		cpumask_clear_cpu(cpu, genpd->cpus);
+}
+
+static void genpd_update_cpumask(struct generic_pm_domain *genpd,
+				 struct device *dev, bool set)
+{
+	bool is_cpu = false;
+	int cpu;
+
+	if (!genpd_is_cpu_domain(genpd))
+		return;
+
+	for_each_possible_cpu(cpu) {
+		if (get_cpu_device(cpu) == dev) {
+			is_cpu = true;
+			break;
+		}
+	}
+
+	if (!is_cpu)
+		return;
+
+	__genpd_update_cpumask(genpd, cpu, set, 0);
+}
+
+static void genpd_set_cpumask(struct generic_pm_domain *genpd,
+			      struct device *dev)
+{
+	genpd_update_cpumask(genpd, dev, true);
+}
+
+static void genpd_clear_cpumask(struct generic_pm_domain *genpd,
+				struct device *dev)
+{
+	genpd_update_cpumask(genpd, dev, false);
+}
+
 static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 			    struct gpd_timing_data *td)
 {
@@ -1403,6 +1461,8 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	if (ret)
 		goto out;
 
+	genpd_set_cpumask(genpd, dev);
+
 	dev_pm_domain_set(dev, &genpd->domain);
 
 	genpd->device_count++;
@@ -1466,6 +1526,7 @@ static int genpd_remove_device(struct generic_pm_domain *genpd,
 	if (genpd->detach_dev)
 		genpd->detach_dev(genpd, dev);
 
+	genpd_clear_cpumask(genpd, dev);
 	dev_pm_domain_set(dev, NULL);
 
 	list_del_init(&pdd->list_node);
@@ -1693,11 +1754,16 @@ int pm_genpd_init(struct generic_pm_domain *genpd,
 	if (genpd_is_always_on(genpd) && !genpd_status_on(genpd))
 		return -EINVAL;
 
+	if (!zalloc_cpumask_var(&genpd->cpus, GFP_KERNEL))
+		return -ENOMEM;
+
 	/* Use only one "off" state if there were no states declared */
 	if (genpd->state_count == 0) {
 		ret = genpd_set_default_power_state(genpd);
-		if (ret)
+		if (ret) {
+			free_cpumask_var(genpd->cpus);
 			return ret;
+		}
 	} else if (!gov) {
 		pr_warn("%s : no governor for states\n", genpd->name);
 	}
@@ -1740,6 +1806,7 @@ static int genpd_remove(struct generic_pm_domain *genpd)
 	list_del(&genpd->gpd_list_node);
 	genpd_unlock(genpd);
 	cancel_work_sync(&genpd->power_off_work);
+	free_cpumask_var(genpd->cpus);
 	kfree(genpd->free);
 	pr_debug("%s: removed %s\n", __func__, genpd->name);
 
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 55ad34d..29ab00c 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -16,12 +16,14 @@
 #include <linux/of.h>
 #include <linux/notifier.h>
 #include <linux/spinlock.h>
+#include <linux/cpumask.h>
 
 /* Defines used for the flags field in the struct generic_pm_domain */
 #define GENPD_FLAG_PM_CLK	 (1U << 0) /* PM domain uses PM clk */
 #define GENPD_FLAG_IRQ_SAFE	 (1U << 1) /* PM domain operates in atomic */
 #define GENPD_FLAG_ALWAYS_ON	 (1U << 2) /* PM domain is always powered on */
 #define GENPD_FLAG_ACTIVE_WAKEUP (1U << 3) /* Keep devices active if wakeup */
+#define GENPD_FLAG_CPU_DOMAIN	 (1U << 4) /* PM domain manages CPUs */
 
 enum gpd_status {
 	GPD_STATE_ACTIVE = 0,	/* PM domain is active */
@@ -66,6 +68,7 @@ struct generic_pm_domain {
 	unsigned int suspended_count;	/* System suspend device counter */
 	unsigned int prepared_count;	/* Suspend counter of prepared devices */
 	unsigned int performance_state;	/* Aggregated max performance state */
+	cpumask_var_t cpus;		/* A cpumask of the attached CPUs */
 	int (*power_off)(struct generic_pm_domain *domain);
 	int (*power_on)(struct generic_pm_domain *domain);
 	int (*set_performance_state)(struct generic_pm_domain *genpd,
-- 
2.7.4

[PATCH v6 05/25] PM / Domains: Add helper functions to attach/detach CPUs to/from genpd

[Ulf Hansson]

Introduce two new genpd helper functions, of_genpd_attach|detach_cpu(),
which takes the CPU-number as an in-parameter.

To attach a CPU to a genpd, of_genpd_attach_cpu() starts by fetching the
struct device belonging to the CPU. Then it calls genpd_dev_pm_attach(),
which via DT tries to hook up the CPU device to its corresponding PM
domain. If it succeeds, of_genpd_attach_cpu() continues to prepare/enable
runtime PM of the device.

To detach a CPU from its PM domain, of_genpd_attach_cpu() reverse the
operations made from of_genpd_attach_cpu(). However, first it checks that
the CPU device has a valid PM domain pointer assigned, as to make sure it
belongs to genpd.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 67 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h   |  9 ++++++
 2 files changed, 76 insertions(+)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index e178521..91ea9b2 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2324,6 +2324,73 @@ int genpd_dev_pm_attach(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(genpd_dev_pm_attach);
 
+/**
+ * of_genpd_attach_cpu() - Attach a CPU to its PM domain
+ * @cpu: The CPU to be attached.
+ *
+ * Try to find the corresponding PM domain provider for the CPU's device. Use
+ * it to attach the CPU's device to its PM domain. If attaching is completed
+ * successfully, we enable runtime PM for the CPU's device.
+ *
+ * Returns zero on success else a negative error code.
+ */
+int of_genpd_attach_cpu(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+	int ret;
+
+	if (!dev) {
+		pr_warn("genpd: no dev for cpu%d\n", cpu);
+		return -ENODEV;
+	}
+
+	ret = genpd_dev_pm_attach(dev);
+	if (ret) {
+		dev_warn(dev, "genpd: attach cpu failed %d\n", ret);
+		return ret;
+	}
+
+	pm_runtime_irq_safe(dev);
+	pm_runtime_get_noresume(dev);
+	pm_runtime_set_active(dev);
+	pm_runtime_enable(dev);
+
+	dev_info(dev, "genpd: attached cpu\n");
+	return 0;
+}
+EXPORT_SYMBOL(of_genpd_attach_cpu);
+
+/**
+ * of_genpd_detach_cpu() - Detach a CPU from its PM domain
+ * @cpu: The CPU to be detached.
+ *
+ * Detach the CPU's device from its corresponding PM domain. If detaching is
+ * completed successfully, disable runtime PM and restore the runtime PM usage
+ * count for the CPU's device.
+ */
+void of_genpd_detach_cpu(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!dev) {
+		pr_warn("genpd: no dev for cpu%d\n", cpu);
+		return;
+	}
+
+	/* Check that the device is attached to a genpd. */
+	if (!(dev->pm_domain && dev->pm_domain->detach == genpd_dev_pm_detach))
+		return;
+
+	genpd_dev_pm_detach(dev, true);
+
+	pm_runtime_disable(dev);
+	pm_runtime_put_noidle(dev);
+	pm_runtime_reinit(dev);
+
+	dev_info(dev, "genpd: detached cpu\n");
+}
+EXPORT_SYMBOL(of_genpd_detach_cpu);
+
 static const struct of_device_id idle_state_match[] = {
 	{ .compatible = "domain-idle-state", },
 	{ }
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 29ab00c..ccd7c94 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -249,6 +249,8 @@ extern int of_genpd_parse_idle_states(struct device_node *dn,
 			struct genpd_power_state **states, int *n);
 
 int genpd_dev_pm_attach(struct device *dev);
+int of_genpd_attach_cpu(int cpu);
+void of_genpd_detach_cpu(int cpu);
 #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
 static inline int of_genpd_add_provider_simple(struct device_node *np,
 					struct generic_pm_domain *genpd)
@@ -287,6 +289,13 @@ static inline int genpd_dev_pm_attach(struct device *dev)
 	return -ENODEV;
 }
 
+static inline int of_genpd_attach_cpu(int cpu)
+{
+	return -ENODEV;
+}
+
+static inline void of_genpd_detach_cpu(int cpu) {}
+
 static inline
 struct generic_pm_domain *of_genpd_remove_last(struct device_node *np)
 {
-- 
2.7.4

[PATCH v6 05/25] PM / Domains: Add helper functions to attach/detach CPUs to/from genpd

[Ulf Hansson]

Introduce two new genpd helper functions, of_genpd_attach|detach_cpu(),
which takes the CPU-number as an in-parameter.

To attach a CPU to a genpd, of_genpd_attach_cpu() starts by fetching the
struct device belonging to the CPU. Then it calls genpd_dev_pm_attach(),
which via DT tries to hook up the CPU device to its corresponding PM
domain. If it succeeds, of_genpd_attach_cpu() continues to prepare/enable
runtime PM of the device.

To detach a CPU from its PM domain, of_genpd_attach_cpu() reverse the
operations made from of_genpd_attach_cpu(). However, first it checks that
the CPU device has a valid PM domain pointer assigned, as to make sure it
belongs to genpd.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain.c | 67 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h   |  9 ++++++
 2 files changed, 76 insertions(+)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index e178521..91ea9b2 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -2324,6 +2324,73 @@ int genpd_dev_pm_attach(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(genpd_dev_pm_attach);
 
+/**
+ * of_genpd_attach_cpu() - Attach a CPU to its PM domain
+ * @cpu: The CPU to be attached.
+ *
+ * Try to find the corresponding PM domain provider for the CPU's device. Use
+ * it to attach the CPU's device to its PM domain. If attaching is completed
+ * successfully, we enable runtime PM for the CPU's device.
+ *
+ * Returns zero on success else a negative error code.
+ */
+int of_genpd_attach_cpu(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+	int ret;
+
+	if (!dev) {
+		pr_warn("genpd: no dev for cpu%d\n", cpu);
+		return -ENODEV;
+	}
+
+	ret = genpd_dev_pm_attach(dev);
+	if (ret) {
+		dev_warn(dev, "genpd: attach cpu failed %d\n", ret);
+		return ret;
+	}
+
+	pm_runtime_irq_safe(dev);
+	pm_runtime_get_noresume(dev);
+	pm_runtime_set_active(dev);
+	pm_runtime_enable(dev);
+
+	dev_info(dev, "genpd: attached cpu\n");
+	return 0;
+}
+EXPORT_SYMBOL(of_genpd_attach_cpu);
+
+/**
+ * of_genpd_detach_cpu() - Detach a CPU from its PM domain
+ * @cpu: The CPU to be detached.
+ *
+ * Detach the CPU's device from its corresponding PM domain. If detaching is
+ * completed successfully, disable runtime PM and restore the runtime PM usage
+ * count for the CPU's device.
+ */
+void of_genpd_detach_cpu(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!dev) {
+		pr_warn("genpd: no dev for cpu%d\n", cpu);
+		return;
+	}
+
+	/* Check that the device is attached to a genpd. */
+	if (!(dev->pm_domain && dev->pm_domain->detach == genpd_dev_pm_detach))
+		return;
+
+	genpd_dev_pm_detach(dev, true);
+
+	pm_runtime_disable(dev);
+	pm_runtime_put_noidle(dev);
+	pm_runtime_reinit(dev);
+
+	dev_info(dev, "genpd: detached cpu\n");
+}
+EXPORT_SYMBOL(of_genpd_detach_cpu);
+
 static const struct of_device_id idle_state_match[] = {
 	{ .compatible = "domain-idle-state", },
 	{ }
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 29ab00c..ccd7c94 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -249,6 +249,8 @@ extern int of_genpd_parse_idle_states(struct device_node *dn,
 			struct genpd_power_state **states, int *n);
 
 int genpd_dev_pm_attach(struct device *dev);
+int of_genpd_attach_cpu(int cpu);
+void of_genpd_detach_cpu(int cpu);
 #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
 static inline int of_genpd_add_provider_simple(struct device_node *np,
 					struct generic_pm_domain *genpd)
@@ -287,6 +289,13 @@ static inline int genpd_dev_pm_attach(struct device *dev)
 	return -ENODEV;
 }
 
+static inline int of_genpd_attach_cpu(int cpu)
+{
+	return -ENODEV;
+}
+
+static inline void of_genpd_detach_cpu(int cpu) {}
+
 static inline
 struct generic_pm_domain *of_genpd_remove_last(struct device_node *np)
 {
-- 
2.7.4

[PATCH v6 06/25] timer: Export next wakeup time of a CPU

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Knowing the sleep duration of CPUs, is known to be needed while selecting
the most energy efficient idle state for a CPU or a group of CPUs.

However, to be able to compute the sleep duration, we need to know at what
time the next expected wakeup is for the CPU. Therefore, let's export this
information via a new function, tick_nohz_get_next_wakeup(). Following
changes make use of it.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 include/linux/tick.h     | 10 ++++++++++
 kernel/time/tick-sched.c | 11 +++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index 7cc3592..e51cf2c 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -120,6 +120,7 @@ extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
 extern unsigned long tick_nohz_get_idle_calls(void);
 extern unsigned long tick_nohz_get_idle_calls_cpu(int cpu);
+extern ktime_t tick_nohz_get_next_wakeup(int cpu);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #else /* !CONFIG_NO_HZ_COMMON */
@@ -132,6 +133,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
 {
 	return NSEC_PER_SEC / HZ;
 }
+
+static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	ktime_t len = NSEC_PER_SEC/HZ;
+
+	/* Next wake up is the tick period, assume it starts now */
+	return ktime_add(len, ktime_get());
+}
+
 static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
 #endif /* !CONFIG_NO_HZ_COMMON */
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 29a5733..da1bff3 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1025,6 +1025,17 @@ unsigned long tick_nohz_get_idle_calls(void)
 	return ts->idle_calls;
 }
 
+/**
+ * tick_nohz_get_next_wakeup - return the next wake up of the CPU
+ */
+ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	struct clock_event_device *dev =
+			per_cpu(tick_cpu_device.evtdev, cpu);
+
+	return dev->next_event;
+}
+
 static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
 {
 #ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
-- 
2.7.4

[PATCH v6 06/25] timer: Export next wakeup time of a CPU

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Knowing the sleep duration of CPUs, is known to be needed while selecting
the most energy efficient idle state for a CPU or a group of CPUs.

However, to be able to compute the sleep duration, we need to know at what
time the next expected wakeup is for the CPU. Therefore, let's export this
information via a new function, tick_nohz_get_next_wakeup(). Following
changes make use of it.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 include/linux/tick.h     | 10 ++++++++++
 kernel/time/tick-sched.c | 11 +++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index 7cc3592..e51cf2c 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -120,6 +120,7 @@ extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
 extern unsigned long tick_nohz_get_idle_calls(void);
 extern unsigned long tick_nohz_get_idle_calls_cpu(int cpu);
+extern ktime_t tick_nohz_get_next_wakeup(int cpu);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #else /* !CONFIG_NO_HZ_COMMON */
@@ -132,6 +133,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
 {
 	return NSEC_PER_SEC / HZ;
 }
+
+static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	ktime_t len = NSEC_PER_SEC/HZ;
+
+	/* Next wake up is the tick period, assume it starts now */
+	return ktime_add(len, ktime_get());
+}
+
 static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
 #endif /* !CONFIG_NO_HZ_COMMON */
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 29a5733..da1bff3 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1025,6 +1025,17 @@ unsigned long tick_nohz_get_idle_calls(void)
 	return ts->idle_calls;
 }
 
+/**
+ * tick_nohz_get_next_wakeup - return the next wake up of the CPU
+ */
+ktime_t tick_nohz_get_next_wakeup(int cpu)
+{
+	struct clock_event_device *dev =
+			per_cpu(tick_cpu_device.evtdev, cpu);
+
+	return dev->next_event;
+}
+
 static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
 {
 #ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
-- 
2.7.4

[PATCH v6 07/25] PM / Domains: Add genpd governor for CPUs

[Ulf Hansson]

As it's now perfectly possible that a PM domain managed by genpd contains
devices belonging to CPUs, we should start to take into account the
residency values for the idle states during the state selection process.
The residency value specifies the minimum duration of time, the CPU or a
group of CPUs, needs to spend in an idle state to not waste energy entering
it.

To deal with this, let's add a new genpd governor, pm_domain_cpu_gov, that
may be used for a PM domain that have CPU devices attached or if the CPUs
are attached through subdomains.

The new governor computes the minimum expected idle duration time for the
online CPUs being attached to the PM domain and its subdomains. Then in the
state selection process, trying the deepest state first, it verifies that
the idle duration time satisfies the state's residency value.

It should be noted that, when computing the minimum expected idle duration
time, we use the information from tick_nohz_get_next_wakeup(), to find the
next wakeup for the related CPUs. Future wise, this may deserve to be
improved, as there are more reasons to why a CPU may be woken up from idle.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain_governor.c | 56 ++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h            |  2 ++
 2 files changed, 58 insertions(+)

diff --git a/drivers/base/power/domain_governor.c b/drivers/base/power/domain_governor.c
index 99896fb..70565a8 100644
--- a/drivers/base/power/domain_governor.c
+++ b/drivers/base/power/domain_governor.c
@@ -10,6 +10,9 @@
 #include <linux/pm_domain.h>
 #include <linux/pm_qos.h>
 #include <linux/hrtimer.h>
+#include <linux/cpumask.h>
+#include <linux/ktime.h>
+#include <linux/tick.h>
 
 static int dev_update_qos_constraint(struct device *dev, void *data)
 {
@@ -245,6 +248,54 @@ static bool always_on_power_down_ok(struct dev_pm_domain *domain)
 	return false;
 }
 
+static bool cpu_power_down_ok(struct dev_pm_domain *pd)
+{
+	struct generic_pm_domain *genpd = pd_to_genpd(pd);
+	ktime_t domain_wakeup, cpu_wakeup;
+	s64 idle_duration_ns;
+	int cpu, i;
+
+	if (!(genpd->flags & GENPD_FLAG_CPU_DOMAIN))
+		return true;
+
+	/*
+	 * Find the next wakeup for any of the online CPUs within the PM domain
+	 * and its subdomains. Note, we only need the genpd->cpus, as it already
+	 * contains a mask of all CPUs from subdomains.
+	 */
+	domain_wakeup = ktime_set(KTIME_SEC_MAX, 0);
+	for_each_cpu_and(cpu, genpd->cpus, cpu_online_mask) {
+		cpu_wakeup = tick_nohz_get_next_wakeup(cpu);
+		if (ktime_before(cpu_wakeup, domain_wakeup))
+			domain_wakeup = cpu_wakeup;
+	}
+
+	/* The minimum idle duration is from now - until the next wakeup. */
+	idle_duration_ns = ktime_to_ns(ktime_sub(domain_wakeup, ktime_get()));
+
+	/*
+	 * Find the deepest idle state, that has its residency value satisfied.
+	 * Start at the deepest supported state.
+	 */
+	i = genpd->state_count - 1;
+	do {
+		if (!genpd->states[i].residency_ns)
+			break;
+
+		/* Check idle_duration_ns >= 0 to compare signed/unsigned. */
+		if (idle_duration_ns >= 0 &&
+		    idle_duration_ns >= genpd->states[i].residency_ns)
+			break;
+		i--;
+	} while (i >= 0);
+
+	if (i < 0)
+		return false;
+
+	genpd->state_idx = i;
+	return true;
+}
+
 struct dev_power_governor simple_qos_governor = {
 	.suspend_ok = default_suspend_ok,
 	.power_down_ok = default_power_down_ok,
@@ -257,3 +308,8 @@ struct dev_power_governor pm_domain_always_on_gov = {
 	.power_down_ok = always_on_power_down_ok,
 	.suspend_ok = default_suspend_ok,
 };
+
+struct dev_power_governor pm_domain_cpu_gov = {
+	.suspend_ok = NULL,
+	.power_down_ok = cpu_power_down_ok,
+};
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index ccd7c94..8f60181 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -161,6 +161,7 @@ extern int dev_pm_genpd_set_performance_state(struct device *dev,
 
 extern struct dev_power_governor simple_qos_governor;
 extern struct dev_power_governor pm_domain_always_on_gov;
+extern struct dev_power_governor pm_domain_cpu_gov;
 #else
 
 static inline struct generic_pm_domain_data *dev_gpd_data(struct device *dev)
@@ -206,6 +207,7 @@ static inline int dev_pm_genpd_set_performance_state(struct device *dev,
 
 #define simple_qos_governor		(*(struct dev_power_governor *)(NULL))
 #define pm_domain_always_on_gov		(*(struct dev_power_governor *)(NULL))
+#define pm_domain_cpu_gov		(*(struct dev_power_governor *)(NULL))
 #endif
 
 static inline int pm_genpd_add_device(struct generic_pm_domain *genpd,
-- 
2.7.4

[PATCH v6 07/25] PM / Domains: Add genpd governor for CPUs

[Ulf Hansson]

As it's now perfectly possible that a PM domain managed by genpd contains
devices belonging to CPUs, we should start to take into account the
residency values for the idle states during the state selection process.
The residency value specifies the minimum duration of time, the CPU or a
group of CPUs, needs to spend in an idle state to not waste energy entering
it.

To deal with this, let's add a new genpd governor, pm_domain_cpu_gov, that
may be used for a PM domain that have CPU devices attached or if the CPUs
are attached through subdomains.

The new governor computes the minimum expected idle duration time for the
online CPUs being attached to the PM domain and its subdomains. Then in the
state selection process, trying the deepest state first, it verifies that
the idle duration time satisfies the state's residency value.

It should be noted that, when computing the minimum expected idle duration
time, we use the information from tick_nohz_get_next_wakeup(), to find the
next wakeup for the related CPUs. Future wise, this may deserve to be
improved, as there are more reasons to why a CPU may be woken up from idle.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain_governor.c | 56 ++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h            |  2 ++
 2 files changed, 58 insertions(+)

diff --git a/drivers/base/power/domain_governor.c b/drivers/base/power/domain_governor.c
index 99896fb..70565a8 100644
--- a/drivers/base/power/domain_governor.c
+++ b/drivers/base/power/domain_governor.c
@@ -10,6 +10,9 @@
 #include <linux/pm_domain.h>
 #include <linux/pm_qos.h>
 #include <linux/hrtimer.h>
+#include <linux/cpumask.h>
+#include <linux/ktime.h>
+#include <linux/tick.h>
 
 static int dev_update_qos_constraint(struct device *dev, void *data)
 {
@@ -245,6 +248,54 @@ static bool always_on_power_down_ok(struct dev_pm_domain *domain)
 	return false;
 }
 
+static bool cpu_power_down_ok(struct dev_pm_domain *pd)
+{
+	struct generic_pm_domain *genpd = pd_to_genpd(pd);
+	ktime_t domain_wakeup, cpu_wakeup;
+	s64 idle_duration_ns;
+	int cpu, i;
+
+	if (!(genpd->flags & GENPD_FLAG_CPU_DOMAIN))
+		return true;
+
+	/*
+	 * Find the next wakeup for any of the online CPUs within the PM domain
+	 * and its subdomains. Note, we only need the genpd->cpus, as it already
+	 * contains a mask of all CPUs from subdomains.
+	 */
+	domain_wakeup = ktime_set(KTIME_SEC_MAX, 0);
+	for_each_cpu_and(cpu, genpd->cpus, cpu_online_mask) {
+		cpu_wakeup = tick_nohz_get_next_wakeup(cpu);
+		if (ktime_before(cpu_wakeup, domain_wakeup))
+			domain_wakeup = cpu_wakeup;
+	}
+
+	/* The minimum idle duration is from now - until the next wakeup. */
+	idle_duration_ns = ktime_to_ns(ktime_sub(domain_wakeup, ktime_get()));
+
+	/*
+	 * Find the deepest idle state, that has its residency value satisfied.
+	 * Start at the deepest supported state.
+	 */
+	i = genpd->state_count - 1;
+	do {
+		if (!genpd->states[i].residency_ns)
+			break;
+
+		/* Check idle_duration_ns >= 0 to compare signed/unsigned. */
+		if (idle_duration_ns >= 0 &&
+		    idle_duration_ns >= genpd->states[i].residency_ns)
+			break;
+		i--;
+	} while (i >= 0);
+
+	if (i < 0)
+		return false;
+
+	genpd->state_idx = i;
+	return true;
+}
+
 struct dev_power_governor simple_qos_governor = {
 	.suspend_ok = default_suspend_ok,
 	.power_down_ok = default_power_down_ok,
@@ -257,3 +308,8 @@ struct dev_power_governor pm_domain_always_on_gov = {
 	.power_down_ok = always_on_power_down_ok,
 	.suspend_ok = default_suspend_ok,
 };
+
+struct dev_power_governor pm_domain_cpu_gov = {
+	.suspend_ok = NULL,
+	.power_down_ok = cpu_power_down_ok,
+};
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index ccd7c94..8f60181 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -161,6 +161,7 @@ extern int dev_pm_genpd_set_performance_state(struct device *dev,
 
 extern struct dev_power_governor simple_qos_governor;
 extern struct dev_power_governor pm_domain_always_on_gov;
+extern struct dev_power_governor pm_domain_cpu_gov;
 #else
 
 static inline struct generic_pm_domain_data *dev_gpd_data(struct device *dev)
@@ -206,6 +207,7 @@ static inline int dev_pm_genpd_set_performance_state(struct device *dev,
 
 #define simple_qos_governor		(*(struct dev_power_governor *)(NULL))
 #define pm_domain_always_on_gov		(*(struct dev_power_governor *)(NULL))
+#define pm_domain_cpu_gov		(*(struct dev_power_governor *)(NULL))
 #endif
 
 static inline int pm_genpd_add_device(struct generic_pm_domain *genpd,
-- 
2.7.4

[PATCH v6 08/25] PM / Domains: Extend genpd CPU governor to cope with QoS constraints

[Ulf Hansson]

CPU devices and other regular devices may share the same PM domain and may
also be hierarchically related via subdomains. In either case, all devices
including CPUs, may be attached to a PM domain managed by genpd, that has
an idle state with an enter/exit latency.

Let's take these latencies into account in the state selection process by
genpd's governor for CPUs. This means the governor, pm_domain_cpu_gov,
becomes extended to satisfy both a state's residency and a potential dev PM
QoS constraint.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain_governor.c | 15 +++++++++++----
 include/linux/pm_domain.h            |  1 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/base/power/domain_governor.c b/drivers/base/power/domain_governor.c
index 70565a8..2f162bc 100644
--- a/drivers/base/power/domain_governor.c
+++ b/drivers/base/power/domain_governor.c
@@ -214,8 +214,10 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 	struct generic_pm_domain *genpd = pd_to_genpd(pd);
 	struct gpd_link *link;
 
-	if (!genpd->max_off_time_changed)
+	if (!genpd->max_off_time_changed) {
+		genpd->state_idx = genpd->cached_power_down_state_idx;
 		return genpd->cached_power_down_ok;
+	}
 
 	/*
 	 * We have to invalidate the cached results for the masters, so
@@ -240,6 +242,7 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 		genpd->state_idx--;
 	}
 
+	genpd->cached_power_down_state_idx = genpd->state_idx;
 	return genpd->cached_power_down_ok;
 }
 
@@ -255,6 +258,10 @@ static bool cpu_power_down_ok(struct dev_pm_domain *pd)
 	s64 idle_duration_ns;
 	int cpu, i;
 
+	/* Validate dev PM QoS constraints. */
+	if (!default_power_down_ok(pd))
+		return false;
+
 	if (!(genpd->flags & GENPD_FLAG_CPU_DOMAIN))
 		return true;
 
@@ -275,9 +282,9 @@ static bool cpu_power_down_ok(struct dev_pm_domain *pd)
 
 	/*
 	 * Find the deepest idle state, that has its residency value satisfied.
-	 * Start at the deepest supported state.
+	 * Start at the state picked by the dev PM QoS constraint validation.
 	 */
-	i = genpd->state_count - 1;
+	i = genpd->state_idx;
 	do {
 		if (!genpd->states[i].residency_ns)
 			break;
@@ -310,6 +317,6 @@ struct dev_power_governor pm_domain_always_on_gov = {
 };
 
 struct dev_power_governor pm_domain_cpu_gov = {
-	.suspend_ok = NULL,
+	.suspend_ok = default_suspend_ok,
 	.power_down_ok = cpu_power_down_ok,
 };
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 8f60181..e7910d2 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -77,6 +77,7 @@ struct generic_pm_domain {
 	s64 max_off_time_ns;	/* Maximum allowed "suspended" time. */
 	bool max_off_time_changed;
 	bool cached_power_down_ok;
+	bool cached_power_down_state_idx;
 	int (*attach_dev)(struct generic_pm_domain *domain,
 			  struct device *dev);
 	void (*detach_dev)(struct generic_pm_domain *domain,
-- 
2.7.4

[PATCH v6 08/25] PM / Domains: Extend genpd CPU governor to cope with QoS constraints

[Ulf Hansson]

CPU devices and other regular devices may share the same PM domain and may
also be hierarchically related via subdomains. In either case, all devices
including CPUs, may be attached to a PM domain managed by genpd, that has
an idle state with an enter/exit latency.

Let's take these latencies into account in the state selection process by
genpd's governor for CPUs. This means the governor, pm_domain_cpu_gov,
becomes extended to satisfy both a state's residency and a potential dev PM
QoS constraint.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/base/power/domain_governor.c | 15 +++++++++++----
 include/linux/pm_domain.h            |  1 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/base/power/domain_governor.c b/drivers/base/power/domain_governor.c
index 70565a8..2f162bc 100644
--- a/drivers/base/power/domain_governor.c
+++ b/drivers/base/power/domain_governor.c
@@ -214,8 +214,10 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 	struct generic_pm_domain *genpd = pd_to_genpd(pd);
 	struct gpd_link *link;
 
-	if (!genpd->max_off_time_changed)
+	if (!genpd->max_off_time_changed) {
+		genpd->state_idx = genpd->cached_power_down_state_idx;
 		return genpd->cached_power_down_ok;
+	}
 
 	/*
 	 * We have to invalidate the cached results for the masters, so
@@ -240,6 +242,7 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 		genpd->state_idx--;
 	}
 
+	genpd->cached_power_down_state_idx = genpd->state_idx;
 	return genpd->cached_power_down_ok;
 }
 
@@ -255,6 +258,10 @@ static bool cpu_power_down_ok(struct dev_pm_domain *pd)
 	s64 idle_duration_ns;
 	int cpu, i;
 
+	/* Validate dev PM QoS constraints. */
+	if (!default_power_down_ok(pd))
+		return false;
+
 	if (!(genpd->flags & GENPD_FLAG_CPU_DOMAIN))
 		return true;
 
@@ -275,9 +282,9 @@ static bool cpu_power_down_ok(struct dev_pm_domain *pd)
 
 	/*
 	 * Find the deepest idle state, that has its residency value satisfied.
-	 * Start at the deepest supported state.
+	 * Start at the state picked by the dev PM QoS constraint validation.
 	 */
-	i = genpd->state_count - 1;
+	i = genpd->state_idx;
 	do {
 		if (!genpd->states[i].residency_ns)
 			break;
@@ -310,6 +317,6 @@ struct dev_power_governor pm_domain_always_on_gov = {
 };
 
 struct dev_power_governor pm_domain_cpu_gov = {
-	.suspend_ok = NULL,
+	.suspend_ok = default_suspend_ok,
 	.power_down_ok = cpu_power_down_ok,
 };
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 8f60181..e7910d2 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -77,6 +77,7 @@ struct generic_pm_domain {
 	s64 max_off_time_ns;	/* Maximum allowed "suspended" time. */
 	bool max_off_time_changed;
 	bool cached_power_down_ok;
+	bool cached_power_down_state_idx;
 	int (*attach_dev)(struct generic_pm_domain *domain,
 			  struct device *dev);
 	void (*detach_dev)(struct generic_pm_domain *domain,
-- 
2.7.4

[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM in the idle path for CPUs

[Ulf Hansson]

To allow CPUs being power managed by PM domains, let's deploy support for
runtime PM for the CPU's corresponding struct device.

More precisely, at the point when the CPU is about to enter an idle state,
decrease the runtime PM usage count for its corresponding struct device,
via calling pm_runtime_put_sync_suspend(). Then, at the point when the CPU
resumes from idle, let's increase the runtime PM usage count, via calling
pm_runtime_get_sync().

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 kernel/cpu_pm.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 67b02e1..71317ff 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -16,9 +16,11 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/cpu.h>
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/pm_runtime.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -91,6 +93,7 @@ int cpu_pm_enter(void)
 {
 	int nr_calls;
 	int ret = 0;
+	struct device *dev = get_cpu_device(smp_processor_id());
 
 	ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
 	if (ret)
@@ -100,6 +103,9 @@ int cpu_pm_enter(void)
 		 */
 		cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
 
+	if (!ret && dev)
+		pm_runtime_put_sync_suspend(dev);
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_enter);
@@ -118,6 +124,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
  */
 int cpu_pm_exit(void)
 {
+	struct device *dev = get_cpu_device(smp_processor_id());
+
+	if (dev)
+		pm_runtime_get_sync(dev);
+
 	return cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
 }
 EXPORT_SYMBOL_GPL(cpu_pm_exit);
-- 
2.7.4

[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM in the idle path for CPUs

[Ulf Hansson]

To allow CPUs being power managed by PM domains, let's deploy support for
runtime PM for the CPU's corresponding struct device.

More precisely, at the point when the CPU is about to enter an idle state,
decrease the runtime PM usage count for its corresponding struct device,
via calling pm_runtime_put_sync_suspend(). Then, at the point when the CPU
resumes from idle, let's increase the runtime PM usage count, via calling
pm_runtime_get_sync().

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 kernel/cpu_pm.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 67b02e1..71317ff 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -16,9 +16,11 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/cpu.h>
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/pm_runtime.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -91,6 +93,7 @@ int cpu_pm_enter(void)
 {
 	int nr_calls;
 	int ret = 0;
+	struct device *dev = get_cpu_device(smp_processor_id());
 
 	ret = cpu_pm_notify(CPU_PM_ENTER, -1, &nr_calls);
 	if (ret)
@@ -100,6 +103,9 @@ int cpu_pm_enter(void)
 		 */
 		cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);
 
+	if (!ret && dev)
+		pm_runtime_put_sync_suspend(dev);
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_enter);
@@ -118,6 +124,11 @@ EXPORT_SYMBOL_GPL(cpu_pm_enter);
  */
 int cpu_pm_exit(void)
 {
+	struct device *dev = get_cpu_device(smp_processor_id());
+
+	if (dev)
+		pm_runtime_get_sync(dev);
+
 	return cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
 }
 EXPORT_SYMBOL_GPL(cpu_pm_exit);
-- 
2.7.4

[PATCH v6 10/25] dt: psci: Update DT bindings to support hierarchical PSCI states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Update DT bindings to represent hierarchical CPU and CPU PM domain idle
states for PSCI. Also update the PSCI examples to clearly show how
flattened and hierarchical idle states can be represented in DT.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
Reviewed-by: Rob Herring <robh@kernel.org>
Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
---
 Documentation/devicetree/bindings/arm/psci.txt | 156 +++++++++++++++++++++++++
 1 file changed, 156 insertions(+)

diff --git a/Documentation/devicetree/bindings/arm/psci.txt b/Documentation/devicetree/bindings/arm/psci.txt
index a2c4f1d..17aa3d3 100644
--- a/Documentation/devicetree/bindings/arm/psci.txt
+++ b/Documentation/devicetree/bindings/arm/psci.txt
@@ -105,7 +105,163 @@ Case 3: PSCI v0.2 and PSCI v0.1.
 		...
 	};
 
+ARM systems can have multiple cores sometimes in hierarchical arrangement.
+This often, but not always, maps directly to the processor power topology of
+the system. Individual nodes in a topology have their own specific power states
+and can be better represented in DT hierarchically.
+
+For these cases, the definitions of the idle states for the CPUs and the CPU
+topology, must conform to the domain idle state specification [3]. The domain
+idle states themselves, must be compatible with the defined 'domain-idle-state'
+binding [1], and also need to specify the arm,psci-suspend-param property for
+each idle state.
+
+DT allows representing CPUs and CPU idle states in two different ways -
+
+The flattened model as given in Example 1, lists CPU's idle states followed by
+the domain idle state that the CPUs may choose. Note that the idle states are
+all compatible with "arm,idle-state".
+
+Example 2 represents the hierarchical model of CPUs and domain idle states.
+CPUs define their domain provider in their psci DT node. The domain controls
+the power to the CPU and possibly other h/w blocks that would enter an idle
+state along with the CPU. The CPU's idle states may therefore be considered as
+the domain's idle states and have the compatible "arm,idle-state". Such domains
+may also be embedded within another domain that may represent common h/w blocks
+between these CPUs. The idle states of the CPU topology shall be represented as
+the domain's idle states.
+
+In PSCI firmware v1.0, the OS-Initiated mode is introduced. In order to use it,
+the hierarchical representation must be used.
+
+Example 1: Flattened representation of CPU and domain idle states
+	cpus {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		CPU0: cpu@0 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a53", "arm,armv8";
+			reg = <0x0>;
+			enable-method = "psci";
+			cpu-idle-states = <&CPU_PWRDN>, <&CLUSTER_RET>,
+					  <&CLUSTER_PWRDN>;
+		};
+
+		CPU1: cpu@1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a57", "arm,armv8";
+			reg = <0x100>;
+			enable-method = "psci";
+			cpu-idle-states = <&CPU_PWRDN>, <&CLUSTER_RET>,
+					  <&CLUSTER_PWRDN>;
+		};
+
+		idle-states {
+			CPU_PWRDN: cpu-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x000001>;
+				entry-latency-us = <10>;
+				exit-latency-us = <10>;
+				min-residency-us = <100>;
+			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
+	};
+
+	psci {
+		compatible = "arm,psci-0.2";
+		method = "smc";
+	};
+
+Example 2: Hierarchical representation of CPU and domain idle states
+
+	cpus {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		CPU0: cpu@0 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a53", "arm,armv8";
+			reg = <0x0>;
+			enable-method = "psci";
+			power-domains = <&CPU_PD0>;
+		};
+
+		CPU1: cpu@1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a57", "arm,armv8";
+			reg = <0x100>;
+			enable-method = "psci";
+			power-domains = <&CPU_PD1>;
+		};
+
+		idle-states {
+			CPU_PWRDN: cpu-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x000001>;
+				entry-latency-us = <10>;
+				exit-latency-us = <10>;
+				min-residency-us = <100>;
+			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-power-down {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
+		};
+	};
+
+	psci {
+		compatible = "arm,psci-1.0";
+		method = "smc";
+
+		CPU_PD0: cpu-pd0 {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CPU_PWRDN>;
+			power-domains = <&CLUSTER_PD>;
+		};
+
+		CPU_PD1: cpu-pd1 {
+			#power-domain-cells = <0>;
+			domain-idle-states =  <&CPU_PWRDN>;
+			power-domains = <&CLUSTER_PD>;
+		};
+
+		CLUSTER_PD: cluster-pd {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CLUSTER_RET>, <&CLUSTER_PWRDN>;
+		};
+	};
+
 [1] Kernel documentation - ARM idle states bindings
     Documentation/devicetree/bindings/arm/idle-states.txt
 [2] Power State Coordination Interface (PSCI) specification
     http://infocenter.arm.com/help/topic/com.arm.doc.den0022c/DEN0022C_Power_State_Coordination_Interface.pdf
+[3]. PM Domains description
+    Documentation/devicetree/bindings/power/power_domain.txt
-- 
2.7.4

[PATCH v6 10/25] dt: psci: Update DT bindings to support hierarchical PSCI states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Update DT bindings to represent hierarchical CPU and CPU PM domain idle
states for PSCI. Also update the PSCI examples to clearly show how
flattened and hierarchical idle states can be represented in DT.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
Reviewed-by: Rob Herring <robh@kernel.org>
Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
---
 Documentation/devicetree/bindings/arm/psci.txt | 156 +++++++++++++++++++++++++
 1 file changed, 156 insertions(+)

diff --git a/Documentation/devicetree/bindings/arm/psci.txt b/Documentation/devicetree/bindings/arm/psci.txt
index a2c4f1d..17aa3d3 100644
--- a/Documentation/devicetree/bindings/arm/psci.txt
+++ b/Documentation/devicetree/bindings/arm/psci.txt
@@ -105,7 +105,163 @@ Case 3: PSCI v0.2 and PSCI v0.1.
 		...
 	};
 
+ARM systems can have multiple cores sometimes in hierarchical arrangement.
+This often, but not always, maps directly to the processor power topology of
+the system. Individual nodes in a topology have their own specific power states
+and can be better represented in DT hierarchically.
+
+For these cases, the definitions of the idle states for the CPUs and the CPU
+topology, must conform to the domain idle state specification [3]. The domain
+idle states themselves, must be compatible with the defined 'domain-idle-state'
+binding [1], and also need to specify the arm,psci-suspend-param property for
+each idle state.
+
+DT allows representing CPUs and CPU idle states in two different ways -
+
+The flattened model as given in Example 1, lists CPU's idle states followed by
+the domain idle state that the CPUs may choose. Note that the idle states are
+all compatible with "arm,idle-state".
+
+Example 2 represents the hierarchical model of CPUs and domain idle states.
+CPUs define their domain provider in their psci DT node. The domain controls
+the power to the CPU and possibly other h/w blocks that would enter an idle
+state along with the CPU. The CPU's idle states may therefore be considered as
+the domain's idle states and have the compatible "arm,idle-state". Such domains
+may also be embedded within another domain that may represent common h/w blocks
+between these CPUs. The idle states of the CPU topology shall be represented as
+the domain's idle states.
+
+In PSCI firmware v1.0, the OS-Initiated mode is introduced. In order to use it,
+the hierarchical representation must be used.
+
+Example 1: Flattened representation of CPU and domain idle states
+	cpus {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		CPU0: cpu at 0 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a53", "arm,armv8";
+			reg = <0x0>;
+			enable-method = "psci";
+			cpu-idle-states = <&CPU_PWRDN>, <&CLUSTER_RET>,
+					  <&CLUSTER_PWRDN>;
+		};
+
+		CPU1: cpu at 1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a57", "arm,armv8";
+			reg = <0x100>;
+			enable-method = "psci";
+			cpu-idle-states = <&CPU_PWRDN>, <&CLUSTER_RET>,
+					  <&CLUSTER_PWRDN>;
+		};
+
+		idle-states {
+			CPU_PWRDN: cpu-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x000001>;
+				entry-latency-us = <10>;
+				exit-latency-us = <10>;
+				min-residency-us = <100>;
+			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
+	};
+
+	psci {
+		compatible = "arm,psci-0.2";
+		method = "smc";
+	};
+
+Example 2: Hierarchical representation of CPU and domain idle states
+
+	cpus {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		CPU0: cpu at 0 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a53", "arm,armv8";
+			reg = <0x0>;
+			enable-method = "psci";
+			power-domains = <&CPU_PD0>;
+		};
+
+		CPU1: cpu at 1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a57", "arm,armv8";
+			reg = <0x100>;
+			enable-method = "psci";
+			power-domains = <&CPU_PD1>;
+		};
+
+		idle-states {
+			CPU_PWRDN: cpu-power-down {
+				compatible = "arm,idle-state";
+				arm,psci-suspend-param = <0x000001>;
+				entry-latency-us = <10>;
+				exit-latency-us = <10>;
+				min-residency-us = <100>;
+			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-power-down {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
+		};
+	};
+
+	psci {
+		compatible = "arm,psci-1.0";
+		method = "smc";
+
+		CPU_PD0: cpu-pd0 {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CPU_PWRDN>;
+			power-domains = <&CLUSTER_PD>;
+		};
+
+		CPU_PD1: cpu-pd1 {
+			#power-domain-cells = <0>;
+			domain-idle-states =  <&CPU_PWRDN>;
+			power-domains = <&CLUSTER_PD>;
+		};
+
+		CLUSTER_PD: cluster-pd {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CLUSTER_RET>, <&CLUSTER_PWRDN>;
+		};
+	};
+
 [1] Kernel documentation - ARM idle states bindings
     Documentation/devicetree/bindings/arm/idle-states.txt
 [2] Power State Coordination Interface (PSCI) specification
     http://infocenter.arm.com/help/topic/com.arm.doc.den0022c/DEN0022C_Power_State_Coordination_Interface.pdf
+[3]. PM Domains description
+    Documentation/devicetree/bindings/power/power_domain.txt
-- 
2.7.4

[PATCH v6 11/25] of: base: Add of_get_cpu_state_node() to get idle states for a CPU node

[Ulf Hansson]

The CPU's idle state nodes are currently parsed at the common cpuidle DT
library, but also when initializing back-end data for the arch specific CPU
operations, as in the PSCI driver case.

To avoid open-coding, let's introduce of_get_cpu_state_node(), which takes
the device node for the CPU and the index to the requested idle state node,
as in-parameters. In case a corresponding idle state node is found, it
returns the node with the refcount incremented for it, else it returns
NULL.

Moreover, for ARM, there are two generic methods, to describe the CPU's
idle states, either via the flattened description through the
"cpu-idle-states" binding [1] or via the hierarchical layout, using the
"power-domains" and the "domain-idle-states" bindings [2]. Hence, let's
take both options into account.

[1]
Documentation/devicetree/bindings/arm/idle-states.txt
[2]
Documentation/devicetree/bindings/arm/psci.txt

Cc: Rob Herring <robh+dt@kernel.org>
Cc: devicetree@vger.kernel.org
Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/of/base.c  | 35 +++++++++++++++++++++++++++++++++++
 include/linux/of.h |  8 ++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/of/base.c b/drivers/of/base.c
index ad28de9..88d36a3 100644
--- a/drivers/of/base.c
+++ b/drivers/of/base.c
@@ -338,6 +338,41 @@ int of_cpu_node_to_id(struct device_node *cpu_node)
 EXPORT_SYMBOL(of_cpu_node_to_id);
 
 /**
+ * of_get_cpu_state_node - Get CPU's idle state node at the given index
+ *
+ * @cpu_node: The device node for the CPU
+ * @index: The index in the list of the idle states
+ *
+ * Two generic methods can be used to describe a CPU's idle states, either via
+ * a flattened description through the "cpu-idle-states" binding or via the
+ * hierarchical layout, using the "power-domains" and the "domain-idle-states"
+ * bindings. This function check for both and returns the idle state node for
+ * the requested index.
+ *
+ * In case and idle state node is found at index, the refcount incremented for
+ * it, so call of_node_put() on it when done. Returns NULL if not found.
+ */
+struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+					  int index)
+{
+	struct of_phandle_args args;
+	int err;
+
+	err = of_parse_phandle_with_args(cpu_node, "power-domains",
+					"#power-domain-cells", 0, &args);
+	if (!err) {
+		struct device_node *state_node =
+			of_parse_phandle(args.np, "domain-idle-states", index);
+
+		of_node_put(args.np);
+		return state_node;
+	}
+
+	return of_parse_phandle(cpu_node, "cpu-idle-states", index);
+}
+EXPORT_SYMBOL(of_get_cpu_state_node);
+
+/**
  * __of_device_is_compatible() - Check if the node matches given constraints
  * @device: pointer to node
  * @compat: required compatible string, NULL or "" for any match
diff --git a/include/linux/of.h b/include/linux/of.h
index da1ee95..288233c 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -348,6 +348,8 @@ extern const void *of_get_property(const struct device_node *node,
 				const char *name,
 				int *lenp);
 extern struct device_node *of_get_cpu_node(int cpu, unsigned int *thread);
+extern struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+						 int index);
 #define for_each_property_of_node(dn, pp) \
 	for (pp = dn->properties; pp != NULL; pp = pp->next)
 
@@ -730,6 +732,12 @@ static inline struct device_node *of_get_cpu_node(int cpu,
 	return NULL;
 }
 
+static inline struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+					int index)
+{
+	return NULL;
+}
+
 static inline int of_n_addr_cells(struct device_node *np)
 {
 	return 0;
-- 
2.7.4

[PATCH v6 11/25] of: base: Add of_get_cpu_state_node() to get idle states for a CPU node

[Ulf Hansson]

The CPU's idle state nodes are currently parsed at the common cpuidle DT
library, but also when initializing back-end data for the arch specific CPU
operations, as in the PSCI driver case.

To avoid open-coding, let's introduce of_get_cpu_state_node(), which takes
the device node for the CPU and the index to the requested idle state node,
as in-parameters. In case a corresponding idle state node is found, it
returns the node with the refcount incremented for it, else it returns
NULL.

Moreover, for ARM, there are two generic methods, to describe the CPU's
idle states, either via the flattened description through the
"cpu-idle-states" binding [1] or via the hierarchical layout, using the
"power-domains" and the "domain-idle-states" bindings [2]. Hence, let's
take both options into account.

[1]
Documentation/devicetree/bindings/arm/idle-states.txt
[2]
Documentation/devicetree/bindings/arm/psci.txt

Cc: Rob Herring <robh+dt@kernel.org>
Cc: devicetree at vger.kernel.org
Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/of/base.c  | 35 +++++++++++++++++++++++++++++++++++
 include/linux/of.h |  8 ++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/of/base.c b/drivers/of/base.c
index ad28de9..88d36a3 100644
--- a/drivers/of/base.c
+++ b/drivers/of/base.c
@@ -338,6 +338,41 @@ int of_cpu_node_to_id(struct device_node *cpu_node)
 EXPORT_SYMBOL(of_cpu_node_to_id);
 
 /**
+ * of_get_cpu_state_node - Get CPU's idle state node at the given index
+ *
+ * @cpu_node: The device node for the CPU
+ * @index: The index in the list of the idle states
+ *
+ * Two generic methods can be used to describe a CPU's idle states, either via
+ * a flattened description through the "cpu-idle-states" binding or via the
+ * hierarchical layout, using the "power-domains" and the "domain-idle-states"
+ * bindings. This function check for both and returns the idle state node for
+ * the requested index.
+ *
+ * In case and idle state node is found at index, the refcount incremented for
+ * it, so call of_node_put() on it when done. Returns NULL if not found.
+ */
+struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+					  int index)
+{
+	struct of_phandle_args args;
+	int err;
+
+	err = of_parse_phandle_with_args(cpu_node, "power-domains",
+					"#power-domain-cells", 0, &args);
+	if (!err) {
+		struct device_node *state_node =
+			of_parse_phandle(args.np, "domain-idle-states", index);
+
+		of_node_put(args.np);
+		return state_node;
+	}
+
+	return of_parse_phandle(cpu_node, "cpu-idle-states", index);
+}
+EXPORT_SYMBOL(of_get_cpu_state_node);
+
+/**
  * __of_device_is_compatible() - Check if the node matches given constraints
  * @device: pointer to node
  * @compat: required compatible string, NULL or "" for any match
diff --git a/include/linux/of.h b/include/linux/of.h
index da1ee95..288233c 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -348,6 +348,8 @@ extern const void *of_get_property(const struct device_node *node,
 				const char *name,
 				int *lenp);
 extern struct device_node *of_get_cpu_node(int cpu, unsigned int *thread);
+extern struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+						 int index);
 #define for_each_property_of_node(dn, pp) \
 	for (pp = dn->properties; pp != NULL; pp = pp->next)
 
@@ -730,6 +732,12 @@ static inline struct device_node *of_get_cpu_node(int cpu,
 	return NULL;
 }
 
+static inline struct device_node *of_get_cpu_state_node(struct device_node *cpu_node,
+					int index)
+{
+	return NULL;
+}
+
 static inline int of_n_addr_cells(struct device_node *np)
 {
 	return 0;
-- 
2.7.4

[PATCH v6 12/25] cpuidle: dt: Support hierarchical CPU idle states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Currently CPU's idle states are represented in a flattened model, via the
"cpu-idle-states" binding from within the CPU's device nodes.

Support the hierarchical layout during parsing and validating of the CPU's
idle states. This is simply done by calling the new OF helper,
of_get_cpu_state_node().

Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/dt_idle_states.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/cpuidle/dt_idle_states.c b/drivers/cpuidle/dt_idle_states.c
index 53342b7..13f9b7c 100644
--- a/drivers/cpuidle/dt_idle_states.c
+++ b/drivers/cpuidle/dt_idle_states.c
@@ -118,8 +118,7 @@ static bool idle_state_valid(struct device_node *state_node, unsigned int idx,
 	for (cpu = cpumask_next(cpumask_first(cpumask), cpumask);
 	     cpu < nr_cpu_ids; cpu = cpumask_next(cpu, cpumask)) {
 		cpu_node = of_cpu_device_node_get(cpu);
-		curr_state_node = of_parse_phandle(cpu_node, "cpu-idle-states",
-						   idx);
+		curr_state_node = of_get_cpu_state_node(cpu_node, idx);
 		if (state_node != curr_state_node)
 			valid = false;
 
@@ -176,7 +175,7 @@ int dt_init_idle_driver(struct cpuidle_driver *drv,
 	cpu_node = of_cpu_device_node_get(cpumask_first(cpumask));
 
 	for (i = 0; ; i++) {
-		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		state_node = of_get_cpu_state_node(cpu_node, i);
 		if (!state_node)
 			break;
 
-- 
2.7.4

[PATCH v6 12/25] cpuidle: dt: Support hierarchical CPU idle states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Currently CPU's idle states are represented in a flattened model, via the
"cpu-idle-states" binding from within the CPU's device nodes.

Support the hierarchical layout during parsing and validating of the CPU's
idle states. This is simply done by calling the new OF helper,
of_get_cpu_state_node().

Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/dt_idle_states.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/cpuidle/dt_idle_states.c b/drivers/cpuidle/dt_idle_states.c
index 53342b7..13f9b7c 100644
--- a/drivers/cpuidle/dt_idle_states.c
+++ b/drivers/cpuidle/dt_idle_states.c
@@ -118,8 +118,7 @@ static bool idle_state_valid(struct device_node *state_node, unsigned int idx,
 	for (cpu = cpumask_next(cpumask_first(cpumask), cpumask);
 	     cpu < nr_cpu_ids; cpu = cpumask_next(cpu, cpumask)) {
 		cpu_node = of_cpu_device_node_get(cpu);
-		curr_state_node = of_parse_phandle(cpu_node, "cpu-idle-states",
-						   idx);
+		curr_state_node = of_get_cpu_state_node(cpu_node, idx);
 		if (state_node != curr_state_node)
 			valid = false;
 
@@ -176,7 +175,7 @@ int dt_init_idle_driver(struct cpuidle_driver *drv,
 	cpu_node = of_cpu_device_node_get(cpumask_first(cpumask));
 
 	for (i = 0; ; i++) {
-		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		state_node = of_get_cpu_state_node(cpu_node, i);
 		if (!state_node)
 			break;
 
-- 
2.7.4

[PATCH v6 13/25] drivers: firmware: psci: Split psci_dt_cpu_init_idle()

[Ulf Hansson]

Let's split psci_dt_cpu_init_idle() function into two functions, as to
allow following changes to re-use some of the code.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 42 +++++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 19 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index c80ec1d..9788bfc 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -270,9 +270,26 @@ static int __init psci_features(u32 psci_func_id)
 #ifdef CONFIG_CPU_IDLE
 static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
 
+static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
+{
+	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
+
+	if (err) {
+		pr_warn("%pOF missing arm,psci-suspend-param property\n", np);
+		return err;
+	}
+
+	if (!psci_power_state_is_valid(*state)) {
+		pr_warn("Invalid PSCI power state %#x\n", *state);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 {
-	int i, ret, count = 0;
+	int i, ret = 0, count = 0;
 	u32 *psci_states;
 	struct device_node *state_node;
 
@@ -291,29 +308,16 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 		return -ENOMEM;
 
 	for (i = 0; i < count; i++) {
-		u32 state;
-
 		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		ret = psci_dt_parse_state_node(state_node, &psci_states[i]);
+		of_node_put(state_node);
 
-		ret = of_property_read_u32(state_node,
-					   "arm,psci-suspend-param",
-					   &state);
-		if (ret) {
-			pr_warn(" * %pOF missing arm,psci-suspend-param property\n",
-				state_node);
-			of_node_put(state_node);
+		if (ret)
 			goto free_mem;
-		}
 
-		of_node_put(state_node);
-		pr_debug("psci-power-state %#x index %d\n", state, i);
-		if (!psci_power_state_is_valid(state)) {
-			pr_warn("Invalid PSCI power state %#x\n", state);
-			ret = -EINVAL;
-			goto free_mem;
-		}
-		psci_states[i] = state;
+		pr_debug("psci-power-state %#x index %d\n", psci_states[i], i);
 	}
+
 	/* Idle states parsed correctly, initialize per-cpu pointer */
 	per_cpu(psci_power_state, cpu) = psci_states;
 	return 0;
-- 
2.7.4

[PATCH v6 13/25] drivers: firmware: psci: Split psci_dt_cpu_init_idle()

[Ulf Hansson]

Let's split psci_dt_cpu_init_idle() function into two functions, as to
allow following changes to re-use some of the code.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 42 +++++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 19 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index c80ec1d..9788bfc 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -270,9 +270,26 @@ static int __init psci_features(u32 psci_func_id)
 #ifdef CONFIG_CPU_IDLE
 static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
 
+static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
+{
+	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
+
+	if (err) {
+		pr_warn("%pOF missing arm,psci-suspend-param property\n", np);
+		return err;
+	}
+
+	if (!psci_power_state_is_valid(*state)) {
+		pr_warn("Invalid PSCI power state %#x\n", *state);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 {
-	int i, ret, count = 0;
+	int i, ret = 0, count = 0;
 	u32 *psci_states;
 	struct device_node *state_node;
 
@@ -291,29 +308,16 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 		return -ENOMEM;
 
 	for (i = 0; i < count; i++) {
-		u32 state;
-
 		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		ret = psci_dt_parse_state_node(state_node, &psci_states[i]);
+		of_node_put(state_node);
 
-		ret = of_property_read_u32(state_node,
-					   "arm,psci-suspend-param",
-					   &state);
-		if (ret) {
-			pr_warn(" * %pOF missing arm,psci-suspend-param property\n",
-				state_node);
-			of_node_put(state_node);
+		if (ret)
 			goto free_mem;
-		}
 
-		of_node_put(state_node);
-		pr_debug("psci-power-state %#x index %d\n", state, i);
-		if (!psci_power_state_is_valid(state)) {
-			pr_warn("Invalid PSCI power state %#x\n", state);
-			ret = -EINVAL;
-			goto free_mem;
-		}
-		psci_states[i] = state;
+		pr_debug("psci-power-state %#x index %d\n", psci_states[i], i);
 	}
+
 	/* Idle states parsed correctly, initialize per-cpu pointer */
 	per_cpu(psci_power_state, cpu) = psci_states;
 	return 0;
-- 
2.7.4

[PATCH v6 14/25] drivers: firmware: psci: Support hierarchical CPU idle states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Currently CPU's idle states are represented in a flattened model, via the
"cpu-idle-states" binding from within the CPU's device nodes.

Support the hierarchical layout, simply by converting to calling the new OF
helper, of_get_cpu_state_node().

Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 9788bfc..256b4ed 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -294,8 +294,7 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 	struct device_node *state_node;
 
 	/* Count idle states */
-	while ((state_node = of_parse_phandle(cpu_node, "cpu-idle-states",
-					      count))) {
+	while ((state_node = of_get_cpu_state_node(cpu_node, count))) {
 		count++;
 		of_node_put(state_node);
 	}
@@ -308,7 +307,7 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 		return -ENOMEM;
 
 	for (i = 0; i < count; i++) {
-		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		state_node = of_get_cpu_state_node(cpu_node, i);
 		ret = psci_dt_parse_state_node(state_node, &psci_states[i]);
 		of_node_put(state_node);
 
-- 
2.7.4

[PATCH v6 14/25] drivers: firmware: psci: Support hierarchical CPU idle states

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

Currently CPU's idle states are represented in a flattened model, via the
"cpu-idle-states" binding from within the CPU's device nodes.

Support the hierarchical layout, simply by converting to calling the new OF
helper, of_get_cpu_state_node().

Cc: Lina Iyer <ilina@codeaurora.org>
Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 9788bfc..256b4ed 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -294,8 +294,7 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 	struct device_node *state_node;
 
 	/* Count idle states */
-	while ((state_node = of_parse_phandle(cpu_node, "cpu-idle-states",
-					      count))) {
+	while ((state_node = of_get_cpu_state_node(cpu_node, count))) {
 		count++;
 		of_node_put(state_node);
 	}
@@ -308,7 +307,7 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 		return -ENOMEM;
 
 	for (i = 0; i < count; i++) {
-		state_node = of_parse_phandle(cpu_node, "cpu-idle-states", i);
+		state_node = of_get_cpu_state_node(cpu_node, i);
 		ret = psci_dt_parse_state_node(state_node, &psci_states[i]);
 		of_node_put(state_node);
 
-- 
2.7.4

[PATCH v6 15/25] drivers: firmware: psci: Simplify error path of psci_dt_init()

[Ulf Hansson]

Instead of having each psci init function taking care of the of_node_put(),
let's deal with that from psci_dt_init(), as this enables a bit simpler
error path for each psci init function.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 256b4ed..3888100 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -608,9 +608,9 @@ static int __init psci_0_2_init(struct device_node *np)
 	int err;
 
 	err = get_set_conduit_method(np);
-
 	if (err)
-		goto out_put_node;
+		return err;
+
 	/*
 	 * Starting with v0.2, the PSCI specification introduced a call
 	 * (PSCI_VERSION) that allows probing the firmware version, so
@@ -618,11 +618,7 @@ static int __init psci_0_2_init(struct device_node *np)
 	 * can be carried out according to the specific version reported
 	 * by firmware
 	 */
-	err = psci_probe();
-
-out_put_node:
-	of_node_put(np);
-	return err;
+	return psci_probe();
 }
 
 /*
@@ -634,9 +630,8 @@ static int __init psci_0_1_init(struct device_node *np)
 	int err;
 
 	err = get_set_conduit_method(np);
-
 	if (err)
-		goto out_put_node;
+		return err;
 
 	pr_info("Using PSCI v0.1 Function IDs from DT\n");
 
@@ -660,9 +655,7 @@ static int __init psci_0_1_init(struct device_node *np)
 		psci_ops.migrate = psci_migrate;
 	}
 
-out_put_node:
-	of_node_put(np);
-	return err;
+	return 0;
 }
 
 static const struct of_device_id psci_of_match[] __initconst = {
@@ -677,6 +670,7 @@ int __init psci_dt_init(void)
 	struct device_node *np;
 	const struct of_device_id *matched_np;
 	psci_initcall_t init_fn;
+	int ret;
 
 	np = of_find_matching_node_and_match(NULL, psci_of_match, &matched_np);
 
@@ -684,7 +678,10 @@ int __init psci_dt_init(void)
 		return -ENODEV;
 
 	init_fn = (psci_initcall_t)matched_np->data;
-	return init_fn(np);
+	ret = init_fn(np);
+
+	of_node_put(np);
+	return ret;
 }
 
 #ifdef CONFIG_ACPI
-- 
2.7.4

[PATCH v6 15/25] drivers: firmware: psci: Simplify error path of psci_dt_init()

[Ulf Hansson]

Instead of having each psci init function taking care of the of_node_put(),
let's deal with that from psci_dt_init(), as this enables a bit simpler
error path for each psci init function.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 256b4ed..3888100 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -608,9 +608,9 @@ static int __init psci_0_2_init(struct device_node *np)
 	int err;
 
 	err = get_set_conduit_method(np);
-
 	if (err)
-		goto out_put_node;
+		return err;
+
 	/*
 	 * Starting with v0.2, the PSCI specification introduced a call
 	 * (PSCI_VERSION) that allows probing the firmware version, so
@@ -618,11 +618,7 @@ static int __init psci_0_2_init(struct device_node *np)
 	 * can be carried out according to the specific version reported
 	 * by firmware
 	 */
-	err = psci_probe();
-
-out_put_node:
-	of_node_put(np);
-	return err;
+	return psci_probe();
 }
 
 /*
@@ -634,9 +630,8 @@ static int __init psci_0_1_init(struct device_node *np)
 	int err;
 
 	err = get_set_conduit_method(np);
-
 	if (err)
-		goto out_put_node;
+		return err;
 
 	pr_info("Using PSCI v0.1 Function IDs from DT\n");
 
@@ -660,9 +655,7 @@ static int __init psci_0_1_init(struct device_node *np)
 		psci_ops.migrate = psci_migrate;
 	}
 
-out_put_node:
-	of_node_put(np);
-	return err;
+	return 0;
 }
 
 static const struct of_device_id psci_of_match[] __initconst = {
@@ -677,6 +670,7 @@ int __init psci_dt_init(void)
 	struct device_node *np;
 	const struct of_device_id *matched_np;
 	psci_initcall_t init_fn;
+	int ret;
 
 	np = of_find_matching_node_and_match(NULL, psci_of_match, &matched_np);
 
@@ -684,7 +678,10 @@ int __init psci_dt_init(void)
 		return -ENODEV;
 
 	init_fn = (psci_initcall_t)matched_np->data;
-	return init_fn(np);
+	ret = init_fn(np);
+
+	of_node_put(np);
+	return ret;
 }
 
 #ifdef CONFIG_ACPI
-- 
2.7.4

[PATCH v6 16/25] drivers: firmware: psci: Announce support for OS initiated suspend mode

[Ulf Hansson]

PSCI firmware v1.0+, supports two different modes for CPU_SUSPEND. The
Platform Coordinated mode, which is the default and mandatory mode, while
support for the OS initiated mode is optional.

This change introduces initial support for the OS initiated mode, in way
that it adds the related PSCI bits from the spec and prints a message in
the log to inform whether the mode is supported by the PSCI FW.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c   | 21 ++++++++++++++++++++-
 include/uapi/linux/psci.h |  5 +++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 3888100..e8f4f84 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -95,6 +95,11 @@ static inline bool psci_has_ext_power_state(void)
 				PSCI_1_0_FEATURES_CPU_SUSPEND_PF_MASK;
 }
 
+static inline bool psci_has_osi_support(void)
+{
+	return psci_cpu_suspend_feature & PSCI_1_0_OS_INITIATED;
+}
+
 static inline bool psci_power_state_loses_context(u32 state)
 {
 	const u32 mask = psci_has_ext_power_state() ?
@@ -658,10 +663,24 @@ static int __init psci_0_1_init(struct device_node *np)
 	return 0;
 }
 
+static int __init psci_1_0_init(struct device_node *np)
+{
+	int err;
+
+	err = psci_0_2_init(np);
+	if (err)
+		return err;
+
+	if (psci_has_osi_support())
+		pr_info("OSI mode supported.\n");
+
+	return 0;
+}
+
 static const struct of_device_id psci_of_match[] __initconst = {
 	{ .compatible = "arm,psci",	.data = psci_0_1_init},
 	{ .compatible = "arm,psci-0.2",	.data = psci_0_2_init},
-	{ .compatible = "arm,psci-1.0",	.data = psci_0_2_init},
+	{ .compatible = "arm,psci-1.0",	.data = psci_1_0_init},
 	{},
 };
 
diff --git a/include/uapi/linux/psci.h b/include/uapi/linux/psci.h
index b3bcabe..581f720 100644
--- a/include/uapi/linux/psci.h
+++ b/include/uapi/linux/psci.h
@@ -49,6 +49,7 @@
 
 #define PSCI_1_0_FN_PSCI_FEATURES		PSCI_0_2_FN(10)
 #define PSCI_1_0_FN_SYSTEM_SUSPEND		PSCI_0_2_FN(14)
+#define PSCI_1_0_FN_SET_SUSPEND_MODE		PSCI_0_2_FN(15)
 
 #define PSCI_1_0_FN64_SYSTEM_SUSPEND		PSCI_0_2_FN64(14)
 
@@ -97,6 +98,10 @@
 #define PSCI_1_0_FEATURES_CPU_SUSPEND_PF_MASK	\
 			(0x1 << PSCI_1_0_FEATURES_CPU_SUSPEND_PF_SHIFT)
 
+#define PSCI_1_0_OS_INITIATED			BIT(0)
+#define PSCI_1_0_SUSPEND_MODE_PC		0
+#define PSCI_1_0_SUSPEND_MODE_OSI		1
+
 /* PSCI return values (inclusive of all PSCI versions) */
 #define PSCI_RET_SUCCESS			0
 #define PSCI_RET_NOT_SUPPORTED			-1
-- 
2.7.4

[PATCH v6 16/25] drivers: firmware: psci: Announce support for OS initiated suspend mode

[Ulf Hansson]

PSCI firmware v1.0+, supports two different modes for CPU_SUSPEND. The
Platform Coordinated mode, which is the default and mandatory mode, while
support for the OS initiated mode is optional.

This change introduces initial support for the OS initiated mode, in way
that it adds the related PSCI bits from the spec and prints a message in
the log to inform whether the mode is supported by the PSCI FW.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c   | 21 ++++++++++++++++++++-
 include/uapi/linux/psci.h |  5 +++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 3888100..e8f4f84 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -95,6 +95,11 @@ static inline bool psci_has_ext_power_state(void)
 				PSCI_1_0_FEATURES_CPU_SUSPEND_PF_MASK;
 }
 
+static inline bool psci_has_osi_support(void)
+{
+	return psci_cpu_suspend_feature & PSCI_1_0_OS_INITIATED;
+}
+
 static inline bool psci_power_state_loses_context(u32 state)
 {
 	const u32 mask = psci_has_ext_power_state() ?
@@ -658,10 +663,24 @@ static int __init psci_0_1_init(struct device_node *np)
 	return 0;
 }
 
+static int __init psci_1_0_init(struct device_node *np)
+{
+	int err;
+
+	err = psci_0_2_init(np);
+	if (err)
+		return err;
+
+	if (psci_has_osi_support())
+		pr_info("OSI mode supported.\n");
+
+	return 0;
+}
+
 static const struct of_device_id psci_of_match[] __initconst = {
 	{ .compatible = "arm,psci",	.data = psci_0_1_init},
 	{ .compatible = "arm,psci-0.2",	.data = psci_0_2_init},
-	{ .compatible = "arm,psci-1.0",	.data = psci_0_2_init},
+	{ .compatible = "arm,psci-1.0",	.data = psci_1_0_init},
 	{},
 };
 
diff --git a/include/uapi/linux/psci.h b/include/uapi/linux/psci.h
index b3bcabe..581f720 100644
--- a/include/uapi/linux/psci.h
+++ b/include/uapi/linux/psci.h
@@ -49,6 +49,7 @@
 
 #define PSCI_1_0_FN_PSCI_FEATURES		PSCI_0_2_FN(10)
 #define PSCI_1_0_FN_SYSTEM_SUSPEND		PSCI_0_2_FN(14)
+#define PSCI_1_0_FN_SET_SUSPEND_MODE		PSCI_0_2_FN(15)
 
 #define PSCI_1_0_FN64_SYSTEM_SUSPEND		PSCI_0_2_FN64(14)
 
@@ -97,6 +98,10 @@
 #define PSCI_1_0_FEATURES_CPU_SUSPEND_PF_MASK	\
 			(0x1 << PSCI_1_0_FEATURES_CPU_SUSPEND_PF_SHIFT)
 
+#define PSCI_1_0_OS_INITIATED			BIT(0)
+#define PSCI_1_0_SUSPEND_MODE_PC		0
+#define PSCI_1_0_SUSPEND_MODE_OSI		1
+
 /* PSCI return values (inclusive of all PSCI versions) */
 #define PSCI_RET_SUCCESS			0
 #define PSCI_RET_NOT_SUPPORTED			-1
-- 
2.7.4

[PATCH v6 17/25] drivers: firmware: psci: Prepare to use OS initiated suspend mode

[Ulf Hansson]

To enable the OS initiated mode, the CPU topology needs to be described
using the hierarchical model in DT. When used, the idle state bits for the
CPU needs to be created by ORing the bits for CPU's selected idle state
with the bits for CPU's PM domain (CPU's cluster) idle state.

Let's prepare the PSCI driver to deal with this, via introducing a per CPU
variable called domain_state and by adding internal helpers to read/write
the value of variable.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index e8f4f84..40b2b89 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -87,8 +87,19 @@ static u32 psci_function_id[PSCI_FN_MAX];
 				(PSCI_1_0_EXT_POWER_STATE_ID_MASK | \
 				PSCI_1_0_EXT_POWER_STATE_TYPE_MASK)
 
+static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
 
+static inline u32 psci_get_domain_state(void)
+{
+	return this_cpu_read(domain_state);
+}
+
+static inline void psci_set_domain_state(u32 state)
+{
+	this_cpu_write(domain_state, state);
+}
+
 static inline bool psci_has_ext_power_state(void)
 {
 	return psci_cpu_suspend_feature &
@@ -187,6 +198,8 @@ static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)
 
 	fn = psci_function_id[PSCI_FN_CPU_ON];
 	err = invoke_psci_fn(fn, cpuid, entry_point, 0);
+	/* Clear the domain state to start fresh. */
+	psci_set_domain_state(0);
 	return psci_to_linux_errno(err);
 }
 
@@ -408,15 +421,17 @@ int psci_cpu_init_idle(unsigned int cpu)
 static int psci_suspend_finisher(unsigned long index)
 {
 	u32 *state = __this_cpu_read(psci_power_state);
+	u32 composite_state = state[index - 1] | psci_get_domain_state();
 
-	return psci_ops.cpu_suspend(state[index - 1],
-				    __pa_symbol(cpu_resume));
+	return psci_ops.cpu_suspend(composite_state, __pa_symbol(cpu_resume));
 }
 
 int psci_cpu_suspend_enter(unsigned long index)
 {
 	int ret;
 	u32 *state = __this_cpu_read(psci_power_state);
+	u32 composite_state = state[index - 1] | psci_get_domain_state();
+
 	/*
 	 * idle state index 0 corresponds to wfi, should never be called
 	 * from the cpu_suspend operations
@@ -424,11 +439,14 @@ int psci_cpu_suspend_enter(unsigned long index)
 	if (WARN_ON_ONCE(!index))
 		return -EINVAL;
 
-	if (!psci_power_state_loses_context(state[index - 1]))
-		ret = psci_ops.cpu_suspend(state[index - 1], 0);
+	if (!psci_power_state_loses_context(composite_state))
+		ret = psci_ops.cpu_suspend(composite_state, 0);
 	else
 		ret = cpu_suspend(index, psci_suspend_finisher);
 
+	/* Clear the domain state to start fresh when back from idle. */
+	psci_set_domain_state(0);
+
 	return ret;
 }
 
-- 
2.7.4

[PATCH v6 17/25] drivers: firmware: psci: Prepare to use OS initiated suspend mode

[Ulf Hansson]

To enable the OS initiated mode, the CPU topology needs to be described
using the hierarchical model in DT. When used, the idle state bits for the
CPU needs to be created by ORing the bits for CPU's selected idle state
with the bits for CPU's PM domain (CPU's cluster) idle state.

Let's prepare the PSCI driver to deal with this, via introducing a per CPU
variable called domain_state and by adding internal helpers to read/write
the value of variable.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index e8f4f84..40b2b89 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -87,8 +87,19 @@ static u32 psci_function_id[PSCI_FN_MAX];
 				(PSCI_1_0_EXT_POWER_STATE_ID_MASK | \
 				PSCI_1_0_EXT_POWER_STATE_TYPE_MASK)
 
+static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
 
+static inline u32 psci_get_domain_state(void)
+{
+	return this_cpu_read(domain_state);
+}
+
+static inline void psci_set_domain_state(u32 state)
+{
+	this_cpu_write(domain_state, state);
+}
+
 static inline bool psci_has_ext_power_state(void)
 {
 	return psci_cpu_suspend_feature &
@@ -187,6 +198,8 @@ static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)
 
 	fn = psci_function_id[PSCI_FN_CPU_ON];
 	err = invoke_psci_fn(fn, cpuid, entry_point, 0);
+	/* Clear the domain state to start fresh. */
+	psci_set_domain_state(0);
 	return psci_to_linux_errno(err);
 }
 
@@ -408,15 +421,17 @@ int psci_cpu_init_idle(unsigned int cpu)
 static int psci_suspend_finisher(unsigned long index)
 {
 	u32 *state = __this_cpu_read(psci_power_state);
+	u32 composite_state = state[index - 1] | psci_get_domain_state();
 
-	return psci_ops.cpu_suspend(state[index - 1],
-				    __pa_symbol(cpu_resume));
+	return psci_ops.cpu_suspend(composite_state, __pa_symbol(cpu_resume));
 }
 
 int psci_cpu_suspend_enter(unsigned long index)
 {
 	int ret;
 	u32 *state = __this_cpu_read(psci_power_state);
+	u32 composite_state = state[index - 1] | psci_get_domain_state();
+
 	/*
 	 * idle state index 0 corresponds to wfi, should never be called
 	 * from the cpu_suspend operations
@@ -424,11 +439,14 @@ int psci_cpu_suspend_enter(unsigned long index)
 	if (WARN_ON_ONCE(!index))
 		return -EINVAL;
 
-	if (!psci_power_state_loses_context(state[index - 1]))
-		ret = psci_ops.cpu_suspend(state[index - 1], 0);
+	if (!psci_power_state_loses_context(composite_state))
+		ret = psci_ops.cpu_suspend(composite_state, 0);
 	else
 		ret = cpu_suspend(index, psci_suspend_finisher);
 
+	/* Clear the domain state to start fresh when back from idle. */
+	psci_set_domain_state(0);
+
 	return ret;
 }
 
-- 
2.7.4

[PATCH v6 18/25] drivers: firmware: psci: Share a few internal PSCI functions

[Ulf Hansson]

Following changes needs to be able to call psci_get|set_domain_state() and
psci_dt_parse_state_node(), but from a separate file. Let's make that
possible by sharing them via a new internal PSCI header file.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 14 ++++++++------
 drivers/firmware/psci.h | 13 +++++++++++++
 2 files changed, 21 insertions(+), 6 deletions(-)
 create mode 100644 drivers/firmware/psci.h

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 40b2b89..463f78c 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -34,6 +34,8 @@
 #include <asm/smp_plat.h>
 #include <asm/suspend.h>
 
+#include "psci.h"
+
 /*
  * While a 64-bit OS can make calls with SMC32 calling conventions, for some
  * calls it is necessary to use SMC64 to pass or return 64-bit values.
@@ -90,12 +92,12 @@ static u32 psci_function_id[PSCI_FN_MAX];
 static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
 
-static inline u32 psci_get_domain_state(void)
+u32 psci_get_domain_state(void)
 {
 	return this_cpu_read(domain_state);
 }
 
-static inline void psci_set_domain_state(u32 state)
+void psci_set_domain_state(u32 state)
 {
 	this_cpu_write(domain_state, state);
 }
@@ -285,10 +287,7 @@ static int __init psci_features(u32 psci_func_id)
 			      psci_func_id, 0, 0);
 }
 
-#ifdef CONFIG_CPU_IDLE
-static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
-
-static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
+int psci_dt_parse_state_node(struct device_node *np, u32 *state)
 {
 	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
 
@@ -305,6 +304,9 @@ static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
 	return 0;
 }
 
+#ifdef CONFIG_CPU_IDLE
+static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
+
 static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 {
 	int i, ret = 0, count = 0;
diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
new file mode 100644
index 0000000..a2b4be5
--- /dev/null
+++ b/drivers/firmware/psci.h
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef __PSCI_H
+#define __PSCI_H
+
+struct device_node;
+
+u32 psci_get_domain_state(void);
+void psci_set_domain_state(u32 state);
+
+int psci_dt_parse_state_node(struct device_node *np, u32 *state);
+
+#endif /* __PSCI_H */
-- 
2.7.4

[PATCH v6 18/25] drivers: firmware: psci: Share a few internal PSCI functions

[Ulf Hansson]

Following changes needs to be able to call psci_get|set_domain_state() and
psci_dt_parse_state_node(), but from a separate file. Let's make that
possible by sharing them via a new internal PSCI header file.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 14 ++++++++------
 drivers/firmware/psci.h | 13 +++++++++++++
 2 files changed, 21 insertions(+), 6 deletions(-)
 create mode 100644 drivers/firmware/psci.h

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 40b2b89..463f78c 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -34,6 +34,8 @@
 #include <asm/smp_plat.h>
 #include <asm/suspend.h>
 
+#include "psci.h"
+
 /*
  * While a 64-bit OS can make calls with SMC32 calling conventions, for some
  * calls it is necessary to use SMC64 to pass or return 64-bit values.
@@ -90,12 +92,12 @@ static u32 psci_function_id[PSCI_FN_MAX];
 static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
 
-static inline u32 psci_get_domain_state(void)
+u32 psci_get_domain_state(void)
 {
 	return this_cpu_read(domain_state);
 }
 
-static inline void psci_set_domain_state(u32 state)
+void psci_set_domain_state(u32 state)
 {
 	this_cpu_write(domain_state, state);
 }
@@ -285,10 +287,7 @@ static int __init psci_features(u32 psci_func_id)
 			      psci_func_id, 0, 0);
 }
 
-#ifdef CONFIG_CPU_IDLE
-static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
-
-static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
+int psci_dt_parse_state_node(struct device_node *np, u32 *state)
 {
 	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
 
@@ -305,6 +304,9 @@ static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
 	return 0;
 }
 
+#ifdef CONFIG_CPU_IDLE
+static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
+
 static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 {
 	int i, ret = 0, count = 0;
diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
new file mode 100644
index 0000000..a2b4be5
--- /dev/null
+++ b/drivers/firmware/psci.h
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef __PSCI_H
+#define __PSCI_H
+
+struct device_node;
+
+u32 psci_get_domain_state(void);
+void psci_set_domain_state(u32 state);
+
+int psci_dt_parse_state_node(struct device_node *np, u32 *state);
+
+#endif /* __PSCI_H */
-- 
2.7.4

[PATCH v6 19/25] drivers: firmware: psci: Add support for PM domains using genpd

[Ulf Hansson]

When the hierarchical layout is used in DT, as to describe the PM topology
for the CPUs, which are managed by PSCI, we want to be able to initialize
and setup the corresponding PM domain data structures.

Let's make this possible via adding a new file, psci_pm_domains.c and
implement the needed interface towards the generic PM domain (aka genpd).
Share a helper function, psci_dt_init_pm_domains(), which the regular PSCI
firmware driver may call when it needs to initialize the PM topology using
genpd.

In principle, the implementation consists of allocating/initializing the
genpd data structures, parsing the domain-idle states DT bindings via
calling of_genpd_parse_idle_states() and to call pm_genpd_init() for the
allocated genpds.

Finally, one genpd OF provider is added per genpd. Via DT, this enables
devices, including CPU devices, to be attached to the created genpds.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/Makefile         |   2 +-
 drivers/firmware/psci.h           |   6 ++
 drivers/firmware/psci_pm_domain.c | 180 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 187 insertions(+), 1 deletion(-)
 create mode 100644 drivers/firmware/psci_pm_domain.c

diff --git a/drivers/firmware/Makefile b/drivers/firmware/Makefile
index b248238..877260e 100644
--- a/drivers/firmware/Makefile
+++ b/drivers/firmware/Makefile
@@ -2,7 +2,7 @@
 #
 # Makefile for the linux kernel.
 #
-obj-$(CONFIG_ARM_PSCI_FW)	+= psci.o
+obj-$(CONFIG_ARM_PSCI_FW)	+= psci.o psci_pm_domain.o
 obj-$(CONFIG_ARM_PSCI_CHECKER)	+= psci_checker.o
 obj-$(CONFIG_ARM_SCPI_PROTOCOL)	+= arm_scpi.o
 obj-$(CONFIG_ARM_SCPI_POWER_DOMAIN) += scpi_pm_domain.o
diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
index a2b4be5..8b6fe51 100644
--- a/drivers/firmware/psci.h
+++ b/drivers/firmware/psci.h
@@ -10,4 +10,10 @@ void psci_set_domain_state(u32 state);
 
 int psci_dt_parse_state_node(struct device_node *np, u32 *state);
 
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+int psci_dt_init_pm_domains(struct device_node *np);
+#else
+static inline int psci_dt_init_pm_domains(struct device_node *np) { return 0; }
+#endif
+
 #endif /* __PSCI_H */
diff --git a/drivers/firmware/psci_pm_domain.c b/drivers/firmware/psci_pm_domain.c
new file mode 100644
index 0000000..f54819e
--- /dev/null
+++ b/drivers/firmware/psci_pm_domain.c
@@ -0,0 +1,180 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PM domains for CPUs via genpd - managed by PSCI.
+ *
+ * Copyright (C) 2018 Linaro Ltd.
+ * Author: Ulf Hansson <ulf.hansson@linaro.org>
+ *
+ */
+
+#define pr_fmt(fmt) "psci: " fmt
+
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/pm_domain.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+
+#include "psci.h"
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+static int psci_pd_power_off(struct generic_pm_domain *pd)
+{
+	struct genpd_power_state *state = &pd->states[pd->state_idx];
+	u32 *pd_state;
+	u32 composite_pd_state;
+
+	if (!state->data)
+		return 0;
+
+	pd_state = state->data;
+	composite_pd_state = *pd_state | psci_get_domain_state();
+	psci_set_domain_state(composite_pd_state);
+
+	return 0;
+}
+
+static int psci_dt_parse_pd_states(struct genpd_power_state *states,
+				   int state_count)
+{
+	int i, err;
+	u32 *psci_states;
+
+	if (!state_count)
+		return 0;
+
+	psci_states = kcalloc(state_count, sizeof(psci_states), GFP_KERNEL);
+	if (!psci_states)
+		return -ENOMEM;
+
+	for (i = 0; i < state_count; i++) {
+		err = psci_dt_parse_state_node(to_of_node(states[i].fwnode),
+					       &psci_states[i]);
+		if (err) {
+			kfree(psci_states);
+			return err;
+		}
+	}
+
+	for (i = 0; i < state_count; i++)
+		states[i].data = &psci_states[i];
+
+	return 0;
+}
+
+static int psci_dt_init_genpd(struct device_node *np,
+			      struct genpd_power_state *states,
+			      unsigned int state_count)
+{
+	struct generic_pm_domain *pd;
+	struct dev_power_governor *pd_gov;
+	int ret = -ENOMEM;
+
+	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+	if (!pd)
+		return -ENOMEM;
+
+	pd->name = kasprintf(GFP_KERNEL, "%pOF", np);
+	if (!pd->name)
+		goto free_pd;
+
+	pd->name = kbasename(pd->name);
+	pd->power_off = psci_pd_power_off;
+	pd->states = states;
+	pd->state_count = state_count;
+	pd->flags |= GENPD_FLAG_IRQ_SAFE | GENPD_FLAG_CPU_DOMAIN;
+
+	/* Use governor for CPU PM domains if it has some states to manage. */
+	pd_gov = state_count > 0 ? &pm_domain_cpu_gov : NULL;
+
+	ret = pm_genpd_init(pd, pd_gov, false);
+	if (ret)
+		goto free_name;
+
+	ret = of_genpd_add_provider_simple(np, pd);
+	if (ret)
+		goto remove_pd;
+
+	pr_info("init PM domain %s\n", pd->name);
+	return 0;
+
+remove_pd:
+	pm_genpd_remove(pd);
+free_name:
+	kfree(pd->name);
+free_pd:
+	kfree(pd);
+	pr_err("failed to init PM domain ret=%d %pOF\n", ret, np);
+	return ret;
+}
+
+static int psci_dt_set_genpd_topology(struct device_node *np)
+{
+	struct device_node *node;
+	struct of_phandle_args child, parent;
+	int ret;
+
+	for_each_child_of_node(np, node) {
+		if (of_parse_phandle_with_args(node, "power-domains",
+					       "#power-domain-cells", 0,
+					       &parent))
+			continue;
+
+		child.np = node;
+		child.args_count = 0;
+
+		ret = of_genpd_add_subdomain(&parent, &child);
+		of_node_put(parent.np);
+		if (ret) {
+			of_node_put(node);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int psci_dt_init_pm_domains(struct device_node *np)
+{
+	struct device_node *node;
+	struct genpd_power_state *states;
+	int state_count;
+	int pd_count = 0;
+	int ret;
+
+	/* Parse child nodes for "#power-domain-cells". */
+	for_each_child_of_node(np, node) {
+		if (!of_find_property(node, "#power-domain-cells", NULL))
+			continue;
+
+		ret = of_genpd_parse_idle_states(node, &states, &state_count);
+		if (ret)
+			goto err_put;
+
+		ret = psci_dt_parse_pd_states(states, state_count);
+		if (ret)
+			goto err_put;
+
+		ret = psci_dt_init_genpd(node, states, state_count);
+		if (ret)
+			goto err_put;
+
+		pd_count++;
+	}
+
+	if (!pd_count)
+		return 0;
+
+	ret = psci_dt_set_genpd_topology(np);
+	if (ret)
+		goto err_msg;
+
+	return pd_count;
+
+err_put:
+	of_node_put(node);
+err_msg:
+	pr_err("failed to create PM domains ret=%d\n", ret);
+	return ret;
+}
+#endif
-- 
2.7.4

[PATCH v6 19/25] drivers: firmware: psci: Add support for PM domains using genpd

[Ulf Hansson]

When the hierarchical layout is used in DT, as to describe the PM topology
for the CPUs, which are managed by PSCI, we want to be able to initialize
and setup the corresponding PM domain data structures.

Let's make this possible via adding a new file, psci_pm_domains.c and
implement the needed interface towards the generic PM domain (aka genpd).
Share a helper function, psci_dt_init_pm_domains(), which the regular PSCI
firmware driver may call when it needs to initialize the PM topology using
genpd.

In principle, the implementation consists of allocating/initializing the
genpd data structures, parsing the domain-idle states DT bindings via
calling of_genpd_parse_idle_states() and to call pm_genpd_init() for the
allocated genpds.

Finally, one genpd OF provider is added per genpd. Via DT, this enables
devices, including CPU devices, to be attached to the created genpds.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/Makefile         |   2 +-
 drivers/firmware/psci.h           |   6 ++
 drivers/firmware/psci_pm_domain.c | 180 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 187 insertions(+), 1 deletion(-)
 create mode 100644 drivers/firmware/psci_pm_domain.c

diff --git a/drivers/firmware/Makefile b/drivers/firmware/Makefile
index b248238..877260e 100644
--- a/drivers/firmware/Makefile
+++ b/drivers/firmware/Makefile
@@ -2,7 +2,7 @@
 #
 # Makefile for the linux kernel.
 #
-obj-$(CONFIG_ARM_PSCI_FW)	+= psci.o
+obj-$(CONFIG_ARM_PSCI_FW)	+= psci.o psci_pm_domain.o
 obj-$(CONFIG_ARM_PSCI_CHECKER)	+= psci_checker.o
 obj-$(CONFIG_ARM_SCPI_PROTOCOL)	+= arm_scpi.o
 obj-$(CONFIG_ARM_SCPI_POWER_DOMAIN) += scpi_pm_domain.o
diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
index a2b4be5..8b6fe51 100644
--- a/drivers/firmware/psci.h
+++ b/drivers/firmware/psci.h
@@ -10,4 +10,10 @@ void psci_set_domain_state(u32 state);
 
 int psci_dt_parse_state_node(struct device_node *np, u32 *state);
 
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+int psci_dt_init_pm_domains(struct device_node *np);
+#else
+static inline int psci_dt_init_pm_domains(struct device_node *np) { return 0; }
+#endif
+
 #endif /* __PSCI_H */
diff --git a/drivers/firmware/psci_pm_domain.c b/drivers/firmware/psci_pm_domain.c
new file mode 100644
index 0000000..f54819e
--- /dev/null
+++ b/drivers/firmware/psci_pm_domain.c
@@ -0,0 +1,180 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PM domains for CPUs via genpd - managed by PSCI.
+ *
+ * Copyright (C) 2018 Linaro Ltd.
+ * Author: Ulf Hansson <ulf.hansson@linaro.org>
+ *
+ */
+
+#define pr_fmt(fmt) "psci: " fmt
+
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/pm_domain.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+
+#include "psci.h"
+
+#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+static int psci_pd_power_off(struct generic_pm_domain *pd)
+{
+	struct genpd_power_state *state = &pd->states[pd->state_idx];
+	u32 *pd_state;
+	u32 composite_pd_state;
+
+	if (!state->data)
+		return 0;
+
+	pd_state = state->data;
+	composite_pd_state = *pd_state | psci_get_domain_state();
+	psci_set_domain_state(composite_pd_state);
+
+	return 0;
+}
+
+static int psci_dt_parse_pd_states(struct genpd_power_state *states,
+				   int state_count)
+{
+	int i, err;
+	u32 *psci_states;
+
+	if (!state_count)
+		return 0;
+
+	psci_states = kcalloc(state_count, sizeof(psci_states), GFP_KERNEL);
+	if (!psci_states)
+		return -ENOMEM;
+
+	for (i = 0; i < state_count; i++) {
+		err = psci_dt_parse_state_node(to_of_node(states[i].fwnode),
+					       &psci_states[i]);
+		if (err) {
+			kfree(psci_states);
+			return err;
+		}
+	}
+
+	for (i = 0; i < state_count; i++)
+		states[i].data = &psci_states[i];
+
+	return 0;
+}
+
+static int psci_dt_init_genpd(struct device_node *np,
+			      struct genpd_power_state *states,
+			      unsigned int state_count)
+{
+	struct generic_pm_domain *pd;
+	struct dev_power_governor *pd_gov;
+	int ret = -ENOMEM;
+
+	pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+	if (!pd)
+		return -ENOMEM;
+
+	pd->name = kasprintf(GFP_KERNEL, "%pOF", np);
+	if (!pd->name)
+		goto free_pd;
+
+	pd->name = kbasename(pd->name);
+	pd->power_off = psci_pd_power_off;
+	pd->states = states;
+	pd->state_count = state_count;
+	pd->flags |= GENPD_FLAG_IRQ_SAFE | GENPD_FLAG_CPU_DOMAIN;
+
+	/* Use governor for CPU PM domains if it has some states to manage. */
+	pd_gov = state_count > 0 ? &pm_domain_cpu_gov : NULL;
+
+	ret = pm_genpd_init(pd, pd_gov, false);
+	if (ret)
+		goto free_name;
+
+	ret = of_genpd_add_provider_simple(np, pd);
+	if (ret)
+		goto remove_pd;
+
+	pr_info("init PM domain %s\n", pd->name);
+	return 0;
+
+remove_pd:
+	pm_genpd_remove(pd);
+free_name:
+	kfree(pd->name);
+free_pd:
+	kfree(pd);
+	pr_err("failed to init PM domain ret=%d %pOF\n", ret, np);
+	return ret;
+}
+
+static int psci_dt_set_genpd_topology(struct device_node *np)
+{
+	struct device_node *node;
+	struct of_phandle_args child, parent;
+	int ret;
+
+	for_each_child_of_node(np, node) {
+		if (of_parse_phandle_with_args(node, "power-domains",
+					       "#power-domain-cells", 0,
+					       &parent))
+			continue;
+
+		child.np = node;
+		child.args_count = 0;
+
+		ret = of_genpd_add_subdomain(&parent, &child);
+		of_node_put(parent.np);
+		if (ret) {
+			of_node_put(node);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int psci_dt_init_pm_domains(struct device_node *np)
+{
+	struct device_node *node;
+	struct genpd_power_state *states;
+	int state_count;
+	int pd_count = 0;
+	int ret;
+
+	/* Parse child nodes for "#power-domain-cells". */
+	for_each_child_of_node(np, node) {
+		if (!of_find_property(node, "#power-domain-cells", NULL))
+			continue;
+
+		ret = of_genpd_parse_idle_states(node, &states, &state_count);
+		if (ret)
+			goto err_put;
+
+		ret = psci_dt_parse_pd_states(states, state_count);
+		if (ret)
+			goto err_put;
+
+		ret = psci_dt_init_genpd(node, states, state_count);
+		if (ret)
+			goto err_put;
+
+		pd_count++;
+	}
+
+	if (!pd_count)
+		return 0;
+
+	ret = psci_dt_set_genpd_topology(np);
+	if (ret)
+		goto err_msg;
+
+	return pd_count;
+
+err_put:
+	of_node_put(node);
+err_msg:
+	pr_err("failed to create PM domains ret=%d\n", ret);
+	return ret;
+}
+#endif
-- 
2.7.4

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

In case the hierarchical layout is used in DT, we want to initialize the
corresponding PM domain topology for the CPUs, by using the generic PM
domain (aka genpd) infrastructure.

At first glance, it may seem feasible to hook into the existing
psci_dt_init() function, although because it's called quite early in the
boot sequence, allocating the dynamic data structure for a genpd doesn't
work.

Therefore, let's export a new init function for PSCI,
psci_dt_topology_init(), which the ARM machine code should call from a
suitable initcall.

Succeeding to initialize the PM domain topology, which means at least one
instance of a genpd becomes created, allows us to continue to enable the
PSCI OS initiated mode for the platform. If everything turns out fine,
let's print a message in log to inform the user about the changed mode.

In case of any failures, we stick to the default PSCI Platform Coordinated
mode.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
 include/linux/psci.h    |  2 ++
 2 files changed, 33 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 463f78c..45d55fc 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -723,6 +723,37 @@ int __init psci_dt_init(void)
 	return ret;
 }
 
+int __init psci_dt_topology_init(void)
+{
+	struct device_node *np;
+	int ret;
+
+	if (!psci_has_osi_support())
+		return 0;
+
+	np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
+	if (!np)
+		return -ENODEV;
+
+	/* Initialize the CPU PM domains based on topology described in DT. */
+	ret = psci_dt_init_pm_domains(np);
+	if (ret <= 0)
+		goto out;
+
+	/* Enable OSI mode. */
+	ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
+			     PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
+	if (ret) {
+		pr_info("failed to enable OSI mode: %d\n", ret);
+		goto out;
+	}
+
+	pr_info("OSI mode enabled.\n");
+out:
+	of_node_put(np);
+	return ret;
+}
+
 #ifdef CONFIG_ACPI
 /*
  * We use PSCI 0.2+ when ACPI is deployed on ARM64 and it's
diff --git a/include/linux/psci.h b/include/linux/psci.h
index 8b1b3b5..298a044 100644
--- a/include/linux/psci.h
+++ b/include/linux/psci.h
@@ -53,8 +53,10 @@ extern struct psci_operations psci_ops;
 
 #if defined(CONFIG_ARM_PSCI_FW)
 int __init psci_dt_init(void);
+int __init psci_dt_topology_init(void);
 #else
 static inline int psci_dt_init(void) { return 0; }
+static inline int psci_dt_topology_init(void) { return 0; }
 #endif
 
 #if defined(CONFIG_ARM_PSCI_FW) && defined(CONFIG_ACPI)
-- 
2.7.4

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

In case the hierarchical layout is used in DT, we want to initialize the
corresponding PM domain topology for the CPUs, by using the generic PM
domain (aka genpd) infrastructure.

At first glance, it may seem feasible to hook into the existing
psci_dt_init() function, although because it's called quite early in the
boot sequence, allocating the dynamic data structure for a genpd doesn't
work.

Therefore, let's export a new init function for PSCI,
psci_dt_topology_init(), which the ARM machine code should call from a
suitable initcall.

Succeeding to initialize the PM domain topology, which means at least one
instance of a genpd becomes created, allows us to continue to enable the
PSCI OS initiated mode for the platform. If everything turns out fine,
let's print a message in log to inform the user about the changed mode.

In case of any failures, we stick to the default PSCI Platform Coordinated
mode.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
 include/linux/psci.h    |  2 ++
 2 files changed, 33 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 463f78c..45d55fc 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -723,6 +723,37 @@ int __init psci_dt_init(void)
 	return ret;
 }
 
+int __init psci_dt_topology_init(void)
+{
+	struct device_node *np;
+	int ret;
+
+	if (!psci_has_osi_support())
+		return 0;
+
+	np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
+	if (!np)
+		return -ENODEV;
+
+	/* Initialize the CPU PM domains based on topology described in DT. */
+	ret = psci_dt_init_pm_domains(np);
+	if (ret <= 0)
+		goto out;
+
+	/* Enable OSI mode. */
+	ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
+			     PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
+	if (ret) {
+		pr_info("failed to enable OSI mode: %d\n", ret);
+		goto out;
+	}
+
+	pr_info("OSI mode enabled.\n");
+out:
+	of_node_put(np);
+	return ret;
+}
+
 #ifdef CONFIG_ACPI
 /*
  * We use PSCI 0.2+ when ACPI is deployed on ARM64 and it's
diff --git a/include/linux/psci.h b/include/linux/psci.h
index 8b1b3b5..298a044 100644
--- a/include/linux/psci.h
+++ b/include/linux/psci.h
@@ -53,8 +53,10 @@ extern struct psci_operations psci_ops;
 
 #if defined(CONFIG_ARM_PSCI_FW)
 int __init psci_dt_init(void);
+int __init psci_dt_topology_init(void);
 #else
 static inline int psci_dt_init(void) { return 0; }
+static inline int psci_dt_topology_init(void) { return 0; }
 #endif
 
 #if defined(CONFIG_ARM_PSCI_FW) && defined(CONFIG_ACPI)
-- 
2.7.4

[PATCH v6 21/25] drivers: firmware: psci: Try to attach CPU devices to their PM domains

[Ulf Hansson]

In case the OS initiated CPU suspend mode have been enabled, the PM domain
topology for CPUs have earlier been created by PSCI. Let's use this
information in psci_dt_cpu_init_idle() as a condition for when it makes
sense to try to attach the CPU to its corresponding PM domain, via calling
of_genpd_attach_cpu().

If the CPU is attached successfully to its PM domain, idle management is
now fully prepared to be controlled through runtime PM for the CPU.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 45d55fc..944d6f6 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -20,6 +20,7 @@
 #include <linux/linkage.h>
 #include <linux/of.h>
 #include <linux/pm.h>
+#include <linux/pm_domain.h>
 #include <linux/printk.h>
 #include <linux/psci.h>
 #include <linux/reboot.h>
@@ -91,6 +92,7 @@ static u32 psci_function_id[PSCI_FN_MAX];
 
 static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
+static bool psci_osi_mode_enabled;
 
 u32 psci_get_domain_state(void)
 {
@@ -339,6 +341,14 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 
 	/* Idle states parsed correctly, initialize per-cpu pointer */
 	per_cpu(psci_power_state, cpu) = psci_states;
+
+	/* If running OSI mode, attach the CPU device to its PM domain. */
+	if (psci_osi_mode_enabled) {
+		ret = of_genpd_attach_cpu(cpu);
+		if (ret)
+			goto free_mem;
+	}
+
 	return 0;
 
 free_mem:
@@ -748,6 +758,7 @@ int __init psci_dt_topology_init(void)
 		goto out;
 	}
 
+	psci_osi_mode_enabled = true;
 	pr_info("OSI mode enabled.\n");
 out:
 	of_node_put(np);
-- 
2.7.4

[PATCH v6 21/25] drivers: firmware: psci: Try to attach CPU devices to their PM domains

[Ulf Hansson]

In case the OS initiated CPU suspend mode have been enabled, the PM domain
topology for CPUs have earlier been created by PSCI. Let's use this
information in psci_dt_cpu_init_idle() as a condition for when it makes
sense to try to attach the CPU to its corresponding PM domain, via calling
of_genpd_attach_cpu().

If the CPU is attached successfully to its PM domain, idle management is
now fully prepared to be controlled through runtime PM for the CPU.

Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 45d55fc..944d6f6 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -20,6 +20,7 @@
 #include <linux/linkage.h>
 #include <linux/of.h>
 #include <linux/pm.h>
+#include <linux/pm_domain.h>
 #include <linux/printk.h>
 #include <linux/psci.h>
 #include <linux/reboot.h>
@@ -91,6 +92,7 @@ static u32 psci_function_id[PSCI_FN_MAX];
 
 static DEFINE_PER_CPU(u32, domain_state);
 static u32 psci_cpu_suspend_feature;
+static bool psci_osi_mode_enabled;
 
 u32 psci_get_domain_state(void)
 {
@@ -339,6 +341,14 @@ static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
 
 	/* Idle states parsed correctly, initialize per-cpu pointer */
 	per_cpu(psci_power_state, cpu) = psci_states;
+
+	/* If running OSI mode, attach the CPU device to its PM domain. */
+	if (psci_osi_mode_enabled) {
+		ret = of_genpd_attach_cpu(cpu);
+		if (ret)
+			goto free_mem;
+	}
+
 	return 0;
 
 free_mem:
@@ -748,6 +758,7 @@ int __init psci_dt_topology_init(void)
 		goto out;
 	}
 
+	psci_osi_mode_enabled = true;
 	pr_info("OSI mode enabled.\n");
 out:
 	of_node_put(np);
-- 
2.7.4

[PATCH v6 22/25] drivers: firmware: psci: Deal with CPU hotplug when using OSI mode

[Ulf Hansson]

To deal with CPU hotplug when OSI mode is used, the CPU device needs to be
detached from its PM domain (genpd) when putting it offline, otherwise the
CPU becomes considered as being in use from genpd and runtime PM point of
view. Obviously, then we also need to re-attach the CPU device when bring
the CPU back online, so let's do this.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 944d6f6..06f3916 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -190,6 +190,10 @@ static int psci_cpu_off(u32 state)
 	int err;
 	u32 fn;
 
+	/* If running OSI mode, detach the CPU device from its PM domain. */
+	if (psci_osi_mode_enabled)
+		of_genpd_detach_cpu(smp_processor_id());
+
 	fn = psci_function_id[PSCI_FN_CPU_OFF];
 	err = invoke_psci_fn(fn, state, 0, 0);
 	return psci_to_linux_errno(err);
@@ -204,6 +208,10 @@ static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)
 	err = invoke_psci_fn(fn, cpuid, entry_point, 0);
 	/* Clear the domain state to start fresh. */
 	psci_set_domain_state(0);
+
+	if (!err && psci_osi_mode_enabled)
+		of_genpd_attach_cpu(cpuid);
+
 	return psci_to_linux_errno(err);
 }
 
-- 
2.7.4

[PATCH v6 22/25] drivers: firmware: psci: Deal with CPU hotplug when using OSI mode

[Ulf Hansson]

To deal with CPU hotplug when OSI mode is used, the CPU device needs to be
detached from its PM domain (genpd) when putting it offline, otherwise the
CPU becomes considered as being in use from genpd and runtime PM point of
view. Obviously, then we also need to re-attach the CPU device when bring
the CPU back online, so let's do this.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/firmware/psci.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
index 944d6f6..06f3916 100644
--- a/drivers/firmware/psci.c
+++ b/drivers/firmware/psci.c
@@ -190,6 +190,10 @@ static int psci_cpu_off(u32 state)
 	int err;
 	u32 fn;
 
+	/* If running OSI mode, detach the CPU device from its PM domain. */
+	if (psci_osi_mode_enabled)
+		of_genpd_detach_cpu(smp_processor_id());
+
 	fn = psci_function_id[PSCI_FN_CPU_OFF];
 	err = invoke_psci_fn(fn, state, 0, 0);
 	return psci_to_linux_errno(err);
@@ -204,6 +208,10 @@ static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)
 	err = invoke_psci_fn(fn, cpuid, entry_point, 0);
 	/* Clear the domain state to start fresh. */
 	psci_set_domain_state(0);
+
+	if (!err && psci_osi_mode_enabled)
+		of_genpd_attach_cpu(cpuid);
+
 	return psci_to_linux_errno(err);
 }
 
-- 
2.7.4

[PATCH v6 23/25] MAINTAINERS: Update files for PSCI

[Ulf Hansson]

Some new files have been for the PSCI firmware driver, let's make the
expression more generic to cover them all.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 MAINTAINERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 4623caf..dda16cf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11082,7 +11082,7 @@ M:	Mark Rutland <mark.rutland@arm.com>
 M:	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
 L:	linux-arm-kernel@lists.infradead.org
 S:	Maintained
-F:	drivers/firmware/psci*.c
+F:	drivers/firmware/psci*
 F:	include/linux/psci.h
 F:	include/uapi/linux/psci.h
 
-- 
2.7.4

[PATCH v6 23/25] MAINTAINERS: Update files for PSCI

[Ulf Hansson]

Some new files have been for the PSCI firmware driver, let's make the
expression more generic to cover them all.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 MAINTAINERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 4623caf..dda16cf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11082,7 +11082,7 @@ M:	Mark Rutland <mark.rutland@arm.com>
 M:	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
 L:	linux-arm-kernel at lists.infradead.org
 S:	Maintained
-F:	drivers/firmware/psci*.c
+F:	drivers/firmware/psci*
 F:	include/linux/psci.h
 F:	include/uapi/linux/psci.h
 
-- 
2.7.4

[PATCH v6 24/25] arm64: kernel: Respect the hierarchical CPU topology in DT for PSCI

[Ulf Hansson]

To let the PSCI driver to parse the CPU topology in DT, as to create CPU PM
domains in case the hierarchical layout is used, let's call
psci_dt_topology_init() from the existing topology_init() subsys_initcall.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 arch/arm64/kernel/setup.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 30ad2f0..574a504 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -331,6 +331,9 @@ static int __init topology_init(void)
 {
 	int i;
 
+	if (acpi_disabled)
+		psci_dt_topology_init();
+
 	for_each_online_node(i)
 		register_one_node(i);
 
-- 
2.7.4

[PATCH v6 24/25] arm64: kernel: Respect the hierarchical CPU topology in DT for PSCI

[Ulf Hansson]

To let the PSCI driver to parse the CPU topology in DT, as to create CPU PM
domains in case the hierarchical layout is used, let's call
psci_dt_topology_init() from the existing topology_init() subsys_initcall.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Lina Iyer <ilina@codeaurora.org>
Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 arch/arm64/kernel/setup.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 30ad2f0..574a504 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -331,6 +331,9 @@ static int __init topology_init(void)
 {
 	int i;
 
+	if (acpi_disabled)
+		psci_dt_topology_init();
+
 	for_each_online_node(i)
 		register_one_node(i);
 
-- 
2.7.4

[PATCH v6 25/25] arm64: dts: Convert to the hierarchical CPU topology layout for MSM8916

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

In the hierarchical layout, we are creating power domains around each CPU
and describes the idle states for them inside the power domain provider
node.  Note that, the CPU's idle states still needs to be compatible with
"arm,idle-state".

Furthermore, represent the CPU cluster as a separate master power domain,
powering the CPU's power domains. The cluster node, contains the idle
states for the cluster and each idle state needs to be compatible with the
"domain-idle-state".

If the running platform is using a PSCI FW that supports the OS initiated
CPU suspend mode, which likely should be the case unless the PSCI FW is
very old, this change makes the PSCI driver to enable it.

Cc: Andy Gross <andy.gross@linaro.org>
Cc: David Brown <david.brown@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 arch/arm64/boot/dts/qcom/msm8916.dtsi | 53 ++++++++++++++++++++++++++++++++---
 1 file changed, 49 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi
index e51b049..d0bb0b9 100644
--- a/arch/arm64/boot/dts/qcom/msm8916.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi
@@ -112,7 +112,7 @@
 			reg = <0x0>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD0>;
 		};
 
 		CPU1: cpu@1 {
@@ -121,7 +121,7 @@
 			reg = <0x1>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD1>;
 		};
 
 		CPU2: cpu@2 {
@@ -130,7 +130,7 @@
 			reg = <0x2>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD2>;
 		};
 
 		CPU3: cpu@3 {
@@ -139,7 +139,7 @@
 			reg = <0x3>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD3>;
 		};
 
 		L2_0: l2-cache {
@@ -156,12 +156,57 @@
 				min-residency-us = <2000>;
 				local-timer-stop;
 			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-gdhs {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
 		};
 	};
 
 	psci {
 		compatible = "arm,psci-1.0";
 		method = "smc";
+
+		CPU_PD0: cpu-pd0 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD1: cpu-pd1 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD2: cpu-pd2 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD3: cpu-pd3 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CLUSTER_PD: cluster-pd {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CLUSTER_RET>, <&CLUSTER_PWRDN>;
+		};
 	};
 
 	pmu {
-- 
2.7.4

[PATCH v6 25/25] arm64: dts: Convert to the hierarchical CPU topology layout for MSM8916

[Ulf Hansson]

From: Lina Iyer <lina.iyer@linaro.org>

In the hierarchical layout, we are creating power domains around each CPU
and describes the idle states for them inside the power domain provider
node.  Note that, the CPU's idle states still needs to be compatible with
"arm,idle-state".

Furthermore, represent the CPU cluster as a separate master power domain,
powering the CPU's power domains. The cluster node, contains the idle
states for the cluster and each idle state needs to be compatible with the
"domain-idle-state".

If the running platform is using a PSCI FW that supports the OS initiated
CPU suspend mode, which likely should be the case unless the PSCI FW is
very old, this change makes the PSCI driver to enable it.

Cc: Andy Gross <andy.gross@linaro.org>
Cc: David Brown <david.brown@linaro.org>
Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
Co-developed-by: Ulf Hansson <ulf.hansson@linaro.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 arch/arm64/boot/dts/qcom/msm8916.dtsi | 53 ++++++++++++++++++++++++++++++++---
 1 file changed, 49 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi
index e51b049..d0bb0b9 100644
--- a/arch/arm64/boot/dts/qcom/msm8916.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi
@@ -112,7 +112,7 @@
 			reg = <0x0>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD0>;
 		};
 
 		CPU1: cpu at 1 {
@@ -121,7 +121,7 @@
 			reg = <0x1>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD1>;
 		};
 
 		CPU2: cpu at 2 {
@@ -130,7 +130,7 @@
 			reg = <0x2>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD2>;
 		};
 
 		CPU3: cpu at 3 {
@@ -139,7 +139,7 @@
 			reg = <0x3>;
 			next-level-cache = <&L2_0>;
 			enable-method = "psci";
-			cpu-idle-states = <&CPU_SPC>;
+			power-domains = <&CPU_PD3>;
 		};
 
 		L2_0: l2-cache {
@@ -156,12 +156,57 @@
 				min-residency-us = <2000>;
 				local-timer-stop;
 			};
+
+			CLUSTER_RET: cluster-retention {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000010>;
+				entry-latency-us = <500>;
+				exit-latency-us = <500>;
+				min-residency-us = <2000>;
+			};
+
+			CLUSTER_PWRDN: cluster-gdhs {
+				compatible = "domain-idle-state";
+				arm,psci-suspend-param = <0x1000030>;
+				entry-latency-us = <2000>;
+				exit-latency-us = <2000>;
+				min-residency-us = <6000>;
+			};
 		};
 	};
 
 	psci {
 		compatible = "arm,psci-1.0";
 		method = "smc";
+
+		CPU_PD0: cpu-pd0 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD1: cpu-pd1 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD2: cpu-pd2 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CPU_PD3: cpu-pd3 {
+			#power-domain-cells = <0>;
+			power-domains = <&CLUSTER_PD>;
+			domain-idle-states = <&CPU_SPC>;
+		};
+
+		CLUSTER_PD: cluster-pd {
+			#power-domain-cells = <0>;
+			domain-idle-states = <&CLUSTER_RET>, <&CLUSTER_PWRDN>;
+		};
 	};
 
 	pmu {
-- 
2.7.4

Re: [PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Mark Rutland]

Hi Ulf,

On Wed, Mar 14, 2018 at 05:58:10PM +0100, Ulf Hansson wrote:
> For ARM, the PSCI firmware interface may be managing the power to the CPUs.
> Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
> could add another level of complexity from a CPU idle management point of view.
> 
> PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
> enables a more fine grained method, allowing Linux to get more control, in
> regards to being energy efficient. This is typically useful for these kind of
> complex battery driven platforms.

> The series is based on v4.16-rc5, but applies also as of today to Rafael's
> linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
> and I except that test coverage should be increased by some more boards
> shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).

Do we have any numbers to show how this affects energy efficiency?

Are there any other platforms with OSI support?

Thanks,
Mark.

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Mark Rutland]

Hi Ulf,

On Wed, Mar 14, 2018 at 05:58:10PM +0100, Ulf Hansson wrote:
> For ARM, the PSCI firmware interface may be managing the power to the CPUs.
> Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
> could add another level of complexity from a CPU idle management point of view.
> 
> PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
> enables a more fine grained method, allowing Linux to get more control, in
> regards to being energy efficient. This is typically useful for these kind of
> complex battery driven platforms.

> The series is based on v4.16-rc5, but applies also as of today to Rafael's
> linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
> and I except that test coverage should be increased by some more boards
> shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).

Do we have any numbers to show how this affects energy efficiency?

Are there any other platforms with OSI support?

Thanks,
Mark.

Re: [PATCH v6 06/25] timer: Export next wakeup time of a CPU

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:16PM +0100, Ulf Hansson wrote:
> @@ -132,6 +133,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
>  {
>  	return NSEC_PER_SEC / HZ;
>  }
> +
> +static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
> +{
> +	ktime_t len = NSEC_PER_SEC/HZ;
> +
> +	/* Next wake up is the tick period, assume it starts now */
> +	return ktime_add(len, ktime_get());
> +}

Thsi could be marginally simpler as:

static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
{
	/* Next wake up is the tick period, assume it starts now */
	return ktime_add(tick_nohz_get_sleep_length(), ktime_get());
}

Thanks,
Mark.

[PATCH v6 06/25] timer: Export next wakeup time of a CPU

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:16PM +0100, Ulf Hansson wrote:
> @@ -132,6 +133,15 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
>  {
>  	return NSEC_PER_SEC / HZ;
>  }
> +
> +static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
> +{
> +	ktime_t len = NSEC_PER_SEC/HZ;
> +
> +	/* Next wake up is the tick period, assume it starts now */
> +	return ktime_add(len, ktime_get());
> +}

Thsi could be marginally simpler as:

static inline ktime_t tick_nohz_get_next_wakeup(int cpu)
{
	/* Next wake up is the tick period, assume it starts now */
	return ktime_add(tick_nohz_get_sleep_length(), ktime_get());
}

Thanks,
Mark.

Re: [PATCH v6 15/25] drivers: firmware: psci: Simplify error path of psci_dt_init()

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:25PM +0100, Ulf Hansson wrote:
> Instead of having each psci init function taking care of the of_node_put(),
> let's deal with that from psci_dt_init(), as this enables a bit simpler
> error path for each psci init function.
> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

Nice cleanup!

Acked-by: Mark Rutland <mark.rutland@arm.com>

Mark.

> ---
>  drivers/firmware/psci.c | 23 ++++++++++-------------
>  1 file changed, 10 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 256b4ed..3888100 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -608,9 +608,9 @@ static int __init psci_0_2_init(struct device_node *np)
>  	int err;
>  
>  	err = get_set_conduit_method(np);
> -
>  	if (err)
> -		goto out_put_node;
> +		return err;
> +
>  	/*
>  	 * Starting with v0.2, the PSCI specification introduced a call
>  	 * (PSCI_VERSION) that allows probing the firmware version, so
> @@ -618,11 +618,7 @@ static int __init psci_0_2_init(struct device_node *np)
>  	 * can be carried out according to the specific version reported
>  	 * by firmware
>  	 */
> -	err = psci_probe();
> -
> -out_put_node:
> -	of_node_put(np);
> -	return err;
> +	return psci_probe();
>  }
>  
>  /*
> @@ -634,9 +630,8 @@ static int __init psci_0_1_init(struct device_node *np)
>  	int err;
>  
>  	err = get_set_conduit_method(np);
> -
>  	if (err)
> -		goto out_put_node;
> +		return err;
>  
>  	pr_info("Using PSCI v0.1 Function IDs from DT\n");
>  
> @@ -660,9 +655,7 @@ static int __init psci_0_1_init(struct device_node *np)
>  		psci_ops.migrate = psci_migrate;
>  	}
>  
> -out_put_node:
> -	of_node_put(np);
> -	return err;
> +	return 0;
>  }
>  
>  static const struct of_device_id psci_of_match[] __initconst = {
> @@ -677,6 +670,7 @@ int __init psci_dt_init(void)
>  	struct device_node *np;
>  	const struct of_device_id *matched_np;
>  	psci_initcall_t init_fn;
> +	int ret;
>  
>  	np = of_find_matching_node_and_match(NULL, psci_of_match, &matched_np);
>  
> @@ -684,7 +678,10 @@ int __init psci_dt_init(void)
>  		return -ENODEV;
>  
>  	init_fn = (psci_initcall_t)matched_np->data;
> -	return init_fn(np);
> +	ret = init_fn(np);
> +
> +	of_node_put(np);
> +	return ret;
>  }
>  
>  #ifdef CONFIG_ACPI
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v6 15/25] drivers: firmware: psci: Simplify error path of psci_dt_init()

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:25PM +0100, Ulf Hansson wrote:
> Instead of having each psci init function taking care of the of_node_put(),
> let's deal with that from psci_dt_init(), as this enables a bit simpler
> error path for each psci init function.
> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

Nice cleanup!

Acked-by: Mark Rutland <mark.rutland@arm.com>

Mark.

> ---
>  drivers/firmware/psci.c | 23 ++++++++++-------------
>  1 file changed, 10 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 256b4ed..3888100 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -608,9 +608,9 @@ static int __init psci_0_2_init(struct device_node *np)
>  	int err;
>  
>  	err = get_set_conduit_method(np);
> -
>  	if (err)
> -		goto out_put_node;
> +		return err;
> +
>  	/*
>  	 * Starting with v0.2, the PSCI specification introduced a call
>  	 * (PSCI_VERSION) that allows probing the firmware version, so
> @@ -618,11 +618,7 @@ static int __init psci_0_2_init(struct device_node *np)
>  	 * can be carried out according to the specific version reported
>  	 * by firmware
>  	 */
> -	err = psci_probe();
> -
> -out_put_node:
> -	of_node_put(np);
> -	return err;
> +	return psci_probe();
>  }
>  
>  /*
> @@ -634,9 +630,8 @@ static int __init psci_0_1_init(struct device_node *np)
>  	int err;
>  
>  	err = get_set_conduit_method(np);
> -
>  	if (err)
> -		goto out_put_node;
> +		return err;
>  
>  	pr_info("Using PSCI v0.1 Function IDs from DT\n");
>  
> @@ -660,9 +655,7 @@ static int __init psci_0_1_init(struct device_node *np)
>  		psci_ops.migrate = psci_migrate;
>  	}
>  
> -out_put_node:
> -	of_node_put(np);
> -	return err;
> +	return 0;
>  }
>  
>  static const struct of_device_id psci_of_match[] __initconst = {
> @@ -677,6 +670,7 @@ int __init psci_dt_init(void)
>  	struct device_node *np;
>  	const struct of_device_id *matched_np;
>  	psci_initcall_t init_fn;
> +	int ret;
>  
>  	np = of_find_matching_node_and_match(NULL, psci_of_match, &matched_np);
>  
> @@ -684,7 +678,10 @@ int __init psci_dt_init(void)
>  		return -ENODEV;
>  
>  	init_fn = (psci_initcall_t)matched_np->data;
> -	return init_fn(np);
> +	ret = init_fn(np);
> +
> +	of_node_put(np);
> +	return ret;
>  }
>  
>  #ifdef CONFIG_ACPI
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v6 18/25] drivers: firmware: psci: Share a few internal PSCI functions

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:28PM +0100, Ulf Hansson wrote:
> Following changes needs to be able to call psci_get|set_domain_state() and
> psci_dt_parse_state_node(), but from a separate file. Let's make that
> possible by sharing them via a new internal PSCI header file.
> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/firmware/psci.c | 14 ++++++++------
>  drivers/firmware/psci.h | 13 +++++++++++++
>  2 files changed, 21 insertions(+), 6 deletions(-)
>  create mode 100644 drivers/firmware/psci.h

To mimimize the clutter in drivers/firmware/, it might be better to
move this into a drivers/firmware/psci/ directory, which would also
simplify the MAINTAINERS modification.

Thanks,
Mark.

> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 40b2b89..463f78c 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -34,6 +34,8 @@
>  #include <asm/smp_plat.h>
>  #include <asm/suspend.h>
>  
> +#include "psci.h"
> +
>  /*
>   * While a 64-bit OS can make calls with SMC32 calling conventions, for some
>   * calls it is necessary to use SMC64 to pass or return 64-bit values.
> @@ -90,12 +92,12 @@ static u32 psci_function_id[PSCI_FN_MAX];
>  static DEFINE_PER_CPU(u32, domain_state);
>  static u32 psci_cpu_suspend_feature;
>  
> -static inline u32 psci_get_domain_state(void)
> +u32 psci_get_domain_state(void)
>  {
>  	return this_cpu_read(domain_state);
>  }
>  
> -static inline void psci_set_domain_state(u32 state)
> +void psci_set_domain_state(u32 state)
>  {
>  	this_cpu_write(domain_state, state);
>  }
> @@ -285,10 +287,7 @@ static int __init psci_features(u32 psci_func_id)
>  			      psci_func_id, 0, 0);
>  }
>  
> -#ifdef CONFIG_CPU_IDLE
> -static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
> -
> -static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
> +int psci_dt_parse_state_node(struct device_node *np, u32 *state)
>  {
>  	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
>  
> @@ -305,6 +304,9 @@ static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
>  	return 0;
>  }
>  
> +#ifdef CONFIG_CPU_IDLE
> +static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
> +
>  static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
>  {
>  	int i, ret = 0, count = 0;
> diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
> new file mode 100644
> index 0000000..a2b4be5
> --- /dev/null
> +++ b/drivers/firmware/psci.h
> @@ -0,0 +1,13 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#ifndef __PSCI_H
> +#define __PSCI_H
> +
> +struct device_node;
> +
> +u32 psci_get_domain_state(void);
> +void psci_set_domain_state(u32 state);
> +
> +int psci_dt_parse_state_node(struct device_node *np, u32 *state);
> +
> +#endif /* __PSCI_H */
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v6 18/25] drivers: firmware: psci: Share a few internal PSCI functions

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:28PM +0100, Ulf Hansson wrote:
> Following changes needs to be able to call psci_get|set_domain_state() and
> psci_dt_parse_state_node(), but from a separate file. Let's make that
> possible by sharing them via a new internal PSCI header file.
> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/firmware/psci.c | 14 ++++++++------
>  drivers/firmware/psci.h | 13 +++++++++++++
>  2 files changed, 21 insertions(+), 6 deletions(-)
>  create mode 100644 drivers/firmware/psci.h

To mimimize the clutter in drivers/firmware/, it might be better to
move this into a drivers/firmware/psci/ directory, which would also
simplify the MAINTAINERS modification.

Thanks,
Mark.

> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 40b2b89..463f78c 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -34,6 +34,8 @@
>  #include <asm/smp_plat.h>
>  #include <asm/suspend.h>
>  
> +#include "psci.h"
> +
>  /*
>   * While a 64-bit OS can make calls with SMC32 calling conventions, for some
>   * calls it is necessary to use SMC64 to pass or return 64-bit values.
> @@ -90,12 +92,12 @@ static u32 psci_function_id[PSCI_FN_MAX];
>  static DEFINE_PER_CPU(u32, domain_state);
>  static u32 psci_cpu_suspend_feature;
>  
> -static inline u32 psci_get_domain_state(void)
> +u32 psci_get_domain_state(void)
>  {
>  	return this_cpu_read(domain_state);
>  }
>  
> -static inline void psci_set_domain_state(u32 state)
> +void psci_set_domain_state(u32 state)
>  {
>  	this_cpu_write(domain_state, state);
>  }
> @@ -285,10 +287,7 @@ static int __init psci_features(u32 psci_func_id)
>  			      psci_func_id, 0, 0);
>  }
>  
> -#ifdef CONFIG_CPU_IDLE
> -static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
> -
> -static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
> +int psci_dt_parse_state_node(struct device_node *np, u32 *state)
>  {
>  	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
>  
> @@ -305,6 +304,9 @@ static int psci_dt_parse_state_node(struct device_node *np, u32 *state)
>  	return 0;
>  }
>  
> +#ifdef CONFIG_CPU_IDLE
> +static DEFINE_PER_CPU_READ_MOSTLY(u32 *, psci_power_state);
> +
>  static int psci_dt_cpu_init_idle(struct device_node *cpu_node, int cpu)
>  {
>  	int i, ret = 0, count = 0;
> diff --git a/drivers/firmware/psci.h b/drivers/firmware/psci.h
> new file mode 100644
> index 0000000..a2b4be5
> --- /dev/null
> +++ b/drivers/firmware/psci.h
> @@ -0,0 +1,13 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#ifndef __PSCI_H
> +#define __PSCI_H
> +
> +struct device_node;
> +
> +u32 psci_get_domain_state(void);
> +void psci_set_domain_state(u32 state);
> +
> +int psci_dt_parse_state_node(struct device_node *np, u32 *state);
> +
> +#endif /* __PSCI_H */
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> In case the hierarchical layout is used in DT, we want to initialize the
> corresponding PM domain topology for the CPUs, by using the generic PM
> domain (aka genpd) infrastructure.
> 
> At first glance, it may seem feasible to hook into the existing
> psci_dt_init() function, although because it's called quite early in the
> boot sequence, allocating the dynamic data structure for a genpd doesn't
> work.
> 
> Therefore, let's export a new init function for PSCI,
> psci_dt_topology_init(), which the ARM machine code should call from a
> suitable initcall.
> 
> Succeeding to initialize the PM domain topology, which means at least one
> instance of a genpd becomes created, allows us to continue to enable the
> PSCI OS initiated mode for the platform. If everything turns out fine,
> let's print a message in log to inform the user about the changed mode.
> 
> In case of any failures, we stick to the default PSCI Platform Coordinated
> mode.

For kexec/kdump we'll need to explicitly set the suspend mode to
platform coordinated if for whatever reason we choose not to use OSI.

> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
>  include/linux/psci.h    |  2 ++
>  2 files changed, 33 insertions(+)
> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 463f78c..45d55fc 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
>  	return ret;
>  }
>  
> +int __init psci_dt_topology_init(void)
> +{
> +	struct device_node *np;
> +	int ret;
> +
> +	if (!psci_has_osi_support())
> +		return 0;
> +
> +	np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
> +	if (!np)
> +		return -ENODEV;
> +
> +	/* Initialize the CPU PM domains based on topology described in DT. */
> +	ret = psci_dt_init_pm_domains(np);
> +	if (ret <= 0)
> +		goto out;
> +
> +	/* Enable OSI mode. */
> +	ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
> +			     PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);

IIUC, this patch will need to be moved after the subsequent two patches
to ensure that this series bisects cleanly.

Thanks,
Mark.

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> In case the hierarchical layout is used in DT, we want to initialize the
> corresponding PM domain topology for the CPUs, by using the generic PM
> domain (aka genpd) infrastructure.
> 
> At first glance, it may seem feasible to hook into the existing
> psci_dt_init() function, although because it's called quite early in the
> boot sequence, allocating the dynamic data structure for a genpd doesn't
> work.
> 
> Therefore, let's export a new init function for PSCI,
> psci_dt_topology_init(), which the ARM machine code should call from a
> suitable initcall.
> 
> Succeeding to initialize the PM domain topology, which means at least one
> instance of a genpd becomes created, allows us to continue to enable the
> PSCI OS initiated mode for the platform. If everything turns out fine,
> let's print a message in log to inform the user about the changed mode.
> 
> In case of any failures, we stick to the default PSCI Platform Coordinated
> mode.

For kexec/kdump we'll need to explicitly set the suspend mode to
platform coordinated if for whatever reason we choose not to use OSI.

> 
> Cc: Lina Iyer <ilina@codeaurora.org>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
>  include/linux/psci.h    |  2 ++
>  2 files changed, 33 insertions(+)
> 
> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> index 463f78c..45d55fc 100644
> --- a/drivers/firmware/psci.c
> +++ b/drivers/firmware/psci.c
> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
>  	return ret;
>  }
>  
> +int __init psci_dt_topology_init(void)
> +{
> +	struct device_node *np;
> +	int ret;
> +
> +	if (!psci_has_osi_support())
> +		return 0;
> +
> +	np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
> +	if (!np)
> +		return -ENODEV;
> +
> +	/* Initialize the CPU PM domains based on topology described in DT. */
> +	ret = psci_dt_init_pm_domains(np);
> +	if (ret <= 0)
> +		goto out;
> +
> +	/* Enable OSI mode. */
> +	ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
> +			     PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);

IIUC, this patch will need to be moved after the subsequent two patches
to ensure that this series bisects cleanly.

Thanks,
Mark.

Re: [PATCH v6 23/25] MAINTAINERS: Update files for PSCI

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:33PM +0100, Ulf Hansson wrote:
> Some new files have been for the PSCI firmware driver, let's make the
> expression more generic to cover them all.
> 
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  MAINTAINERS | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4623caf..dda16cf 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11082,7 +11082,7 @@ M:	Mark Rutland <mark.rutland@arm.com>
>  M:	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>  L:	linux-arm-kernel@lists.infradead.org
>  S:	Maintained
> -F:	drivers/firmware/psci*.c
> +F:	drivers/firmware/psci*
>  F:	include/linux/psci.h
>  F:	include/uapi/linux/psci.h

As mentioned in my another reply, I suspect it's better to give
ourselves a drivers/firmware/psci/ directory earlier in the series
(updating MAINTAINERS at the same time).

Otherwise, this looks fine to me.

Thanks,
Mark.

[PATCH v6 23/25] MAINTAINERS: Update files for PSCI

[Mark Rutland]

On Wed, Mar 14, 2018 at 05:58:33PM +0100, Ulf Hansson wrote:
> Some new files have been for the PSCI firmware driver, let's make the
> expression more generic to cover them all.
> 
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  MAINTAINERS | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4623caf..dda16cf 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11082,7 +11082,7 @@ M:	Mark Rutland <mark.rutland@arm.com>
>  M:	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>  L:	linux-arm-kernel at lists.infradead.org
>  S:	Maintained
> -F:	drivers/firmware/psci*.c
> +F:	drivers/firmware/psci*
>  F:	include/linux/psci.h
>  F:	include/uapi/linux/psci.h

As mentioned in my another reply, I suspect it's better to give
ourselves a drivers/firmware/psci/ directory earlier in the series
(updating MAINTAINERS at the same time).

Otherwise, this looks fine to me.

Thanks,
Mark.

Re: [PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Geert Uytterhoeven]

Hi Ulf,

On Wed, Mar 14, 2018 at 5:58 PM, Ulf Hansson <ulf.hansson@linaro.org> wrote:
> This series is a re-worked version from Lina Iyer's two series [1] that got
> posted more than a year ago by now. I have picked up the series and done a
> significant re-work of it and here's the result. All patches have been changed,
> some have been dropped, some are entirely new. For this reason I decided to not
> include a version history, as I think people need a fresh start anyway.

Thanks for your series!

I gave it a try on a few Renesas boards and SoCs, without adding any new DT
descriptions.  On all of them it triggers

    BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057

during system suspend (s2ram).

On R-Car Gen2 and SH-Mobile AG5 (arm32, no PSCI):

Disabling non-boot CPUs ...
BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057
in_atomic(): 0, irqs_disabled(): 128, pid: 1725, name: s2ram
CPU: 0 PID: 1725 Comm: s2ram Not tainted
4.16.0-rc5-koelsch-00475-gaa69fc46cc44c3d7 #4001
Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[<c020f6a0>] (unwind_backtrace) from [<c020b324>] (show_stack+0x10/0x14)
[<c020b324>] (show_stack) from [<c07693c8>] (dump_stack+0x7c/0x9c)
[<c07693c8>] (dump_stack) from [<c02422ec>] (___might_sleep+0x128/0x164)
[<c02422ec>] (___might_sleep) from [<c0508efc>] (__pm_runtime_suspend+0x70/0xa8)
[<c0508efc>] (__pm_runtime_suspend) from [<c0299144>] (cpu_pm_enter+0x78/0x9c)
[<c0299144>] (cpu_pm_enter) from [<c0299170>] (cpu_pm_suspend+0x8/0x18)
[<c0299170>] (cpu_pm_suspend) from [<c04ff8c4>] (syscore_suspend+0x88/0x138)
[<c04ff8c4>] (syscore_suspend) from [<c0263010>]
(suspend_devices_and_enter+0x21c/0x564)
[<c0263010>] (suspend_devices_and_enter) from [<c02635a8>]
(pm_suspend+0x250/0x2c8)
[<c02635a8>] (pm_suspend) from [<c0262054>] (state_store+0xac/0xcc)
[<c0262054>] (state_store) from [<c035f238>] (kernfs_fop_write+0x170/0x1b0)
[<c035f238>] (kernfs_fop_write) from [<c02f8a5c>] (__vfs_write+0x2c/0x140)
[<c02f8a5c>] (__vfs_write) from [<c02f8ce4>] (vfs_write+0xb8/0x144)
[<c02f8ce4>] (vfs_write) from [<c02f8ea4>] (SyS_write+0x54/0xac)
[<c02f8ea4>] (SyS_write) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
Exception stack(0xeabcbfa8 to 0xeabcbff0)
bfa0:                   00000004 000ce408 00000001 000ce408 00000004 00000000
bfc0: 00000004 000ce408 b6e80b50 00000004 00000004 00000000 000c5758 00000000
bfe0: 00000000 be866754 b6de3c85 b6e1ef26


On R-Car Gen3 (arm64, PSCI):

Disabling non-boot CPUs ...
CPU1: shutdown
psci: CPU1 killed.
CPU2: shutdown
psci: CPU2 killed.
CPU3: shutdown
psci: CPU3 killed.
CPU4: shutdown
psci: CPU4 killed.
CPU5: shutdown
psci: CPU5 killed.
CPU6: shutdown
psci: CPU6 killed.
CPU7: shutdown
psci: CPU7 killed.
BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057
in_atomic(): 0, irqs_disabled(): 128, pid: 2592, name: s2ram
4 locks held by s2ram/2592:
 #0:  (sb_writers#7){.+.+}, at: [<00000000cae1f0e5>] vfs_write+0xb0/0x164
 #1:  (&of->mutex){+.+.}, at: [<000000003002e527>] kernfs_fop_write+0x114/0x1bc
 #2:  (kn->count#71){.+.+}, at: [<000000008c0217e1>]
kernfs_fop_write+0x11c/0x1bc
 #3:  (pm_mutex){+.+.}, at: [<000000009a6c23e2>] pm_suspend+0x194/0xb10
irq event stamp: 69308
hardirqs last  enabled at (69307): [<00000000e9d09767>]
_raw_spin_unlock_irq+0x2c/0x4c
hardirqs last disabled at (69308): [<00000000619169c4>]
arch_suspend_disable_irqs+0x10/0x18
softirqs last  enabled at (69246): [<00000000b8a7706e>]
hrtimers_dead_cpu+0x2b8/0x2f0
softirqs last disabled at (69242): [<000000004dee0c40>]
hrtimers_dead_cpu+0x48/0x2f0
CPU: 0 PID: 2592 Comm: s2ram Not tainted
4.16.0-rc5-salvator-x-00470-g319cfb3643965f46 #1685
Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT)
Call trace:
 dump_backtrace+0x0/0x140
 show_stack+0x14/0x1c
 dump_stack+0xb4/0xf0
 ___might_sleep+0x1fc/0x218
 __might_sleep+0x70/0x80
 __pm_runtime_suspend+0x6c/0xac
 cpu_pm_enter+0x74/0x9c
 cpu_pm_suspend+0xc/0x1c
 syscore_suspend+0x1b8/0x410
 suspend_devices_and_enter+0x210/0xd9c
 pm_suspend+0x9a4/0xb10
 state_store+0xd4/0xf8
 kobj_attr_store+0x18/0x28
 sysfs_kf_write+0x50/0x5c
 kernfs_fop_write+0x178/0x1bc
 __vfs_write+0x38/0x140
 vfs_write+0xc4/0x164
 SyS_write+0x54/0xa4
 el0_svc_naked+0x30/0x34


I've bisected this to "[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM
in the idle path for CPUs".

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Geert Uytterhoeven]

Hi Ulf,

On Wed, Mar 14, 2018 at 5:58 PM, Ulf Hansson <ulf.hansson@linaro.org> wrote:
> This series is a re-worked version from Lina Iyer's two series [1] that got
> posted more than a year ago by now. I have picked up the series and done a
> significant re-work of it and here's the result. All patches have been changed,
> some have been dropped, some are entirely new. For this reason I decided to not
> include a version history, as I think people need a fresh start anyway.

Thanks for your series!

I gave it a try on a few Renesas boards and SoCs, without adding any new DT
descriptions.  On all of them it triggers

    BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057

during system suspend (s2ram).

On R-Car Gen2 and SH-Mobile AG5 (arm32, no PSCI):

Disabling non-boot CPUs ...
BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057
in_atomic(): 0, irqs_disabled(): 128, pid: 1725, name: s2ram
CPU: 0 PID: 1725 Comm: s2ram Not tainted
4.16.0-rc5-koelsch-00475-gaa69fc46cc44c3d7 #4001
Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[<c020f6a0>] (unwind_backtrace) from [<c020b324>] (show_stack+0x10/0x14)
[<c020b324>] (show_stack) from [<c07693c8>] (dump_stack+0x7c/0x9c)
[<c07693c8>] (dump_stack) from [<c02422ec>] (___might_sleep+0x128/0x164)
[<c02422ec>] (___might_sleep) from [<c0508efc>] (__pm_runtime_suspend+0x70/0xa8)
[<c0508efc>] (__pm_runtime_suspend) from [<c0299144>] (cpu_pm_enter+0x78/0x9c)
[<c0299144>] (cpu_pm_enter) from [<c0299170>] (cpu_pm_suspend+0x8/0x18)
[<c0299170>] (cpu_pm_suspend) from [<c04ff8c4>] (syscore_suspend+0x88/0x138)
[<c04ff8c4>] (syscore_suspend) from [<c0263010>]
(suspend_devices_and_enter+0x21c/0x564)
[<c0263010>] (suspend_devices_and_enter) from [<c02635a8>]
(pm_suspend+0x250/0x2c8)
[<c02635a8>] (pm_suspend) from [<c0262054>] (state_store+0xac/0xcc)
[<c0262054>] (state_store) from [<c035f238>] (kernfs_fop_write+0x170/0x1b0)
[<c035f238>] (kernfs_fop_write) from [<c02f8a5c>] (__vfs_write+0x2c/0x140)
[<c02f8a5c>] (__vfs_write) from [<c02f8ce4>] (vfs_write+0xb8/0x144)
[<c02f8ce4>] (vfs_write) from [<c02f8ea4>] (SyS_write+0x54/0xac)
[<c02f8ea4>] (SyS_write) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
Exception stack(0xeabcbfa8 to 0xeabcbff0)
bfa0:                   00000004 000ce408 00000001 000ce408 00000004 00000000
bfc0: 00000004 000ce408 b6e80b50 00000004 00000004 00000000 000c5758 00000000
bfe0: 00000000 be866754 b6de3c85 b6e1ef26


On R-Car Gen3 (arm64, PSCI):

Disabling non-boot CPUs ...
CPU1: shutdown
psci: CPU1 killed.
CPU2: shutdown
psci: CPU2 killed.
CPU3: shutdown
psci: CPU3 killed.
CPU4: shutdown
psci: CPU4 killed.
CPU5: shutdown
psci: CPU5 killed.
CPU6: shutdown
psci: CPU6 killed.
CPU7: shutdown
psci: CPU7 killed.
BUG: sleeping function called from invalid context at
drivers/base/power/runtime.c:1057
in_atomic(): 0, irqs_disabled(): 128, pid: 2592, name: s2ram
4 locks held by s2ram/2592:
 #0:  (sb_writers#7){.+.+}, at: [<00000000cae1f0e5>] vfs_write+0xb0/0x164
 #1:  (&of->mutex){+.+.}, at: [<000000003002e527>] kernfs_fop_write+0x114/0x1bc
 #2:  (kn->count#71){.+.+}, at: [<000000008c0217e1>]
kernfs_fop_write+0x11c/0x1bc
 #3:  (pm_mutex){+.+.}, at: [<000000009a6c23e2>] pm_suspend+0x194/0xb10
irq event stamp: 69308
hardirqs last  enabled at (69307): [<00000000e9d09767>]
_raw_spin_unlock_irq+0x2c/0x4c
hardirqs last disabled at (69308): [<00000000619169c4>]
arch_suspend_disable_irqs+0x10/0x18
softirqs last  enabled at (69246): [<00000000b8a7706e>]
hrtimers_dead_cpu+0x2b8/0x2f0
softirqs last disabled at (69242): [<000000004dee0c40>]
hrtimers_dead_cpu+0x48/0x2f0
CPU: 0 PID: 2592 Comm: s2ram Not tainted
4.16.0-rc5-salvator-x-00470-g319cfb3643965f46 #1685
Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT)
Call trace:
 dump_backtrace+0x0/0x140
 show_stack+0x14/0x1c
 dump_stack+0xb4/0xf0
 ___might_sleep+0x1fc/0x218
 __might_sleep+0x70/0x80
 __pm_runtime_suspend+0x6c/0xac
 cpu_pm_enter+0x74/0x9c
 cpu_pm_suspend+0xc/0x1c
 syscore_suspend+0x1b8/0x410
 suspend_devices_and_enter+0x210/0xd9c
 pm_suspend+0x9a4/0xb10
 state_store+0xd4/0xf8
 kobj_attr_store+0x18/0x28
 sysfs_kf_write+0x50/0x5c
 kernfs_fop_write+0x178/0x1bc
 __vfs_write+0x38/0x140
 vfs_write+0xc4/0x164
 SyS_write+0x54/0xa4
 el0_svc_naked+0x30/0x34


I've bisected this to "[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM
in the idle path for CPUs".

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

On 15 March 2018 at 12:00, Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> Hi Ulf,
>
> On Wed, Mar 14, 2018 at 5:58 PM, Ulf Hansson <ulf.hansson@linaro.org> wrote:
>> This series is a re-worked version from Lina Iyer's two series [1] that got
>> posted more than a year ago by now. I have picked up the series and done a
>> significant re-work of it and here's the result. All patches have been changed,
>> some have been dropped, some are entirely new. For this reason I decided to not
>> include a version history, as I think people need a fresh start anyway.
>
> Thanks for your series!
>
> I gave it a try on a few Renesas boards and SoCs, without adding any new DT
> descriptions.  On all of them it triggers

Thanks a lot for testing!

>
>     BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
>
> during system suspend (s2ram).
>
> On R-Car Gen2 and SH-Mobile AG5 (arm32, no PSCI):
>
> Disabling non-boot CPUs ...
> BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
> in_atomic(): 0, irqs_disabled(): 128, pid: 1725, name: s2ram
> CPU: 0 PID: 1725 Comm: s2ram Not tainted
> 4.16.0-rc5-koelsch-00475-gaa69fc46cc44c3d7 #4001
> Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [<c020f6a0>] (unwind_backtrace) from [<c020b324>] (show_stack+0x10/0x14)
> [<c020b324>] (show_stack) from [<c07693c8>] (dump_stack+0x7c/0x9c)
> [<c07693c8>] (dump_stack) from [<c02422ec>] (___might_sleep+0x128/0x164)
> [<c02422ec>] (___might_sleep) from [<c0508efc>] (__pm_runtime_suspend+0x70/0xa8)
> [<c0508efc>] (__pm_runtime_suspend) from [<c0299144>] (cpu_pm_enter+0x78/0x9c)
> [<c0299144>] (cpu_pm_enter) from [<c0299170>] (cpu_pm_suspend+0x8/0x18)
> [<c0299170>] (cpu_pm_suspend) from [<c04ff8c4>] (syscore_suspend+0x88/0x138)
> [<c04ff8c4>] (syscore_suspend) from [<c0263010>]
> (suspend_devices_and_enter+0x21c/0x564)
> [<c0263010>] (suspend_devices_and_enter) from [<c02635a8>]
> (pm_suspend+0x250/0x2c8)
> [<c02635a8>] (pm_suspend) from [<c0262054>] (state_store+0xac/0xcc)
> [<c0262054>] (state_store) from [<c035f238>] (kernfs_fop_write+0x170/0x1b0)
> [<c035f238>] (kernfs_fop_write) from [<c02f8a5c>] (__vfs_write+0x2c/0x140)
> [<c02f8a5c>] (__vfs_write) from [<c02f8ce4>] (vfs_write+0xb8/0x144)
> [<c02f8ce4>] (vfs_write) from [<c02f8ea4>] (SyS_write+0x54/0xac)
> [<c02f8ea4>] (SyS_write) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
> Exception stack(0xeabcbfa8 to 0xeabcbff0)
> bfa0:                   00000004 000ce408 00000001 000ce408 00000004 00000000
> bfc0: 00000004 000ce408 b6e80b50 00000004 00000004 00000000 000c5758 00000000
> bfe0: 00000000 be866754 b6de3c85 b6e1ef26
>
>
> On R-Car Gen3 (arm64, PSCI):
>
> Disabling non-boot CPUs ...
> CPU1: shutdown
> psci: CPU1 killed.
> CPU2: shutdown
> psci: CPU2 killed.
> CPU3: shutdown
> psci: CPU3 killed.
> CPU4: shutdown
> psci: CPU4 killed.
> CPU5: shutdown
> psci: CPU5 killed.
> CPU6: shutdown
> psci: CPU6 killed.
> CPU7: shutdown
> psci: CPU7 killed.
> BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
> in_atomic(): 0, irqs_disabled(): 128, pid: 2592, name: s2ram
> 4 locks held by s2ram/2592:
>  #0:  (sb_writers#7){.+.+}, at: [<00000000cae1f0e5>] vfs_write+0xb0/0x164
>  #1:  (&of->mutex){+.+.}, at: [<000000003002e527>] kernfs_fop_write+0x114/0x1bc
>  #2:  (kn->count#71){.+.+}, at: [<000000008c0217e1>]
> kernfs_fop_write+0x11c/0x1bc
>  #3:  (pm_mutex){+.+.}, at: [<000000009a6c23e2>] pm_suspend+0x194/0xb10
> irq event stamp: 69308
> hardirqs last  enabled at (69307): [<00000000e9d09767>]
> _raw_spin_unlock_irq+0x2c/0x4c
> hardirqs last disabled at (69308): [<00000000619169c4>]
> arch_suspend_disable_irqs+0x10/0x18
> softirqs last  enabled at (69246): [<00000000b8a7706e>]
> hrtimers_dead_cpu+0x2b8/0x2f0
> softirqs last disabled at (69242): [<000000004dee0c40>]
> hrtimers_dead_cpu+0x48/0x2f0
> CPU: 0 PID: 2592 Comm: s2ram Not tainted
> 4.16.0-rc5-salvator-x-00470-g319cfb3643965f46 #1685
> Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT)
> Call trace:
>  dump_backtrace+0x0/0x140
>  show_stack+0x14/0x1c
>  dump_stack+0xb4/0xf0
>  ___might_sleep+0x1fc/0x218
>  __might_sleep+0x70/0x80
>  __pm_runtime_suspend+0x6c/0xac
>  cpu_pm_enter+0x74/0x9c
>  cpu_pm_suspend+0xc/0x1c
>  syscore_suspend+0x1b8/0x410
>  suspend_devices_and_enter+0x210/0xd9c
>  pm_suspend+0x9a4/0xb10
>  state_store+0xd4/0xf8
>  kobj_attr_store+0x18/0x28
>  sysfs_kf_write+0x50/0x5c
>  kernfs_fop_write+0x178/0x1bc
>  __vfs_write+0x38/0x140
>  vfs_write+0xc4/0x164
>  SyS_write+0x54/0xa4
>  el0_svc_naked+0x30/0x34
>
>
> I've bisected this to "[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM
> in the idle path for CPUs".

Thanks for the report, very much appreciated!

I realized that pm_runtime_irq_safe() won't be called for CPU devices
that hasn't been hooked up to a genpd - as of_genpd_attach_cpu()
hasn't been called for them.

I figure something out for the next version on how to address this
properly, until then you may test with the following change:

diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 71317ff..57250ee 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -103,7 +103,7 @@ int cpu_pm_enter(void)
                 */
                cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);

-       if (!ret && dev)
+       if (!ret && dev && pm_runtime_enabled(dev))
                pm_runtime_put_sync_suspend(dev);

        return ret;
@@ -126,7 +126,7 @@ int cpu_pm_exit(void)
 {
        struct device *dev = get_cpu_device(smp_processor_id());

-       if (dev)
+       if (dev && pm_runtime_enabled(dev))
                pm_runtime_get_sync(dev);

        return cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
-- 

Kind regards
Uffe

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

On 15 March 2018 at 12:00, Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> Hi Ulf,
>
> On Wed, Mar 14, 2018 at 5:58 PM, Ulf Hansson <ulf.hansson@linaro.org> wrote:
>> This series is a re-worked version from Lina Iyer's two series [1] that got
>> posted more than a year ago by now. I have picked up the series and done a
>> significant re-work of it and here's the result. All patches have been changed,
>> some have been dropped, some are entirely new. For this reason I decided to not
>> include a version history, as I think people need a fresh start anyway.
>
> Thanks for your series!
>
> I gave it a try on a few Renesas boards and SoCs, without adding any new DT
> descriptions.  On all of them it triggers

Thanks a lot for testing!

>
>     BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
>
> during system suspend (s2ram).
>
> On R-Car Gen2 and SH-Mobile AG5 (arm32, no PSCI):
>
> Disabling non-boot CPUs ...
> BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
> in_atomic(): 0, irqs_disabled(): 128, pid: 1725, name: s2ram
> CPU: 0 PID: 1725 Comm: s2ram Not tainted
> 4.16.0-rc5-koelsch-00475-gaa69fc46cc44c3d7 #4001
> Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [<c020f6a0>] (unwind_backtrace) from [<c020b324>] (show_stack+0x10/0x14)
> [<c020b324>] (show_stack) from [<c07693c8>] (dump_stack+0x7c/0x9c)
> [<c07693c8>] (dump_stack) from [<c02422ec>] (___might_sleep+0x128/0x164)
> [<c02422ec>] (___might_sleep) from [<c0508efc>] (__pm_runtime_suspend+0x70/0xa8)
> [<c0508efc>] (__pm_runtime_suspend) from [<c0299144>] (cpu_pm_enter+0x78/0x9c)
> [<c0299144>] (cpu_pm_enter) from [<c0299170>] (cpu_pm_suspend+0x8/0x18)
> [<c0299170>] (cpu_pm_suspend) from [<c04ff8c4>] (syscore_suspend+0x88/0x138)
> [<c04ff8c4>] (syscore_suspend) from [<c0263010>]
> (suspend_devices_and_enter+0x21c/0x564)
> [<c0263010>] (suspend_devices_and_enter) from [<c02635a8>]
> (pm_suspend+0x250/0x2c8)
> [<c02635a8>] (pm_suspend) from [<c0262054>] (state_store+0xac/0xcc)
> [<c0262054>] (state_store) from [<c035f238>] (kernfs_fop_write+0x170/0x1b0)
> [<c035f238>] (kernfs_fop_write) from [<c02f8a5c>] (__vfs_write+0x2c/0x140)
> [<c02f8a5c>] (__vfs_write) from [<c02f8ce4>] (vfs_write+0xb8/0x144)
> [<c02f8ce4>] (vfs_write) from [<c02f8ea4>] (SyS_write+0x54/0xac)
> [<c02f8ea4>] (SyS_write) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
> Exception stack(0xeabcbfa8 to 0xeabcbff0)
> bfa0:                   00000004 000ce408 00000001 000ce408 00000004 00000000
> bfc0: 00000004 000ce408 b6e80b50 00000004 00000004 00000000 000c5758 00000000
> bfe0: 00000000 be866754 b6de3c85 b6e1ef26
>
>
> On R-Car Gen3 (arm64, PSCI):
>
> Disabling non-boot CPUs ...
> CPU1: shutdown
> psci: CPU1 killed.
> CPU2: shutdown
> psci: CPU2 killed.
> CPU3: shutdown
> psci: CPU3 killed.
> CPU4: shutdown
> psci: CPU4 killed.
> CPU5: shutdown
> psci: CPU5 killed.
> CPU6: shutdown
> psci: CPU6 killed.
> CPU7: shutdown
> psci: CPU7 killed.
> BUG: sleeping function called from invalid context at
> drivers/base/power/runtime.c:1057
> in_atomic(): 0, irqs_disabled(): 128, pid: 2592, name: s2ram
> 4 locks held by s2ram/2592:
>  #0:  (sb_writers#7){.+.+}, at: [<00000000cae1f0e5>] vfs_write+0xb0/0x164
>  #1:  (&of->mutex){+.+.}, at: [<000000003002e527>] kernfs_fop_write+0x114/0x1bc
>  #2:  (kn->count#71){.+.+}, at: [<000000008c0217e1>]
> kernfs_fop_write+0x11c/0x1bc
>  #3:  (pm_mutex){+.+.}, at: [<000000009a6c23e2>] pm_suspend+0x194/0xb10
> irq event stamp: 69308
> hardirqs last  enabled at (69307): [<00000000e9d09767>]
> _raw_spin_unlock_irq+0x2c/0x4c
> hardirqs last disabled at (69308): [<00000000619169c4>]
> arch_suspend_disable_irqs+0x10/0x18
> softirqs last  enabled at (69246): [<00000000b8a7706e>]
> hrtimers_dead_cpu+0x2b8/0x2f0
> softirqs last disabled at (69242): [<000000004dee0c40>]
> hrtimers_dead_cpu+0x48/0x2f0
> CPU: 0 PID: 2592 Comm: s2ram Not tainted
> 4.16.0-rc5-salvator-x-00470-g319cfb3643965f46 #1685
> Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT)
> Call trace:
>  dump_backtrace+0x0/0x140
>  show_stack+0x14/0x1c
>  dump_stack+0xb4/0xf0
>  ___might_sleep+0x1fc/0x218
>  __might_sleep+0x70/0x80
>  __pm_runtime_suspend+0x6c/0xac
>  cpu_pm_enter+0x74/0x9c
>  cpu_pm_suspend+0xc/0x1c
>  syscore_suspend+0x1b8/0x410
>  suspend_devices_and_enter+0x210/0xd9c
>  pm_suspend+0x9a4/0xb10
>  state_store+0xd4/0xf8
>  kobj_attr_store+0x18/0x28
>  sysfs_kf_write+0x50/0x5c
>  kernfs_fop_write+0x178/0x1bc
>  __vfs_write+0x38/0x140
>  vfs_write+0xc4/0x164
>  SyS_write+0x54/0xa4
>  el0_svc_naked+0x30/0x34
>
>
> I've bisected this to "[PATCH v6 09/25] kernel/cpu_pm: Manage runtime PM
> in the idle path for CPUs".

Thanks for the report, very much appreciated!

I realized that pm_runtime_irq_safe() won't be called for CPU devices
that hasn't been hooked up to a genpd - as of_genpd_attach_cpu()
hasn't been called for them.

I figure something out for the next version on how to address this
properly, until then you may test with the following change:

diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index 71317ff..57250ee 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -103,7 +103,7 @@ int cpu_pm_enter(void)
                 */
                cpu_pm_notify(CPU_PM_ENTER_FAILED, nr_calls - 1, NULL);

-       if (!ret && dev)
+       if (!ret && dev && pm_runtime_enabled(dev))
                pm_runtime_put_sync_suspend(dev);

        return ret;
@@ -126,7 +126,7 @@ int cpu_pm_exit(void)
 {
        struct device *dev = get_cpu_device(smp_processor_id());

-       if (dev)
+       if (dev && pm_runtime_enabled(dev))
                pm_runtime_get_sync(dev);

        return cpu_pm_notify(CPU_PM_EXIT, -1, NULL);
-- 

Kind regards
Uffe

Re: [PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

On 14 March 2018 at 18:19, Mark Rutland <mark.rutland@arm.com> wrote:
> Hi Ulf,
>
> On Wed, Mar 14, 2018 at 05:58:10PM +0100, Ulf Hansson wrote:
>> For ARM, the PSCI firmware interface may be managing the power to the CPUs.
>> Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
>> could add another level of complexity from a CPU idle management point of view.
>>
>> PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
>> enables a more fine grained method, allowing Linux to get more control, in
>> regards to being energy efficient. This is typically useful for these kind of
>> complex battery driven platforms.
>
>> The series is based on v4.16-rc5, but applies also as of today to Rafael's
>> linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
>> and I except that test coverage should be increased by some more boards
>> shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).
>
> Do we have any numbers to show how this affects energy efficiency?

What kind of numbers are you looking for?

I don't have fresh energy numbers at hand, but of course I can run
some use cases and trying to get some data from the ARM energy probe.

What I have done recently, is of course to verify the solution using
trace_printk(). That in way by checking statistics from it, to for
example find out how often we enter an idle state for the CPU cluster
(and how long we stay in it).

However, as I have been working on the QCOM 410c, which currently uses
a proprietary ATF binary and which has the PSCI OSI mode supported.
Just leaving it in the Platform Coordinated mode and test to get a
comparison, won't actually say us much - because we don't know, in
what way QCOM implemented the PC mode (if at all), but of course I can
ask.

>
> Are there any other platforms with OSI support?

Our goal is to implement OSI support in the upstream version of ATF,
so anybody that want it can pick it for their ARM SoC.

We have so far played with a few hacks in ATF to enable OSI mode, as
to test this on a few other ARM SoCs (Hikey and a MTK developer board
for example).

Thanks for reviewing!

Kind regards
Uffe

[PATCH v6 00/25] PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)

[Ulf Hansson]

On 14 March 2018 at 18:19, Mark Rutland <mark.rutland@arm.com> wrote:
> Hi Ulf,
>
> On Wed, Mar 14, 2018 at 05:58:10PM +0100, Ulf Hansson wrote:
>> For ARM, the PSCI firmware interface may be managing the power to the CPUs.
>> Depending on the SoC, CPUs may also be arranged in hierarchical manner, which
>> could add another level of complexity from a CPU idle management point of view.
>>
>> PSCI v1.0+ adds support for the so called OS initiated CPU suspend mode, which
>> enables a more fine grained method, allowing Linux to get more control, in
>> regards to being energy efficient. This is typically useful for these kind of
>> complex battery driven platforms.
>
>> The series is based on v4.16-rc5, but applies also as of today to Rafael's
>> linux-pm.git next branch. The code has been tested on a QCOM 410c dragonboard
>> and I except that test coverage should be increased by some more boards
>> shortly (for ARM64 with PSCI OSI support - only a few DT changes are needed).
>
> Do we have any numbers to show how this affects energy efficiency?

What kind of numbers are you looking for?

I don't have fresh energy numbers at hand, but of course I can run
some use cases and trying to get some data from the ARM energy probe.

What I have done recently, is of course to verify the solution using
trace_printk(). That in way by checking statistics from it, to for
example find out how often we enter an idle state for the CPU cluster
(and how long we stay in it).

However, as I have been working on the QCOM 410c, which currently uses
a proprietary ATF binary and which has the PSCI OSI mode supported.
Just leaving it in the Platform Coordinated mode and test to get a
comparison, won't actually say us much - because we don't know, in
what way QCOM implemented the PC mode (if at all), but of course I can
ask.

>
> Are there any other platforms with OSI support?

Our goal is to implement OSI support in the upstream version of ATF,
so anybody that want it can pick it for their ARM SoC.

We have so far played with a few hacks in ATF to enable OSI mode, as
to test this on a few other ARM SoCs (Hikey and a MTK developer board
for example).

Thanks for reviewing!

Kind regards
Uffe

Re: [PATCH v6 11/25] of: base: Add of_get_cpu_state_node() to get idle states for a CPU node

[Rob Herring]

On Wed, Mar 14, 2018 at 05:58:21PM +0100, Ulf Hansson wrote:
> The CPU's idle state nodes are currently parsed at the common cpuidle DT
> library, but also when initializing back-end data for the arch specific CPU
> operations, as in the PSCI driver case.
> 
> To avoid open-coding, let's introduce of_get_cpu_state_node(), which takes
> the device node for the CPU and the index to the requested idle state node,
> as in-parameters. In case a corresponding idle state node is found, it
> returns the node with the refcount incremented for it, else it returns
> NULL.
> 
> Moreover, for ARM, there are two generic methods, to describe the CPU's
> idle states, either via the flattened description through the
> "cpu-idle-states" binding [1] or via the hierarchical layout, using the
> "power-domains" and the "domain-idle-states" bindings [2]. Hence, let's
> take both options into account.
> 
> [1]
> Documentation/devicetree/bindings/arm/idle-states.txt
> [2]
> Documentation/devicetree/bindings/arm/psci.txt
> 
> Cc: Rob Herring <robh+dt@kernel.org>
> Cc: devicetree@vger.kernel.org
> Cc: Lina Iyer <ilina@codeaurora.org>
> Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/of/base.c  | 35 +++++++++++++++++++++++++++++++++++
>  include/linux/of.h |  8 ++++++++
>  2 files changed, 43 insertions(+)

I'd prefer not to add this here, but don't have a better suggestion. If 
more cpu related functions are added, I'd like to move them to a 
separate file in drivers/of/, but that can wait.

Reviewed-by: Rob Herring <robh@kernel.org>

[PATCH v6 11/25] of: base: Add of_get_cpu_state_node() to get idle states for a CPU node

[Rob Herring]

On Wed, Mar 14, 2018 at 05:58:21PM +0100, Ulf Hansson wrote:
> The CPU's idle state nodes are currently parsed at the common cpuidle DT
> library, but also when initializing back-end data for the arch specific CPU
> operations, as in the PSCI driver case.
> 
> To avoid open-coding, let's introduce of_get_cpu_state_node(), which takes
> the device node for the CPU and the index to the requested idle state node,
> as in-parameters. In case a corresponding idle state node is found, it
> returns the node with the refcount incremented for it, else it returns
> NULL.
> 
> Moreover, for ARM, there are two generic methods, to describe the CPU's
> idle states, either via the flattened description through the
> "cpu-idle-states" binding [1] or via the hierarchical layout, using the
> "power-domains" and the "domain-idle-states" bindings [2]. Hence, let's
> take both options into account.
> 
> [1]
> Documentation/devicetree/bindings/arm/idle-states.txt
> [2]
> Documentation/devicetree/bindings/arm/psci.txt
> 
> Cc: Rob Herring <robh+dt@kernel.org>
> Cc: devicetree at vger.kernel.org
> Cc: Lina Iyer <ilina@codeaurora.org>
> Suggested-by: Sudeep Holla <sudeep.holla@arm.com>
> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/of/base.c  | 35 +++++++++++++++++++++++++++++++++++
>  include/linux/of.h |  8 ++++++++
>  2 files changed, 43 insertions(+)

I'd prefer not to add this here, but don't have a better suggestion. If 
more cpu related functions are added, I'd like to move them to a 
separate file in drivers/of/, but that can wait.

Reviewed-by: Rob Herring <robh@kernel.org>

Re: [PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
>> In case the hierarchical layout is used in DT, we want to initialize the
>> corresponding PM domain topology for the CPUs, by using the generic PM
>> domain (aka genpd) infrastructure.
>>
>> At first glance, it may seem feasible to hook into the existing
>> psci_dt_init() function, although because it's called quite early in the
>> boot sequence, allocating the dynamic data structure for a genpd doesn't
>> work.
>>
>> Therefore, let's export a new init function for PSCI,
>> psci_dt_topology_init(), which the ARM machine code should call from a
>> suitable initcall.
>>
>> Succeeding to initialize the PM domain topology, which means at least one
>> instance of a genpd becomes created, allows us to continue to enable the
>> PSCI OS initiated mode for the platform. If everything turns out fine,
>> let's print a message in log to inform the user about the changed mode.
>>
>> In case of any failures, we stick to the default PSCI Platform Coordinated
>> mode.
>
> For kexec/kdump we'll need to explicitly set the suspend mode to
> platform coordinated if for whatever reason we choose not to use OSI.

Could you please elaborate on this? I am not really understanding what
you are suggesting me to do and why.

>
>>
>> Cc: Lina Iyer <ilina@codeaurora.org>
>> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
>> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
>> ---
>>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
>>  include/linux/psci.h    |  2 ++
>>  2 files changed, 33 insertions(+)
>>
>> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
>> index 463f78c..45d55fc 100644
>> --- a/drivers/firmware/psci.c
>> +++ b/drivers/firmware/psci.c
>> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
>>       return ret;
>>  }
>>
>> +int __init psci_dt_topology_init(void)
>> +{
>> +     struct device_node *np;
>> +     int ret;
>> +
>> +     if (!psci_has_osi_support())
>> +             return 0;
>> +
>> +     np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
>> +     if (!np)
>> +             return -ENODEV;
>> +
>> +     /* Initialize the CPU PM domains based on topology described in DT. */
>> +     ret = psci_dt_init_pm_domains(np);
>> +     if (ret <= 0)
>> +             goto out;
>> +
>> +     /* Enable OSI mode. */
>> +     ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
>> +                          PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
>
> IIUC, this patch will need to be moved after the subsequent two patches
> to ensure that this series bisects cleanly.

Actually not.

$subject patch adds a new init function, which purpose is to
initialize the PSCI PM domains. At this point none is calling it.

The following changes, builds on top, making it possible to add CPU
devices to the PSCI PM domains (if the PM domains has been created and
OSI mode successfully enabled) and also to properly deal CPU hotplug
when OSI mode is enabled.

The purpose doing it this way is to keep changes small and
review-able, but in principle I can squash 20, 21 and 22 - if you
prefer.

Again, thanks for reviewing!

Kind regards
Uffe

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
>> In case the hierarchical layout is used in DT, we want to initialize the
>> corresponding PM domain topology for the CPUs, by using the generic PM
>> domain (aka genpd) infrastructure.
>>
>> At first glance, it may seem feasible to hook into the existing
>> psci_dt_init() function, although because it's called quite early in the
>> boot sequence, allocating the dynamic data structure for a genpd doesn't
>> work.
>>
>> Therefore, let's export a new init function for PSCI,
>> psci_dt_topology_init(), which the ARM machine code should call from a
>> suitable initcall.
>>
>> Succeeding to initialize the PM domain topology, which means at least one
>> instance of a genpd becomes created, allows us to continue to enable the
>> PSCI OS initiated mode for the platform. If everything turns out fine,
>> let's print a message in log to inform the user about the changed mode.
>>
>> In case of any failures, we stick to the default PSCI Platform Coordinated
>> mode.
>
> For kexec/kdump we'll need to explicitly set the suspend mode to
> platform coordinated if for whatever reason we choose not to use OSI.

Could you please elaborate on this? I am not really understanding what
you are suggesting me to do and why.

>
>>
>> Cc: Lina Iyer <ilina@codeaurora.org>
>> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
>> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
>> ---
>>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
>>  include/linux/psci.h    |  2 ++
>>  2 files changed, 33 insertions(+)
>>
>> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
>> index 463f78c..45d55fc 100644
>> --- a/drivers/firmware/psci.c
>> +++ b/drivers/firmware/psci.c
>> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
>>       return ret;
>>  }
>>
>> +int __init psci_dt_topology_init(void)
>> +{
>> +     struct device_node *np;
>> +     int ret;
>> +
>> +     if (!psci_has_osi_support())
>> +             return 0;
>> +
>> +     np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
>> +     if (!np)
>> +             return -ENODEV;
>> +
>> +     /* Initialize the CPU PM domains based on topology described in DT. */
>> +     ret = psci_dt_init_pm_domains(np);
>> +     if (ret <= 0)
>> +             goto out;
>> +
>> +     /* Enable OSI mode. */
>> +     ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
>> +                          PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
>
> IIUC, this patch will need to be moved after the subsequent two patches
> to ensure that this series bisects cleanly.

Actually not.

$subject patch adds a new init function, which purpose is to
initialize the PSCI PM domains. At this point none is calling it.

The following changes, builds on top, making it possible to add CPU
devices to the PSCI PM domains (if the PM domains has been created and
OSI mode successfully enabled) and also to properly deal CPU hotplug
when OSI mode is enabled.

The purpose doing it this way is to keep changes small and
review-able, but in principle I can squash 20, 21 and 22 - if you
prefer.

Again, thanks for reviewing!

Kind regards
Uffe

Re: [PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> >> In case the hierarchical layout is used in DT, we want to initialize the
> >> corresponding PM domain topology for the CPUs, by using the generic PM
> >> domain (aka genpd) infrastructure.
> >>
> >> At first glance, it may seem feasible to hook into the existing
> >> psci_dt_init() function, although because it's called quite early in the
> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
> >> work.
> >>
> >> Therefore, let's export a new init function for PSCI,
> >> psci_dt_topology_init(), which the ARM machine code should call from a
> >> suitable initcall.
> >>
> >> Succeeding to initialize the PM domain topology, which means at least one
> >> instance of a genpd becomes created, allows us to continue to enable the
> >> PSCI OS initiated mode for the platform. If everything turns out fine,
> >> let's print a message in log to inform the user about the changed mode.
> >>
> >> In case of any failures, we stick to the default PSCI Platform Coordinated
> >> mode.
> >
> > For kexec/kdump we'll need to explicitly set the suspend mode to
> > platform coordinated if for whatever reason we choose not to use OSI.
> 
> Could you please elaborate on this? I am not really understanding what
> you are suggesting me to do and why.

Sorry for not being clear.

What I'd like to see is that we always call SET_SUSPEND_MODE before
invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
setting it to platform co-ordinated early on in boot and later switching
it over.

That way we can be sure of the suspend mode, even if we've kexec'd from
a kernel that had fiddled with it.

> >> Cc: Lina Iyer <ilina@codeaurora.org>
> >> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> >> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> >> ---
> >>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
> >>  include/linux/psci.h    |  2 ++
> >>  2 files changed, 33 insertions(+)
> >>
> >> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> >> index 463f78c..45d55fc 100644
> >> --- a/drivers/firmware/psci.c
> >> +++ b/drivers/firmware/psci.c
> >> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
> >>       return ret;
> >>  }
> >>
> >> +int __init psci_dt_topology_init(void)
> >> +{
> >> +     struct device_node *np;
> >> +     int ret;
> >> +
> >> +     if (!psci_has_osi_support())
> >> +             return 0;
> >> +
> >> +     np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
> >> +     if (!np)
> >> +             return -ENODEV;
> >> +
> >> +     /* Initialize the CPU PM domains based on topology described in DT. */
> >> +     ret = psci_dt_init_pm_domains(np);
> >> +     if (ret <= 0)
> >> +             goto out;
> >> +
> >> +     /* Enable OSI mode. */
> >> +     ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
> >> +                          PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
> >
> > IIUC, this patch will need to be moved after the subsequent two patches
> > to ensure that this series bisects cleanly.
> 
> Actually not.
> 
> $subject patch adds a new init function, which purpose is to
> initialize the PSCI PM domains. At this point none is calling it.

Ah. I missed the fact there were no callers. Sorry for the noise!

Thanks,
Mark.

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> >> In case the hierarchical layout is used in DT, we want to initialize the
> >> corresponding PM domain topology for the CPUs, by using the generic PM
> >> domain (aka genpd) infrastructure.
> >>
> >> At first glance, it may seem feasible to hook into the existing
> >> psci_dt_init() function, although because it's called quite early in the
> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
> >> work.
> >>
> >> Therefore, let's export a new init function for PSCI,
> >> psci_dt_topology_init(), which the ARM machine code should call from a
> >> suitable initcall.
> >>
> >> Succeeding to initialize the PM domain topology, which means at least one
> >> instance of a genpd becomes created, allows us to continue to enable the
> >> PSCI OS initiated mode for the platform. If everything turns out fine,
> >> let's print a message in log to inform the user about the changed mode.
> >>
> >> In case of any failures, we stick to the default PSCI Platform Coordinated
> >> mode.
> >
> > For kexec/kdump we'll need to explicitly set the suspend mode to
> > platform coordinated if for whatever reason we choose not to use OSI.
> 
> Could you please elaborate on this? I am not really understanding what
> you are suggesting me to do and why.

Sorry for not being clear.

What I'd like to see is that we always call SET_SUSPEND_MODE before
invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
setting it to platform co-ordinated early on in boot and later switching
it over.

That way we can be sure of the suspend mode, even if we've kexec'd from
a kernel that had fiddled with it.

> >> Cc: Lina Iyer <ilina@codeaurora.org>
> >> Co-developed-by: Lina Iyer <lina.iyer@linaro.org>
> >> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> >> ---
> >>  drivers/firmware/psci.c | 31 +++++++++++++++++++++++++++++++
> >>  include/linux/psci.h    |  2 ++
> >>  2 files changed, 33 insertions(+)
> >>
> >> diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c
> >> index 463f78c..45d55fc 100644
> >> --- a/drivers/firmware/psci.c
> >> +++ b/drivers/firmware/psci.c
> >> @@ -723,6 +723,37 @@ int __init psci_dt_init(void)
> >>       return ret;
> >>  }
> >>
> >> +int __init psci_dt_topology_init(void)
> >> +{
> >> +     struct device_node *np;
> >> +     int ret;
> >> +
> >> +     if (!psci_has_osi_support())
> >> +             return 0;
> >> +
> >> +     np = of_find_matching_node_and_match(NULL, psci_of_match, NULL);
> >> +     if (!np)
> >> +             return -ENODEV;
> >> +
> >> +     /* Initialize the CPU PM domains based on topology described in DT. */
> >> +     ret = psci_dt_init_pm_domains(np);
> >> +     if (ret <= 0)
> >> +             goto out;
> >> +
> >> +     /* Enable OSI mode. */
> >> +     ret = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE,
> >> +                          PSCI_1_0_SUSPEND_MODE_OSI, 0, 0);
> >
> > IIUC, this patch will need to be moved after the subsequent two patches
> > to ensure that this series bisects cleanly.
> 
> Actually not.
> 
> $subject patch adds a new init function, which purpose is to
> initialize the PSCI PM domains. At this point none is calling it.

Ah. I missed the fact there were no callers. Sorry for the noise!

Thanks,
Mark.

Re: [PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

On 10 April 2018 at 13:10, Mark Rutland <mark.rutland@arm.com> wrote:
> On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
>> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
>> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
>> >> In case the hierarchical layout is used in DT, we want to initialize the
>> >> corresponding PM domain topology for the CPUs, by using the generic PM
>> >> domain (aka genpd) infrastructure.
>> >>
>> >> At first glance, it may seem feasible to hook into the existing
>> >> psci_dt_init() function, although because it's called quite early in the
>> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
>> >> work.
>> >>
>> >> Therefore, let's export a new init function for PSCI,
>> >> psci_dt_topology_init(), which the ARM machine code should call from a
>> >> suitable initcall.
>> >>
>> >> Succeeding to initialize the PM domain topology, which means at least one
>> >> instance of a genpd becomes created, allows us to continue to enable the
>> >> PSCI OS initiated mode for the platform. If everything turns out fine,
>> >> let's print a message in log to inform the user about the changed mode.
>> >>
>> >> In case of any failures, we stick to the default PSCI Platform Coordinated
>> >> mode.
>> >
>> > For kexec/kdump we'll need to explicitly set the suspend mode to
>> > platform coordinated if for whatever reason we choose not to use OSI.
>>
>> Could you please elaborate on this? I am not really understanding what
>> you are suggesting me to do and why.
>
> Sorry for not being clear.
>
> What I'd like to see is that we always call SET_SUSPEND_MODE before
> invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
> setting it to platform co-ordinated early on in boot and later switching
> it over.
>
> That way we can be sure of the suspend mode, even if we've kexec'd from
> a kernel that had fiddled with it.

Right, good point. Let's me update the patch to deal with that.

One thing though, kexec from a "new" kernel having OSI mode enabled
into an "old" kernel that doesn't even have the new OSI mode support
in psci driver, is going to be difficult to deal with.
On the other hand I guess that isn't a very important usecase we need
to cover, or is it?

[...]

Kind regards
Uffe

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Ulf Hansson]

On 10 April 2018 at 13:10, Mark Rutland <mark.rutland@arm.com> wrote:
> On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
>> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
>> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
>> >> In case the hierarchical layout is used in DT, we want to initialize the
>> >> corresponding PM domain topology for the CPUs, by using the generic PM
>> >> domain (aka genpd) infrastructure.
>> >>
>> >> At first glance, it may seem feasible to hook into the existing
>> >> psci_dt_init() function, although because it's called quite early in the
>> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
>> >> work.
>> >>
>> >> Therefore, let's export a new init function for PSCI,
>> >> psci_dt_topology_init(), which the ARM machine code should call from a
>> >> suitable initcall.
>> >>
>> >> Succeeding to initialize the PM domain topology, which means at least one
>> >> instance of a genpd becomes created, allows us to continue to enable the
>> >> PSCI OS initiated mode for the platform. If everything turns out fine,
>> >> let's print a message in log to inform the user about the changed mode.
>> >>
>> >> In case of any failures, we stick to the default PSCI Platform Coordinated
>> >> mode.
>> >
>> > For kexec/kdump we'll need to explicitly set the suspend mode to
>> > platform coordinated if for whatever reason we choose not to use OSI.
>>
>> Could you please elaborate on this? I am not really understanding what
>> you are suggesting me to do and why.
>
> Sorry for not being clear.
>
> What I'd like to see is that we always call SET_SUSPEND_MODE before
> invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
> setting it to platform co-ordinated early on in boot and later switching
> it over.
>
> That way we can be sure of the suspend mode, even if we've kexec'd from
> a kernel that had fiddled with it.

Right, good point. Let's me update the patch to deal with that.

One thing though, kexec from a "new" kernel having OSI mode enabled
into an "old" kernel that doesn't even have the new OSI mode support
in psci driver, is going to be difficult to deal with.
On the other hand I guess that isn't a very important usecase we need
to cover, or is it?

[...]

Kind regards
Uffe

Re: [PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Tue, Apr 10, 2018 at 02:25:52PM +0200, Ulf Hansson wrote:
> On 10 April 2018 at 13:10, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
> >> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> >> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> >> >> In case the hierarchical layout is used in DT, we want to initialize the
> >> >> corresponding PM domain topology for the CPUs, by using the generic PM
> >> >> domain (aka genpd) infrastructure.
> >> >>
> >> >> At first glance, it may seem feasible to hook into the existing
> >> >> psci_dt_init() function, although because it's called quite early in the
> >> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
> >> >> work.
> >> >>
> >> >> Therefore, let's export a new init function for PSCI,
> >> >> psci_dt_topology_init(), which the ARM machine code should call from a
> >> >> suitable initcall.
> >> >>
> >> >> Succeeding to initialize the PM domain topology, which means at least one
> >> >> instance of a genpd becomes created, allows us to continue to enable the
> >> >> PSCI OS initiated mode for the platform. If everything turns out fine,
> >> >> let's print a message in log to inform the user about the changed mode.
> >> >>
> >> >> In case of any failures, we stick to the default PSCI Platform Coordinated
> >> >> mode.
> >> >
> >> > For kexec/kdump we'll need to explicitly set the suspend mode to
> >> > platform coordinated if for whatever reason we choose not to use OSI.
> >>
> >> Could you please elaborate on this? I am not really understanding what
> >> you are suggesting me to do and why.
> >
> > Sorry for not being clear.
> >
> > What I'd like to see is that we always call SET_SUSPEND_MODE before
> > invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
> > setting it to platform co-ordinated early on in boot and later switching
> > it over.
> >
> > That way we can be sure of the suspend mode, even if we've kexec'd from
> > a kernel that had fiddled with it.
> 
> Right, good point. Let's me update the patch to deal with that.
> 
> One thing though, kexec from a "new" kernel having OSI mode enabled
> into an "old" kernel that doesn't even have the new OSI mode support
> in psci driver, is going to be difficult to deal with.
> On the other hand I guess that isn't a very important usecase we need
> to cover, or is it?

I think we can also transition back to platform-coordinated in a reboot
notifier, which should cater for kexec to an old kernel. I agree it
shouldn't be a big deal, though.

Thanks,
Mark.

[PATCH v6 20/25] drivers: firmware: psci: Introduce psci_dt_topology_init()

[Mark Rutland]

On Tue, Apr 10, 2018 at 02:25:52PM +0200, Ulf Hansson wrote:
> On 10 April 2018 at 13:10, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Tue, Apr 10, 2018 at 09:19:26AM +0200, Ulf Hansson wrote:
> >> On 14 March 2018 at 18:38, Mark Rutland <mark.rutland@arm.com> wrote:
> >> > On Wed, Mar 14, 2018 at 05:58:30PM +0100, Ulf Hansson wrote:
> >> >> In case the hierarchical layout is used in DT, we want to initialize the
> >> >> corresponding PM domain topology for the CPUs, by using the generic PM
> >> >> domain (aka genpd) infrastructure.
> >> >>
> >> >> At first glance, it may seem feasible to hook into the existing
> >> >> psci_dt_init() function, although because it's called quite early in the
> >> >> boot sequence, allocating the dynamic data structure for a genpd doesn't
> >> >> work.
> >> >>
> >> >> Therefore, let's export a new init function for PSCI,
> >> >> psci_dt_topology_init(), which the ARM machine code should call from a
> >> >> suitable initcall.
> >> >>
> >> >> Succeeding to initialize the PM domain topology, which means at least one
> >> >> instance of a genpd becomes created, allows us to continue to enable the
> >> >> PSCI OS initiated mode for the platform. If everything turns out fine,
> >> >> let's print a message in log to inform the user about the changed mode.
> >> >>
> >> >> In case of any failures, we stick to the default PSCI Platform Coordinated
> >> >> mode.
> >> >
> >> > For kexec/kdump we'll need to explicitly set the suspend mode to
> >> > platform coordinated if for whatever reason we choose not to use OSI.
> >>
> >> Could you please elaborate on this? I am not really understanding what
> >> you are suggesting me to do and why.
> >
> > Sorry for not being clear.
> >
> > What I'd like to see is that we always call SET_SUSPEND_MODE before
> > invoking CPU_SUSPEND. Either deciding early if we can use OSI, or always
> > setting it to platform co-ordinated early on in boot and later switching
> > it over.
> >
> > That way we can be sure of the suspend mode, even if we've kexec'd from
> > a kernel that had fiddled with it.
> 
> Right, good point. Let's me update the patch to deal with that.
> 
> One thing though, kexec from a "new" kernel having OSI mode enabled
> into an "old" kernel that doesn't even have the new OSI mode support
> in psci driver, is going to be difficult to deal with.
> On the other hand I guess that isn't a very important usecase we need
> to cover, or is it?

I think we can also transition back to platform-coordinated in a reboot
notifier, which should cater for kexec to an old kernel. I agree it
shouldn't be a big deal, though.

Thanks,
Mark.