[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

Power Domains currently support turning on/off only in process context.
This restricts the usage of PM domains to devices and domains that
could be powered on/off in irq disabled contexts as the mutexes used in
GenPD allows for cpu sleep while waiting for locks.

Genpd inherently provides support for devices, domain hierarchy and can
be used to represent cpu clusters like in ARM's big.Little, where, each
cpu cluster is in its domain, with supporting caches and other
peripheral hardware. Multiple such domains could be part of another
domain. Because mutexes are used to protect and synchronize domain
operations and cpu idle operations are inherently atomic, the use of
genpd is not possible for runtime suspend and resume of the pm domain.
Replacing the locks to spinlocks would allow cpu domain to be be powered
off to save power, when all the cpus are powered off.

However, not all domains can operate in irq-safe contexts and usually
would need to sleep during domain operations. So genpd has to support
both the cases, where the domain is or is not irq-safe. The irq-safe
attribute is therefore domain specific.

To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
while defining the domain. This determines if the domain should use a
spinlock instead of a mutex. Locking is abstracted through
genpd_lock_domain() and genpd_unlock_domain() functions that use the
flag to determine the locking to be used for this domain.

The restriction this imposes on the domain hierarchy is that subdomains
and all devices in the hierarchy also be irq-safe. Non irq-safe domains
may continue to have irq-safe devices, but not the other way around.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
 include/linux/pm_domain.h   |  11 ++-
 2 files changed, 164 insertions(+), 47 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index dfd7595..8b89d15 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -50,6 +50,71 @@
 static LIST_HEAD(gpd_list);
 static DEFINE_MUTEX(gpd_list_lock);
 
+static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->slock)
+{
+	unsigned long flags;
+
+	if (unlikely(subclass > 0))
+		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
+	else
+		spin_lock_irqsave(&genpd->slock, flags);
+
+	genpd->flags = flags;
+
+	return 0;
+}
+
+static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
+	__releases(&genpd->slock)
+{
+	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
+	return 0;
+}
+
+static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->mlock)
+{
+	if (unlikely(subclass > 0))
+		mutex_lock_nested(&genpd->mlock, subclass);
+	else
+		mutex_lock(&genpd->mlock);
+
+	return 0;
+}
+
+static inline int genpd_lock_domain_interruptible_irq(
+				struct generic_pm_domain *genpd)
+	__acquires(&genpd->mlock)
+{
+	return mutex_lock_interruptible(&genpd->mlock);
+}
+
+static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
+	__releases(&genpd->mlock)
+{
+	mutex_unlock(&genpd->mlock);
+	return 0;
+}
+
+#define genpd_lock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_irq(genpd, 0))
+
+#define genpd_lock_domain_nested(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
+			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
+
+#define genpd_unlock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
+			: genpd_unlock_domain_irq(genpd))
+
+#define genpd_lock_domain_interruptible(genpd)			\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_interruptible_irq(genpd))
+
 static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
 {
 	struct generic_pm_domain *genpd = NULL, *gpd;
@@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
 {
 	int ret;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
 		spin_unlock_irq(&dev->power.lock);
 
 		if (!IS_ERR(genpd)) {
-			mutex_lock(&genpd->lock);
+			genpd_lock_domain(genpd);
 			genpd->max_off_time_changed = true;
-			mutex_unlock(&genpd->lock);
+			genpd_unlock_domain(genpd);
 		}
 
 		dev = dev->parent;
@@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
 			return -EBUSY;
 
 		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
-		    || pdd->dev->power.irq_safe))
+			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
 			not_suspended++;
 	}
 
@@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
 
 	genpd = container_of(work, struct generic_pm_domain, power_off_work);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	pm_genpd_poweroff(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 }
 
 /**
@@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/*
-	 * We can't allow to power off the PM domain if it holds an irq_safe
-	 * device. That's beacuse we use mutexes to protect data while power
-	 * off and on the PM domain, thus we can't execute in atomic context.
-	 */
-	if (dev->power.irq_safe)
+	/* We can't allow to power off a domain that is also not irq safe. */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return -EBUSY;
 
 	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
@@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 		return ret;
 	}
 
-	mutex_lock(&genpd->lock);
+	/*
+	 * If power.irq_safe is set, this routine will be run with interrupts
+	 * off, so suspend only if the power domain is irq_safe.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
+		return 0;
+
+	genpd_lock_domain(genpd);
+
 	genpd->in_progress++;
 	pm_genpd_poweroff(genpd);
 	genpd->in_progress--;
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
@@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/* If power.irq_safe, the PM domain is never powered off. */
-	if (dev->power.irq_safe)
+	/*
+	 * If power.irq_safe and domain is not, then
+	 * the PM domain is never powered off.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return genpd_start_dev_no_timing(genpd, dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		return ret;
@@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
 	if (resume_needed(dev, genpd))
 		pm_runtime_resume(dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count++ == 0) {
 		genpd->suspended_count = 0;
 		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (genpd->suspend_power_off) {
 		pm_runtime_put_noidle(dev);
@@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
 
 	ret = pm_generic_prepare(dev);
 	if (ret) {
-		mutex_lock(&genpd->lock);
+		genpd_lock_domain(genpd);
 
 		if (--genpd->prepared_count == 0)
 			genpd->suspend_power_off = false;
 
-		mutex_unlock(&genpd->lock);
+		genpd_unlock_domain(genpd);
 		pm_runtime_enable(dev);
 	}
 
@@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
 	if (IS_ERR(genpd))
 		return;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	run_complete = !genpd->suspend_power_off;
 	if (--genpd->prepared_count == 0)
 		genpd->suspend_power_off = false;
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (run_complete) {
 		pm_generic_complete(dev);
@@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
 		return -EINVAL;
 
+	/* Devices in an IRQ safe PM Domain have to be irq safe too */
+	if (genpd->irq_safe && !dev->power.irq_safe) {
+		dev_warn(dev,
+			"Devices in an irq-safe domain have to be irq safe.\n");
+		return -EINVAL;
+	}
+
 	gpd_data = genpd_alloc_dev_data(dev, genpd, td);
 	if (IS_ERR(gpd_data))
 		return PTR_ERR(gpd_data);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		genpd_free_dev_data(dev, gpd_data);
@@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 	gpd_data = to_gpd_data(pdd);
 	dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 
 	list_del_init(&pdd->list_node);
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	genpd_free_dev_data(dev, gpd_data);
 
 	return 0;
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	dev_pm_qos_add_notifier(dev, &gpd_data->nb);
 
 	return ret;
@@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	/*
+	 * If the domain can be powered on/off in an irq safe
+	 * context, ensure that the subdomain can also be
+	 * powered on/off in that context.
+	 */
+	if (genpd->irq_safe && !subdomain->irq_safe) {
+		pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
+				subdomain->name, genpd->name);
+		return -EINVAL;
+	}
+
 	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
-	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+	genpd_lock_domain(genpd);
+	genpd_lock_domain_nested(subdomain);
 
 	if (genpd->status == GPD_STATE_POWER_OFF
 	    &&  subdomain->status != GPD_STATE_POWER_OFF) {
@@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		genpd_sd_counter_inc(genpd);
 
  out:
-	mutex_unlock(&subdomain->lock);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(subdomain);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	list_for_each_entry(link, &genpd->master_links, master_node) {
 		if (link->slave != subdomain)
 			continue;
 
-		mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+		genpd_lock_domain_nested(subdomain);
 
 		list_del(&link->master_node);
 		list_del(&link->slave_node);
@@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 		if (subdomain->status != GPD_STATE_POWER_OFF)
 			genpd_sd_counter_dec(genpd);
 
-		mutex_unlock(&subdomain->lock);
+		genpd_unlock_domain(subdomain);
 
 		ret = 0;
 		break;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	if (genpd->irq_safe) {
+		pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
+				genpd->name);
+		return -EINVAL;
+	}
+
 	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
 	if (!cpuidle_data)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
@@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	genpd_recalc_cpu_exit_latency(genpd);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 
  err:
@@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	if (IS_ERR_OR_NULL(genpd))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	cpuidle_data = genpd->cpuidle_data;
 	if (!cpuidle_data) {
@@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	kfree(cpuidle_data);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
 	return cb ? cb(dev) : 0;
 }
 
+static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
+{
+	if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
+		spin_lock_init(&genpd->slock);
+		genpd->irq_safe = true;
+	} else {
+		mutex_init(&genpd->mlock);
+		genpd->irq_safe = false;
+	}
+}
+
 /**
  * pm_genpd_init - Initialize a generic I/O PM domain object.
  * @genpd: PM domain object to initialize.
@@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
 	INIT_LIST_HEAD(&genpd->master_links);
 	INIT_LIST_HEAD(&genpd->slave_links);
 	INIT_LIST_HEAD(&genpd->dev_list);
-	mutex_init(&genpd->lock);
+	genpd_lock_domain_init(genpd);
 	genpd->gov = gov;
 	INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
 	genpd->in_progress = 0;
@@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	struct gpd_link *link;
 	int ret;
 
-	ret = mutex_lock_interruptible(&genpd->lock);
+	ret = genpd_lock_domain_interruptible(genpd);
 	if (ret)
 		return -ERESTARTSYS;
 
@@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	}
 
 	list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
-		kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
+		kobj_path = kobject_get_path(&pm_data->dev->kobj,
+				genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
 		if (kobj_path == NULL)
 			continue;
 
@@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 
 	seq_puts(s, "\n");
 exit:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index b2725e6..dc7cb53 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -16,9 +16,11 @@
 #include <linux/of.h>
 #include <linux/notifier.h>
 #include <linux/cpuidle.h>
+#include <linux/spinlock.h>
 
 /* Defines used for the flags field in the struct generic_pm_domain */
 #define GENPD_FLAG_PM_CLK	(1U << 0) /* PM domain uses PM clk */
+#define GENPD_FLAG_IRQ_SAFE	(1U << 1) /* PM domain operates in atomic */
 
 enum gpd_status {
 	GPD_STATE_ACTIVE = 0,	/* PM domain is active */
@@ -49,7 +51,6 @@ struct generic_pm_domain {
 	struct list_head master_links;	/* Links with PM domain as a master */
 	struct list_head slave_links;	/* Links with PM domain as a slave */
 	struct list_head dev_list;	/* List of devices */
-	struct mutex lock;
 	struct dev_power_governor *gov;
 	struct work_struct power_off_work;
 	const char *name;
@@ -74,6 +75,14 @@ struct generic_pm_domain {
 	void (*detach_dev)(struct generic_pm_domain *domain,
 			   struct device *dev);
 	unsigned int flags;		/* Bit field of configs for genpd */
+	bool irq_safe;
+	union {
+		struct mutex mlock;
+		struct {
+			spinlock_t slock;
+			unsigned long lock_flags;
+		};
+	};
 };
 
 static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
-- 
2.1.4

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

Generally cpus are grouped under a power domain in a SoC. When all cpus
in the domain are in their power off state, the cpu domain can also be
powered off. Genpd provides the framework for defining cpus as devices
that are part of a cpu domain.

Introduce support for defining and adding a generic power domain for the
cpus based on the DT specification of power domain providers and
consumers.  SoC's that have the cpu domain defined in their DT, can
setup a genpd with a name and the power_on/power_off callbacks. Calling
pm_cpu_domain_init() will register the genpd and attach the cpus for
this domain with the genpd.

CPU_PM notifications for are used to pm_runtime_get_sync() and
pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
last cpu going down would call the genpd->power_off(). Correspondingly,
the first cpu up would call the genpd->power_on() callback before
resuming from idle.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/Makefile     |   1 +
 drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h       |  12 +++
 kernel/power/Kconfig            |  12 +++
 4 files changed, 212 insertions(+)
 create mode 100644 drivers/base/power/cpu_domain.c

diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
index 1cb8544..debfc74 100644
--- a/drivers/base/power/Makefile
+++ b/drivers/base/power/Makefile
@@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
 obj-$(CONFIG_PM_OPP)	+= opp.o
 obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
 obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
+obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
 
 ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
new file mode 100644
index 0000000..ee90094
--- /dev/null
+++ b/drivers/base/power/cpu_domain.c
@@ -0,0 +1,187 @@
+/*
+ * Generic CPU domain runtime power on/off support
+ *
+ * Copyright (C) 2015 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpu_pm.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/pm_domain.h>
+#include <linux/pm_runtime.h>
+
+static struct cpumask cpus_handled;
+
+static void do_cpu(void *unused)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	pm_runtime_get_sync(dev);
+}
+
+static int cpuidle_genpd_device_init(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	/*
+	 * CPU device have to be irq safe for use with cpuidle, which runs
+	 * with irqs disabled.
+	 */
+	pm_runtime_irq_safe(dev);
+	pm_runtime_enable(dev);
+
+	genpd_dev_pm_attach(dev);
+
+	/*
+	 * Execute the below on 'that' cpu to ensure that the reference
+	 * counting is correct. Its possible that while this code is
+	 * executed, the cpu may be in idle but we may incorrectly
+	 * increment the usage. By executing the do_cpu on 'that' cpu,
+	 * we can ensure that the cpu and the usage count are matched.
+	 */
+	return smp_call_function_single(cpu, do_cpu, NULL, true);
+}
+
+static int cpu_state_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_PM_ENTER:
+		pm_runtime_put_sync(dev);
+		break;
+
+	case CPU_PM_ENTER_FAILED:
+	case CPU_PM_EXIT:
+		pm_runtime_get_sync(dev);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static int cpu_online_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = (unsigned long)hcpu;
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_STARTING:
+	case CPU_STARTING_FROZEN:
+		/*
+		 * Attach the cpu to its domain if the cpu is coming up
+		 * for the first time.
+		 * Called from the cpu that is coming up.
+		 */
+		if (!genpd_dev_pm_attach(dev))
+			do_cpu(NULL);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block hotplug_notifier = {
+	.notifier_call = cpu_online_notifier,
+};
+
+static struct notifier_block cpu_pm_notifier = {
+	.notifier_call = cpu_state_notifier,
+};
+
+static struct generic_pm_domain *get_cpu_domain(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+	struct of_phandle_args pd_args;
+	int ret;
+
+	/* Make sure we are a domain consumer */
+	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
+				"#power-domain-cells", 0, &pd_args);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* Attach cpus only for this domain */
+	return of_genpd_get_from_provider(&pd_args);
+}
+
+int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
+{
+	int cpu;
+	int ret;
+	cpumask_var_t tmpmask;
+	struct generic_pm_domain *cpupd;
+
+	if (!genpd || !dn)
+		return -EINVAL;
+
+	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
+		return -ENOMEM;
+
+	/* CPU genpds have to operate in IRQ safe mode */
+	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
+
+	pm_genpd_init(genpd, NULL, false);
+	ret = of_genpd_add_provider_simple(dn, genpd);
+	if (ret)
+		return ret;
+
+	/* Only add those cpus to whom we are the domain provider */
+	for_each_online_cpu(cpu) {
+		cpupd = get_cpu_domain(cpu);
+
+		if (IS_ERR(cpupd))
+			continue;
+
+		if (genpd == cpupd) {
+			cpuidle_genpd_device_init(cpu);
+			cpumask_set_cpu(cpu, tmpmask);
+		}
+	}
+
+	if (cpumask_empty(tmpmask))
+		goto done;
+
+	/*
+	 * Not all cpus may be online at this point. Use the hotplug
+	 * notifier to be notified of when the cpu comes online, then
+	 * attach it to the domain.
+	 *
+	 * Register hotplug and cpu_pm notification once for all
+	 * domains.
+	 */
+	if (cpumask_empty(&cpus_handled)) {
+		cpu_pm_register_notifier(&cpu_pm_notifier);
+		register_cpu_notifier(&hotplug_notifier);
+	}
+
+	cpumask_copy(&cpus_handled, tmpmask);
+
+done:
+	free_cpumask_var(tmpmask);
+	return 0;
+}
+EXPORT_SYMBOL(pm_cpu_domain_init);
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index dc7cb53..fc97ad8 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
 					void *data);
 
 int genpd_dev_pm_attach(struct device *dev);
+
 #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
 static inline int __of_genpd_add_provider(struct device_node *np,
 					genpd_xlate_t xlate, void *data)
@@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
 static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
 #endif
 
+#ifdef CONFIG_PM_CPU_DOMAIN
+extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn);
+#else
+static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* _LINUX_PM_DOMAIN_H */
diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
index 7e01f78..55d49f6 100644
--- a/kernel/power/Kconfig
+++ b/kernel/power/Kconfig
@@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
 
 config CPU_PM
 	bool
+
+config PM_CPU_DOMAIN
+	def_bool y
+	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
+	help
+	  When cpuidle powers of the cpus in a domain, the domain can also be
+	  powered off.
+	  This config option allow for cpus to be registered with the domain
+	  provider specified in the DT and when the cpu is powered off, calls
+	  the runtime PM methods to do the reference counting. The last cpu
+	  going down powers the domain off as well.
+
-- 
2.1.4

[PATCH RFC 0/3] PM / Domains: Generic PM domains for cpus

[Lina Iyer]

This is an attempt to provide a generic PM domain for cpus, so as to allow the
cpu domain to be powered off, when all the cpus are in power down state during
cpuidle. The rationale behind the change is that newer SoCs can power down the
cpus for very short sleep times to save on leakage power. The domain which
usually has the cpus, a second level cache and some peripheral hardware is
powered by a rail that can also be turned off when the cpus are not in use.
Devices for each cpu, L2 and other related blocks could be attached to the
domain and when there are no uses of these devices (device idle) the domain
could also be powered off. Generic PM domains provides all the backend needed
for such an organization.

In the first 2 patches, I make genpd usable in atomic context as well. CPUIdle
runs with irqs disabled and therefore use of mutexes in the current genpd
implementation is a limitation. But, not all PM domains need to operate in irq
safe context, those that need to can specify explicitly the irq safe
requirement of the genpd at init. Devices and sub-domains that attach to an irq
safe genpd also have to be irq safe.

The third patch, adds a generic PM domain for the cpus. GenPD provider can be
specified in the DT and individual cpus that are part of the domain would be
the domain consumers. A new API pm_cpu_domain_init() has been introduced that
would initialize the genpd and attach cpus specified as consumers in the DT to
the domain. When the cpus enter their idle state cpu_pm notifications are sent
out for that cpu. The last cpu to send cpu_pm notification would trigger the
domain power_off callback and the first cpu to come up would trigger the genpd
power_on callback.  Generally, the power_off callback is where the caches are
flushed in preparation for a power down and the domain hardware configured to
power down when the cpu finishes execution. In the power_on callback, the
domain hardware is reset to a state to allow cpus to be active or entire idle
states individually.

A future addition to this feature, could be a new genpd governor for the
specific case of the cpu domain.  The governor may look up the time available
to sleep between the last cpu down and the first cpu up, in determining if it
would be more efficient to just keep the domain powered on.

This patch is based on Ulf's patch for simplifying domain power down states
[1], which removes a bunch of complexity in genpd, simplifying atomic genpd.

I have tested this on QCOM 8916, 8084 boards. The dependencies for those
restrict the addition of the platform code to the series. An example of changes
needed to add cpu domain could looks like this.

CPU as the domain consumer (only cpu0 is provided as the example here) - 

CPU0: cpu@0 {
        device_type = "cpu";
        compatible = "arm,cortex-a53", "arm,armv8";
        reg = <0x0>;
        enable-method = "qcom,arm-cortex-acc";
        qcom,acc = <&acc0>;
        next-level-cache = <&L2_0>;
        qcom,saw = <&saw0>;
        cpu-idle-states = <&CPU_SPC>;
        power-domains = <&saw_l2>;
};

Power controller for the L2 Cache is also the domain provider -

saw_l2: power-controller@B012000 {
        compatible = "qcom,msm8916-saw2-v3.0-l2";
        reg = <0xB012000 0x1000>, <0xB009000 0x1000>;
        #power-domain-cells = <0>;
};

The platform driver for the power-controller -

{
	...
	genpd.name = kstrdup("cpu-domain", GFP_KERNEL);
	genpd.power_on = cpu_pd_power_on;
	genpd.power_off = cpu_pd_power_off;
	ret = pm_cpu_domain_init(&genpd, pdev->dev.of_node);
	...
}

[1]. https://patches.linaro.org/49014/

Lina Iyer (3):
  PM / Domains: Allocate memory outside domain locks
  PM / Domains: Support atomic PM domains
  PM / Domains: Introduce generic PM domain for cpu domain

 drivers/base/power/Makefile     |   1 +
 drivers/base/power/cpu_domain.c | 187 +++++++++++++++++++++++++++++++++
 drivers/base/power/domain.c     | 225 +++++++++++++++++++++++++++++-----------
 include/linux/pm_domain.h       |  23 +++-
 kernel/power/Kconfig            |  12 +++
 5 files changed, 388 insertions(+), 60 deletions(-)
 create mode 100644 drivers/base/power/cpu_domain.c

-- 
2.1.4

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Lina Iyer]

In order for domains to be powered on/off in irq locked context, the
domain locks could either be a spinlock or a mutex, depending on the
domain. In preparation for atomic support, allocate domain data outside
the domain locks, so the allocation calls dont have to be context
sensitive.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 3b3367b..dfd7595 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 			   struct generic_pm_domain *subdomain)
 {
-	struct gpd_link *link;
+	struct gpd_link *link, *itr;
 	int ret = 0;
 
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
 
@@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		goto out;
 	}
 
-	list_for_each_entry(link, &genpd->master_links, master_node) {
-		if (link->slave == subdomain && link->master == genpd) {
+	list_for_each_entry(itr, &genpd->master_links, master_node) {
+		if (itr->slave == subdomain && itr->master == genpd) {
 			ret = -EINVAL;
 			goto out;
 		}
 	}
 
-	link = kzalloc(sizeof(*link), GFP_KERNEL);
-	if (!link) {
-		ret = -ENOMEM;
-		goto out;
-	}
 	link->master = genpd;
 	list_add_tail(&link->master_node, &genpd->master_links);
 	link->slave = subdomain;
@@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
+	if (!cpuidle_data)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
 		goto out;
 	}
-	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
-	if (!cpuidle_data) {
-		ret = -ENOMEM;
-		goto out;
-	}
+
 	cpuidle_drv = cpuidle_driver_ref();
 	if (!cpuidle_drv) {
 		ret = -ENODEV;
-- 
2.1.4

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

Power Domains currently support turning on/off only in process context.
This restricts the usage of PM domains to devices and domains that
could be powered on/off in irq disabled contexts as the mutexes used in
GenPD allows for cpu sleep while waiting for locks.

Genpd inherently provides support for devices, domain hierarchy and can
be used to represent cpu clusters like in ARM's big.Little, where, each
cpu cluster is in its domain, with supporting caches and other
peripheral hardware. Multiple such domains could be part of another
domain. Because mutexes are used to protect and synchronize domain
operations and cpu idle operations are inherently atomic, the use of
genpd is not possible for runtime suspend and resume of the pm domain.
Replacing the locks to spinlocks would allow cpu domain to be be powered
off to save power, when all the cpus are powered off.

However, not all domains can operate in irq-safe contexts and usually
would need to sleep during domain operations. So genpd has to support
both the cases, where the domain is or is not irq-safe. The irq-safe
attribute is therefore domain specific.

To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
while defining the domain. This determines if the domain should use a
spinlock instead of a mutex. Locking is abstracted through
genpd_lock_domain() and genpd_unlock_domain() functions that use the
flag to determine the locking to be used for this domain.

The restriction this imposes on the domain hierarchy is that subdomains
and all devices in the hierarchy also be irq-safe. Non irq-safe domains
may continue to have irq-safe devices, but not the other way around.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
 include/linux/pm_domain.h   |  11 ++-
 2 files changed, 164 insertions(+), 47 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index dfd7595..8b89d15 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -50,6 +50,71 @@
 static LIST_HEAD(gpd_list);
 static DEFINE_MUTEX(gpd_list_lock);
 
+static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->slock)
+{
+	unsigned long flags;
+
+	if (unlikely(subclass > 0))
+		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
+	else
+		spin_lock_irqsave(&genpd->slock, flags);
+
+	genpd->flags = flags;
+
+	return 0;
+}
+
+static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
+	__releases(&genpd->slock)
+{
+	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
+	return 0;
+}
+
+static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->mlock)
+{
+	if (unlikely(subclass > 0))
+		mutex_lock_nested(&genpd->mlock, subclass);
+	else
+		mutex_lock(&genpd->mlock);
+
+	return 0;
+}
+
+static inline int genpd_lock_domain_interruptible_irq(
+				struct generic_pm_domain *genpd)
+	__acquires(&genpd->mlock)
+{
+	return mutex_lock_interruptible(&genpd->mlock);
+}
+
+static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
+	__releases(&genpd->mlock)
+{
+	mutex_unlock(&genpd->mlock);
+	return 0;
+}
+
+#define genpd_lock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_irq(genpd, 0))
+
+#define genpd_lock_domain_nested(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
+			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
+
+#define genpd_unlock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
+			: genpd_unlock_domain_irq(genpd))
+
+#define genpd_lock_domain_interruptible(genpd)			\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_interruptible_irq(genpd))
+
 static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
 {
 	struct generic_pm_domain *genpd = NULL, *gpd;
@@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
 {
 	int ret;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
 		spin_unlock_irq(&dev->power.lock);
 
 		if (!IS_ERR(genpd)) {
-			mutex_lock(&genpd->lock);
+			genpd_lock_domain(genpd);
 			genpd->max_off_time_changed = true;
-			mutex_unlock(&genpd->lock);
+			genpd_unlock_domain(genpd);
 		}
 
 		dev = dev->parent;
@@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
 			return -EBUSY;
 
 		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
-		    || pdd->dev->power.irq_safe))
+			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
 			not_suspended++;
 	}
 
@@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
 
 	genpd = container_of(work, struct generic_pm_domain, power_off_work);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	pm_genpd_poweroff(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 }
 
 /**
@@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/*
-	 * We can't allow to power off the PM domain if it holds an irq_safe
-	 * device. That's beacuse we use mutexes to protect data while power
-	 * off and on the PM domain, thus we can't execute in atomic context.
-	 */
-	if (dev->power.irq_safe)
+	/* We can't allow to power off a domain that is also not irq safe. */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return -EBUSY;
 
 	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
@@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 		return ret;
 	}
 
-	mutex_lock(&genpd->lock);
+	/*
+	 * If power.irq_safe is set, this routine will be run with interrupts
+	 * off, so suspend only if the power domain is irq_safe.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
+		return 0;
+
+	genpd_lock_domain(genpd);
+
 	genpd->in_progress++;
 	pm_genpd_poweroff(genpd);
 	genpd->in_progress--;
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
@@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/* If power.irq_safe, the PM domain is never powered off. */
-	if (dev->power.irq_safe)
+	/*
+	 * If power.irq_safe and domain is not, then
+	 * the PM domain is never powered off.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return genpd_start_dev_no_timing(genpd, dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		return ret;
@@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
 	if (resume_needed(dev, genpd))
 		pm_runtime_resume(dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count++ == 0) {
 		genpd->suspended_count = 0;
 		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (genpd->suspend_power_off) {
 		pm_runtime_put_noidle(dev);
@@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
 
 	ret = pm_generic_prepare(dev);
 	if (ret) {
-		mutex_lock(&genpd->lock);
+		genpd_lock_domain(genpd);
 
 		if (--genpd->prepared_count == 0)
 			genpd->suspend_power_off = false;
 
-		mutex_unlock(&genpd->lock);
+		genpd_unlock_domain(genpd);
 		pm_runtime_enable(dev);
 	}
 
@@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
 	if (IS_ERR(genpd))
 		return;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	run_complete = !genpd->suspend_power_off;
 	if (--genpd->prepared_count == 0)
 		genpd->suspend_power_off = false;
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (run_complete) {
 		pm_generic_complete(dev);
@@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
 		return -EINVAL;
 
+	/* Devices in an IRQ safe PM Domain have to be irq safe too */
+	if (genpd->irq_safe && !dev->power.irq_safe) {
+		dev_warn(dev,
+			"Devices in an irq-safe domain have to be irq safe.\n");
+		return -EINVAL;
+	}
+
 	gpd_data = genpd_alloc_dev_data(dev, genpd, td);
 	if (IS_ERR(gpd_data))
 		return PTR_ERR(gpd_data);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		genpd_free_dev_data(dev, gpd_data);
@@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 	gpd_data = to_gpd_data(pdd);
 	dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 
 	list_del_init(&pdd->list_node);
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	genpd_free_dev_data(dev, gpd_data);
 
 	return 0;
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	dev_pm_qos_add_notifier(dev, &gpd_data->nb);
 
 	return ret;
@@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	/*
+	 * If the domain can be powered on/off in an irq safe
+	 * context, ensure that the subdomain can also be
+	 * powered on/off in that context.
+	 */
+	if (genpd->irq_safe && !subdomain->irq_safe) {
+		pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
+				subdomain->name, genpd->name);
+		return -EINVAL;
+	}
+
 	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
-	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+	genpd_lock_domain(genpd);
+	genpd_lock_domain_nested(subdomain);
 
 	if (genpd->status == GPD_STATE_POWER_OFF
 	    &&  subdomain->status != GPD_STATE_POWER_OFF) {
@@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		genpd_sd_counter_inc(genpd);
 
  out:
-	mutex_unlock(&subdomain->lock);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(subdomain);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	list_for_each_entry(link, &genpd->master_links, master_node) {
 		if (link->slave != subdomain)
 			continue;
 
-		mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+		genpd_lock_domain_nested(subdomain);
 
 		list_del(&link->master_node);
 		list_del(&link->slave_node);
@@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 		if (subdomain->status != GPD_STATE_POWER_OFF)
 			genpd_sd_counter_dec(genpd);
 
-		mutex_unlock(&subdomain->lock);
+		genpd_unlock_domain(subdomain);
 
 		ret = 0;
 		break;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	if (genpd->irq_safe) {
+		pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
+				genpd->name);
+		return -EINVAL;
+	}
+
 	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
 	if (!cpuidle_data)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
@@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	genpd_recalc_cpu_exit_latency(genpd);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 
  err:
@@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	if (IS_ERR_OR_NULL(genpd))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	cpuidle_data = genpd->cpuidle_data;
 	if (!cpuidle_data) {
@@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	kfree(cpuidle_data);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
 	return cb ? cb(dev) : 0;
 }
 
+static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
+{
+	if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
+		spin_lock_init(&genpd->slock);
+		genpd->irq_safe = true;
+	} else {
+		mutex_init(&genpd->mlock);
+		genpd->irq_safe = false;
+	}
+}
+
 /**
  * pm_genpd_init - Initialize a generic I/O PM domain object.
  * @genpd: PM domain object to initialize.
@@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
 	INIT_LIST_HEAD(&genpd->master_links);
 	INIT_LIST_HEAD(&genpd->slave_links);
 	INIT_LIST_HEAD(&genpd->dev_list);
-	mutex_init(&genpd->lock);
+	genpd_lock_domain_init(genpd);
 	genpd->gov = gov;
 	INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
 	genpd->in_progress = 0;
@@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	struct gpd_link *link;
 	int ret;
 
-	ret = mutex_lock_interruptible(&genpd->lock);
+	ret = genpd_lock_domain_interruptible(genpd);
 	if (ret)
 		return -ERESTARTSYS;
 
@@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	}
 
 	list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
-		kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
+		kobj_path = kobject_get_path(&pm_data->dev->kobj,
+				genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
 		if (kobj_path == NULL)
 			continue;
 
@@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 
 	seq_puts(s, "\n");
 exit:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index b2725e6..dc7cb53 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -16,9 +16,11 @@
 #include <linux/of.h>
 #include <linux/notifier.h>
 #include <linux/cpuidle.h>
+#include <linux/spinlock.h>
 
 /* Defines used for the flags field in the struct generic_pm_domain */
 #define GENPD_FLAG_PM_CLK	(1U << 0) /* PM domain uses PM clk */
+#define GENPD_FLAG_IRQ_SAFE	(1U << 1) /* PM domain operates in atomic */
 
 enum gpd_status {
 	GPD_STATE_ACTIVE = 0,	/* PM domain is active */
@@ -49,7 +51,6 @@ struct generic_pm_domain {
 	struct list_head master_links;	/* Links with PM domain as a master */
 	struct list_head slave_links;	/* Links with PM domain as a slave */
 	struct list_head dev_list;	/* List of devices */
-	struct mutex lock;
 	struct dev_power_governor *gov;
 	struct work_struct power_off_work;
 	const char *name;
@@ -74,6 +75,14 @@ struct generic_pm_domain {
 	void (*detach_dev)(struct generic_pm_domain *domain,
 			   struct device *dev);
 	unsigned int flags;		/* Bit field of configs for genpd */
+	bool irq_safe;
+	union {
+		struct mutex mlock;
+		struct {
+			spinlock_t slock;
+			unsigned long lock_flags;
+		};
+	};
 };
 
 static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
-- 
2.1.4

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Lina Iyer]

In order for domains to be powered on/off in irq locked context, the
domain locks could either be a spinlock or a mutex, depending on the
domain. In preparation for atomic support, allocate domain data outside
the domain locks, so the allocation calls dont have to be context
sensitive.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 3b3367b..dfd7595 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 			   struct generic_pm_domain *subdomain)
 {
-	struct gpd_link *link;
+	struct gpd_link *link, *itr;
 	int ret = 0;
 
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
 
@@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		goto out;
 	}
 
-	list_for_each_entry(link, &genpd->master_links, master_node) {
-		if (link->slave == subdomain && link->master == genpd) {
+	list_for_each_entry(itr, &genpd->master_links, master_node) {
+		if (itr->slave == subdomain && itr->master == genpd) {
 			ret = -EINVAL;
 			goto out;
 		}
 	}
 
-	link = kzalloc(sizeof(*link), GFP_KERNEL);
-	if (!link) {
-		ret = -ENOMEM;
-		goto out;
-	}
 	link->master = genpd;
 	list_add_tail(&link->master_node, &genpd->master_links);
 	link->slave = subdomain;
@@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
+	if (!cpuidle_data)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
 		goto out;
 	}
-	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
-	if (!cpuidle_data) {
-		ret = -ENOMEM;
-		goto out;
-	}
+
 	cpuidle_drv = cpuidle_driver_ref();
 	if (!cpuidle_drv) {
 		ret = -ENODEV;
-- 
2.1.4

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

Generally cpus are grouped under a power domain in a SoC. When all cpus
in the domain are in their power off state, the cpu domain can also be
powered off. Genpd provides the framework for defining cpus as devices
that are part of a cpu domain.

Introduce support for defining and adding a generic power domain for the
cpus based on the DT specification of power domain providers and
consumers.  SoC's that have the cpu domain defined in their DT, can
setup a genpd with a name and the power_on/power_off callbacks. Calling
pm_cpu_domain_init() will register the genpd and attach the cpus for
this domain with the genpd.

CPU_PM notifications for are used to pm_runtime_get_sync() and
pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
last cpu going down would call the genpd->power_off(). Correspondingly,
the first cpu up would call the genpd->power_on() callback before
resuming from idle.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/Makefile     |   1 +
 drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h       |  12 +++
 kernel/power/Kconfig            |  12 +++
 4 files changed, 212 insertions(+)
 create mode 100644 drivers/base/power/cpu_domain.c

diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
index 1cb8544..debfc74 100644
--- a/drivers/base/power/Makefile
+++ b/drivers/base/power/Makefile
@@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
 obj-$(CONFIG_PM_OPP)	+= opp.o
 obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
 obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
+obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
 
 ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
new file mode 100644
index 0000000..ee90094
--- /dev/null
+++ b/drivers/base/power/cpu_domain.c
@@ -0,0 +1,187 @@
+/*
+ * Generic CPU domain runtime power on/off support
+ *
+ * Copyright (C) 2015 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpu_pm.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/pm_domain.h>
+#include <linux/pm_runtime.h>
+
+static struct cpumask cpus_handled;
+
+static void do_cpu(void *unused)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	pm_runtime_get_sync(dev);
+}
+
+static int cpuidle_genpd_device_init(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	/*
+	 * CPU device have to be irq safe for use with cpuidle, which runs
+	 * with irqs disabled.
+	 */
+	pm_runtime_irq_safe(dev);
+	pm_runtime_enable(dev);
+
+	genpd_dev_pm_attach(dev);
+
+	/*
+	 * Execute the below on 'that' cpu to ensure that the reference
+	 * counting is correct. Its possible that while this code is
+	 * executed, the cpu may be in idle but we may incorrectly
+	 * increment the usage. By executing the do_cpu on 'that' cpu,
+	 * we can ensure that the cpu and the usage count are matched.
+	 */
+	return smp_call_function_single(cpu, do_cpu, NULL, true);
+}
+
+static int cpu_state_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_PM_ENTER:
+		pm_runtime_put_sync(dev);
+		break;
+
+	case CPU_PM_ENTER_FAILED:
+	case CPU_PM_EXIT:
+		pm_runtime_get_sync(dev);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static int cpu_online_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = (unsigned long)hcpu;
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_STARTING:
+	case CPU_STARTING_FROZEN:
+		/*
+		 * Attach the cpu to its domain if the cpu is coming up
+		 * for the first time.
+		 * Called from the cpu that is coming up.
+		 */
+		if (!genpd_dev_pm_attach(dev))
+			do_cpu(NULL);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block hotplug_notifier = {
+	.notifier_call = cpu_online_notifier,
+};
+
+static struct notifier_block cpu_pm_notifier = {
+	.notifier_call = cpu_state_notifier,
+};
+
+static struct generic_pm_domain *get_cpu_domain(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+	struct of_phandle_args pd_args;
+	int ret;
+
+	/* Make sure we are a domain consumer */
+	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
+				"#power-domain-cells", 0, &pd_args);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* Attach cpus only for this domain */
+	return of_genpd_get_from_provider(&pd_args);
+}
+
+int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
+{
+	int cpu;
+	int ret;
+	cpumask_var_t tmpmask;
+	struct generic_pm_domain *cpupd;
+
+	if (!genpd || !dn)
+		return -EINVAL;
+
+	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
+		return -ENOMEM;
+
+	/* CPU genpds have to operate in IRQ safe mode */
+	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
+
+	pm_genpd_init(genpd, NULL, false);
+	ret = of_genpd_add_provider_simple(dn, genpd);
+	if (ret)
+		return ret;
+
+	/* Only add those cpus to whom we are the domain provider */
+	for_each_online_cpu(cpu) {
+		cpupd = get_cpu_domain(cpu);
+
+		if (IS_ERR(cpupd))
+			continue;
+
+		if (genpd == cpupd) {
+			cpuidle_genpd_device_init(cpu);
+			cpumask_set_cpu(cpu, tmpmask);
+		}
+	}
+
+	if (cpumask_empty(tmpmask))
+		goto done;
+
+	/*
+	 * Not all cpus may be online at this point. Use the hotplug
+	 * notifier to be notified of when the cpu comes online, then
+	 * attach it to the domain.
+	 *
+	 * Register hotplug and cpu_pm notification once for all
+	 * domains.
+	 */
+	if (cpumask_empty(&cpus_handled)) {
+		cpu_pm_register_notifier(&cpu_pm_notifier);
+		register_cpu_notifier(&hotplug_notifier);
+	}
+
+	cpumask_copy(&cpus_handled, tmpmask);
+
+done:
+	free_cpumask_var(tmpmask);
+	return 0;
+}
+EXPORT_SYMBOL(pm_cpu_domain_init);
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index dc7cb53..fc97ad8 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
 					void *data);
 
 int genpd_dev_pm_attach(struct device *dev);
+
 #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
 static inline int __of_genpd_add_provider(struct device_node *np,
 					genpd_xlate_t xlate, void *data)
@@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
 static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
 #endif
 
+#ifdef CONFIG_PM_CPU_DOMAIN
+extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn);
+#else
+static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* _LINUX_PM_DOMAIN_H */
diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
index 7e01f78..55d49f6 100644
--- a/kernel/power/Kconfig
+++ b/kernel/power/Kconfig
@@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
 
 config CPU_PM
 	bool
+
+config PM_CPU_DOMAIN
+	def_bool y
+	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
+	help
+	  When cpuidle powers of the cpus in a domain, the domain can also be
+	  powered off.
+	  This config option allow for cpus to be registered with the domain
+	  provider specified in the DT and when the cpu is powered off, calls
+	  the runtime PM methods to do the reference counting. The last cpu
+	  going down powers the domain off as well.
+
-- 
2.1.4

[PATCH RFC 0/3] PM / Domains: Generic PM domains for cpus

[Lina Iyer]

This is an attempt to provide a generic PM domain for cpus, so as to allow the
cpu domain to be powered off, when all the cpus are in power down state during
cpuidle. The rationale behind the change is that newer SoCs can power down the
cpus for very short sleep times to save on leakage power. The domain which
usually has the cpus, a second level cache and some peripheral hardware is
powered by a rail that can also be turned off when the cpus are not in use.
Devices for each cpu, L2 and other related blocks could be attached to the
domain and when there are no uses of these devices (device idle) the domain
could also be powered off. Generic PM domains provides all the backend needed
for such an organization.

In the first 2 patches, I make genpd usable in atomic context as well. CPUIdle
runs with irqs disabled and therefore use of mutexes in the current genpd
implementation is a limitation. But, not all PM domains need to operate in irq
safe context, those that need to can specify explicitly the irq safe
requirement of the genpd at init. Devices and sub-domains that attach to an irq
safe genpd also have to be irq safe.

The third patch, adds a generic PM domain for the cpus. GenPD provider can be
specified in the DT and individual cpus that are part of the domain would be
the domain consumers. A new API pm_cpu_domain_init() has been introduced that
would initialize the genpd and attach cpus specified as consumers in the DT to
the domain. When the cpus enter their idle state cpu_pm notifications are sent
out for that cpu. The last cpu to send cpu_pm notification would trigger the
domain power_off callback and the first cpu to come up would trigger the genpd
power_on callback.  Generally, the power_off callback is where the caches are
flushed in preparation for a power down and the domain hardware configured to
power down when the cpu finishes execution. In the power_on callback, the
domain hardware is reset to a state to allow cpus to be active or entire idle
states individually.

A future addition to this feature, could be a new genpd governor for the
specific case of the cpu domain.  The governor may look up the time available
to sleep between the last cpu down and the first cpu up, in determining if it
would be more efficient to just keep the domain powered on.

This patch is based on Ulf's patch for simplifying domain power down states
[1], which removes a bunch of complexity in genpd, simplifying atomic genpd.

I have tested this on QCOM 8916, 8084 boards. The dependencies for those
restrict the addition of the platform code to the series. An example of changes
needed to add cpu domain could looks like this.

CPU as the domain consumer (only cpu0 is provided as the example here) - 

CPU0: cpu at 0 {
        device_type = "cpu";
        compatible = "arm,cortex-a53", "arm,armv8";
        reg = <0x0>;
        enable-method = "qcom,arm-cortex-acc";
        qcom,acc = <&acc0>;
        next-level-cache = <&L2_0>;
        qcom,saw = <&saw0>;
        cpu-idle-states = <&CPU_SPC>;
        power-domains = <&saw_l2>;
};

Power controller for the L2 Cache is also the domain provider -

saw_l2: power-controller at B012000 {
        compatible = "qcom,msm8916-saw2-v3.0-l2";
        reg = <0xB012000 0x1000>, <0xB009000 0x1000>;
        #power-domain-cells = <0>;
};

The platform driver for the power-controller -

{
	...
	genpd.name = kstrdup("cpu-domain", GFP_KERNEL);
	genpd.power_on = cpu_pd_power_on;
	genpd.power_off = cpu_pd_power_off;
	ret = pm_cpu_domain_init(&genpd, pdev->dev.of_node);
	...
}

[1]. https://patches.linaro.org/49014/

Lina Iyer (3):
  PM / Domains: Allocate memory outside domain locks
  PM / Domains: Support atomic PM domains
  PM / Domains: Introduce generic PM domain for cpu domain

 drivers/base/power/Makefile     |   1 +
 drivers/base/power/cpu_domain.c | 187 +++++++++++++++++++++++++++++++++
 drivers/base/power/domain.c     | 225 +++++++++++++++++++++++++++++-----------
 include/linux/pm_domain.h       |  23 +++-
 kernel/power/Kconfig            |  12 +++
 5 files changed, 388 insertions(+), 60 deletions(-)
 create mode 100644 drivers/base/power/cpu_domain.c

-- 
2.1.4

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Lina Iyer]

In order for domains to be powered on/off in irq locked context, the
domain locks could either be a spinlock or a mutex, depending on the
domain. In preparation for atomic support, allocate domain data outside
the domain locks, so the allocation calls dont have to be context
sensitive.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 3b3367b..dfd7595 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 			   struct generic_pm_domain *subdomain)
 {
-	struct gpd_link *link;
+	struct gpd_link *link, *itr;
 	int ret = 0;
 
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
 
@@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		goto out;
 	}
 
-	list_for_each_entry(link, &genpd->master_links, master_node) {
-		if (link->slave == subdomain && link->master == genpd) {
+	list_for_each_entry(itr, &genpd->master_links, master_node) {
+		if (itr->slave == subdomain && itr->master == genpd) {
 			ret = -EINVAL;
 			goto out;
 		}
 	}
 
-	link = kzalloc(sizeof(*link), GFP_KERNEL);
-	if (!link) {
-		ret = -ENOMEM;
-		goto out;
-	}
 	link->master = genpd;
 	list_add_tail(&link->master_node, &genpd->master_links);
 	link->slave = subdomain;
@@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
+	if (!cpuidle_data)
+		return -ENOMEM;
+
 	mutex_lock(&genpd->lock);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
 		goto out;
 	}
-	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
-	if (!cpuidle_data) {
-		ret = -ENOMEM;
-		goto out;
-	}
+
 	cpuidle_drv = cpuidle_driver_ref();
 	if (!cpuidle_drv) {
 		ret = -ENODEV;
-- 
2.1.4

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

Power Domains currently support turning on/off only in process context.
This restricts the usage of PM domains to devices and domains that
could be powered on/off in irq disabled contexts as the mutexes used in
GenPD allows for cpu sleep while waiting for locks.

Genpd inherently provides support for devices, domain hierarchy and can
be used to represent cpu clusters like in ARM's big.Little, where, each
cpu cluster is in its domain, with supporting caches and other
peripheral hardware. Multiple such domains could be part of another
domain. Because mutexes are used to protect and synchronize domain
operations and cpu idle operations are inherently atomic, the use of
genpd is not possible for runtime suspend and resume of the pm domain.
Replacing the locks to spinlocks would allow cpu domain to be be powered
off to save power, when all the cpus are powered off.

However, not all domains can operate in irq-safe contexts and usually
would need to sleep during domain operations. So genpd has to support
both the cases, where the domain is or is not irq-safe. The irq-safe
attribute is therefore domain specific.

To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
while defining the domain. This determines if the domain should use a
spinlock instead of a mutex. Locking is abstracted through
genpd_lock_domain() and genpd_unlock_domain() functions that use the
flag to determine the locking to be used for this domain.

The restriction this imposes on the domain hierarchy is that subdomains
and all devices in the hierarchy also be irq-safe. Non irq-safe domains
may continue to have irq-safe devices, but not the other way around.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
 include/linux/pm_domain.h   |  11 ++-
 2 files changed, 164 insertions(+), 47 deletions(-)

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index dfd7595..8b89d15 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -50,6 +50,71 @@
 static LIST_HEAD(gpd_list);
 static DEFINE_MUTEX(gpd_list_lock);
 
+static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->slock)
+{
+	unsigned long flags;
+
+	if (unlikely(subclass > 0))
+		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
+	else
+		spin_lock_irqsave(&genpd->slock, flags);
+
+	genpd->flags = flags;
+
+	return 0;
+}
+
+static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
+	__releases(&genpd->slock)
+{
+	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
+	return 0;
+}
+
+static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
+					unsigned int subclass)
+	__acquires(&genpd->mlock)
+{
+	if (unlikely(subclass > 0))
+		mutex_lock_nested(&genpd->mlock, subclass);
+	else
+		mutex_lock(&genpd->mlock);
+
+	return 0;
+}
+
+static inline int genpd_lock_domain_interruptible_irq(
+				struct generic_pm_domain *genpd)
+	__acquires(&genpd->mlock)
+{
+	return mutex_lock_interruptible(&genpd->mlock);
+}
+
+static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
+	__releases(&genpd->mlock)
+{
+	mutex_unlock(&genpd->mlock);
+	return 0;
+}
+
+#define genpd_lock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_irq(genpd, 0))
+
+#define genpd_lock_domain_nested(genpd)				\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
+			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
+
+#define genpd_unlock_domain(genpd)				\
+	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
+			: genpd_unlock_domain_irq(genpd))
+
+#define genpd_lock_domain_interruptible(genpd)			\
+	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
+			: genpd_lock_domain_interruptible_irq(genpd))
+
 static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
 {
 	struct generic_pm_domain *genpd = NULL, *gpd;
@@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
 {
 	int ret;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
 		spin_unlock_irq(&dev->power.lock);
 
 		if (!IS_ERR(genpd)) {
-			mutex_lock(&genpd->lock);
+			genpd_lock_domain(genpd);
 			genpd->max_off_time_changed = true;
-			mutex_unlock(&genpd->lock);
+			genpd_unlock_domain(genpd);
 		}
 
 		dev = dev->parent;
@@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
 			return -EBUSY;
 
 		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
-		    || pdd->dev->power.irq_safe))
+			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
 			not_suspended++;
 	}
 
@@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
 
 	genpd = container_of(work, struct generic_pm_domain, power_off_work);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	pm_genpd_poweroff(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 }
 
 /**
@@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/*
-	 * We can't allow to power off the PM domain if it holds an irq_safe
-	 * device. That's beacuse we use mutexes to protect data while power
-	 * off and on the PM domain, thus we can't execute in atomic context.
-	 */
-	if (dev->power.irq_safe)
+	/* We can't allow to power off a domain that is also not irq safe. */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return -EBUSY;
 
 	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
@@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
 		return ret;
 	}
 
-	mutex_lock(&genpd->lock);
+	/*
+	 * If power.irq_safe is set, this routine will be run with interrupts
+	 * off, so suspend only if the power domain is irq_safe.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
+		return 0;
+
+	genpd_lock_domain(genpd);
+
 	genpd->in_progress++;
 	pm_genpd_poweroff(genpd);
 	genpd->in_progress--;
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
@@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
 	if (IS_ERR(genpd))
 		return -EINVAL;
 
-	/* If power.irq_safe, the PM domain is never powered off. */
-	if (dev->power.irq_safe)
+	/*
+	 * If power.irq_safe and domain is not, then
+	 * the PM domain is never powered off.
+	 */
+	if (dev->power.irq_safe && !genpd->irq_safe)
 		return genpd_start_dev_no_timing(genpd, dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 	ret = __pm_genpd_poweron(genpd);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		return ret;
@@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
 	if (resume_needed(dev, genpd))
 		pm_runtime_resume(dev);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count++ == 0) {
 		genpd->suspended_count = 0;
 		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (genpd->suspend_power_off) {
 		pm_runtime_put_noidle(dev);
@@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
 
 	ret = pm_generic_prepare(dev);
 	if (ret) {
-		mutex_lock(&genpd->lock);
+		genpd_lock_domain(genpd);
 
 		if (--genpd->prepared_count == 0)
 			genpd->suspend_power_off = false;
 
-		mutex_unlock(&genpd->lock);
+		genpd_unlock_domain(genpd);
 		pm_runtime_enable(dev);
 	}
 
@@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
 	if (IS_ERR(genpd))
 		return;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	run_complete = !genpd->suspend_power_off;
 	if (--genpd->prepared_count == 0)
 		genpd->suspend_power_off = false;
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (run_complete) {
 		pm_generic_complete(dev);
@@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
 		return -EINVAL;
 
+	/* Devices in an IRQ safe PM Domain have to be irq safe too */
+	if (genpd->irq_safe && !dev->power.irq_safe) {
+		dev_warn(dev,
+			"Devices in an irq-safe domain have to be irq safe.\n");
+		return -EINVAL;
+	}
+
 	gpd_data = genpd_alloc_dev_data(dev, genpd, td);
 	if (IS_ERR(gpd_data))
 		return PTR_ERR(gpd_data);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
 	list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	if (ret)
 		genpd_free_dev_data(dev, gpd_data);
@@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 	gpd_data = to_gpd_data(pdd);
 	dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->prepared_count > 0) {
 		ret = -EAGAIN;
@@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
 
 	list_del_init(&pdd->list_node);
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	genpd_free_dev_data(dev, gpd_data);
 
 	return 0;
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	dev_pm_qos_add_notifier(dev, &gpd_data->nb);
 
 	return ret;
@@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 	    || genpd == subdomain)
 		return -EINVAL;
 
+	/*
+	 * If the domain can be powered on/off in an irq safe
+	 * context, ensure that the subdomain can also be
+	 * powered on/off in that context.
+	 */
+	if (genpd->irq_safe && !subdomain->irq_safe) {
+		pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
+				subdomain->name, genpd->name);
+		return -EINVAL;
+	}
+
 	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
-	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+	genpd_lock_domain(genpd);
+	genpd_lock_domain_nested(subdomain);
 
 	if (genpd->status == GPD_STATE_POWER_OFF
 	    &&  subdomain->status != GPD_STATE_POWER_OFF) {
@@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
 		genpd_sd_counter_inc(genpd);
 
  out:
-	mutex_unlock(&subdomain->lock);
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(subdomain);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	list_for_each_entry(link, &genpd->master_links, master_node) {
 		if (link->slave != subdomain)
 			continue;
 
-		mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
+		genpd_lock_domain_nested(subdomain);
 
 		list_del(&link->master_node);
 		list_del(&link->slave_node);
@@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
 		if (subdomain->status != GPD_STATE_POWER_OFF)
 			genpd_sd_counter_dec(genpd);
 
-		mutex_unlock(&subdomain->lock);
+		genpd_unlock_domain(subdomain);
 
 		ret = 0;
 		break;
 	}
 
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return ret;
 }
@@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	if (IS_ERR_OR_NULL(genpd) || state < 0)
 		return -EINVAL;
 
+	if (genpd->irq_safe) {
+		pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
+				genpd->name);
+		return -EINVAL;
+	}
+
 	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
 	if (!cpuidle_data)
 		return -ENOMEM;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	if (genpd->cpuidle_data) {
 		ret = -EEXIST;
@@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
 	genpd_recalc_cpu_exit_latency(genpd);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 
  err:
@@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	if (IS_ERR_OR_NULL(genpd))
 		return -EINVAL;
 
-	mutex_lock(&genpd->lock);
+	genpd_lock_domain(genpd);
 
 	cpuidle_data = genpd->cpuidle_data;
 	if (!cpuidle_data) {
@@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
 	kfree(cpuidle_data);
 
  out:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 	return ret;
 }
 
@@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
 	return cb ? cb(dev) : 0;
 }
 
+static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
+{
+	if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
+		spin_lock_init(&genpd->slock);
+		genpd->irq_safe = true;
+	} else {
+		mutex_init(&genpd->mlock);
+		genpd->irq_safe = false;
+	}
+}
+
 /**
  * pm_genpd_init - Initialize a generic I/O PM domain object.
  * @genpd: PM domain object to initialize.
@@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
 	INIT_LIST_HEAD(&genpd->master_links);
 	INIT_LIST_HEAD(&genpd->slave_links);
 	INIT_LIST_HEAD(&genpd->dev_list);
-	mutex_init(&genpd->lock);
+	genpd_lock_domain_init(genpd);
 	genpd->gov = gov;
 	INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
 	genpd->in_progress = 0;
@@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	struct gpd_link *link;
 	int ret;
 
-	ret = mutex_lock_interruptible(&genpd->lock);
+	ret = genpd_lock_domain_interruptible(genpd);
 	if (ret)
 		return -ERESTARTSYS;
 
@@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
 	}
 
 	list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
-		kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
+		kobj_path = kobject_get_path(&pm_data->dev->kobj,
+				genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
 		if (kobj_path == NULL)
 			continue;
 
@@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
 
 	seq_puts(s, "\n");
 exit:
-	mutex_unlock(&genpd->lock);
+	genpd_unlock_domain(genpd);
 
 	return 0;
 }
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index b2725e6..dc7cb53 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -16,9 +16,11 @@
 #include <linux/of.h>
 #include <linux/notifier.h>
 #include <linux/cpuidle.h>
+#include <linux/spinlock.h>
 
 /* Defines used for the flags field in the struct generic_pm_domain */
 #define GENPD_FLAG_PM_CLK	(1U << 0) /* PM domain uses PM clk */
+#define GENPD_FLAG_IRQ_SAFE	(1U << 1) /* PM domain operates in atomic */
 
 enum gpd_status {
 	GPD_STATE_ACTIVE = 0,	/* PM domain is active */
@@ -49,7 +51,6 @@ struct generic_pm_domain {
 	struct list_head master_links;	/* Links with PM domain as a master */
 	struct list_head slave_links;	/* Links with PM domain as a slave */
 	struct list_head dev_list;	/* List of devices */
-	struct mutex lock;
 	struct dev_power_governor *gov;
 	struct work_struct power_off_work;
 	const char *name;
@@ -74,6 +75,14 @@ struct generic_pm_domain {
 	void (*detach_dev)(struct generic_pm_domain *domain,
 			   struct device *dev);
 	unsigned int flags;		/* Bit field of configs for genpd */
+	bool irq_safe;
+	union {
+		struct mutex mlock;
+		struct {
+			spinlock_t slock;
+			unsigned long lock_flags;
+		};
+	};
 };
 
 static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
-- 
2.1.4

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

Generally cpus are grouped under a power domain in a SoC. When all cpus
in the domain are in their power off state, the cpu domain can also be
powered off. Genpd provides the framework for defining cpus as devices
that are part of a cpu domain.

Introduce support for defining and adding a generic power domain for the
cpus based on the DT specification of power domain providers and
consumers.  SoC's that have the cpu domain defined in their DT, can
setup a genpd with a name and the power_on/power_off callbacks. Calling
pm_cpu_domain_init() will register the genpd and attach the cpus for
this domain with the genpd.

CPU_PM notifications for are used to pm_runtime_get_sync() and
pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
last cpu going down would call the genpd->power_off(). Correspondingly,
the first cpu up would call the genpd->power_on() callback before
resuming from idle.

Cc: Ulf Hansson <ulf.hansson@linaro.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Kevin Hilman <khilman@linaro.org>
Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
---
 drivers/base/power/Makefile     |   1 +
 drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_domain.h       |  12 +++
 kernel/power/Kconfig            |  12 +++
 4 files changed, 212 insertions(+)
 create mode 100644 drivers/base/power/cpu_domain.c

diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
index 1cb8544..debfc74 100644
--- a/drivers/base/power/Makefile
+++ b/drivers/base/power/Makefile
@@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
 obj-$(CONFIG_PM_OPP)	+= opp.o
 obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
 obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
+obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
 
 ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
new file mode 100644
index 0000000..ee90094
--- /dev/null
+++ b/drivers/base/power/cpu_domain.c
@@ -0,0 +1,187 @@
+/*
+ * Generic CPU domain runtime power on/off support
+ *
+ * Copyright (C) 2015 Linaro Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpu_pm.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/pm_domain.h>
+#include <linux/pm_runtime.h>
+
+static struct cpumask cpus_handled;
+
+static void do_cpu(void *unused)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	pm_runtime_get_sync(dev);
+}
+
+static int cpuidle_genpd_device_init(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+
+	/*
+	 * CPU device have to be irq safe for use with cpuidle, which runs
+	 * with irqs disabled.
+	 */
+	pm_runtime_irq_safe(dev);
+	pm_runtime_enable(dev);
+
+	genpd_dev_pm_attach(dev);
+
+	/*
+	 * Execute the below on 'that' cpu to ensure that the reference
+	 * counting is correct. Its possible that while this code is
+	 * executed, the cpu may be in idle but we may incorrectly
+	 * increment the usage. By executing the do_cpu on 'that' cpu,
+	 * we can ensure that the cpu and the usage count are matched.
+	 */
+	return smp_call_function_single(cpu, do_cpu, NULL, true);
+}
+
+static int cpu_state_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = smp_processor_id();
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_PM_ENTER:
+		pm_runtime_put_sync(dev);
+		break;
+
+	case CPU_PM_ENTER_FAILED:
+	case CPU_PM_EXIT:
+		pm_runtime_get_sync(dev);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static int cpu_online_notifier(struct notifier_block *n,
+			unsigned long action, void *hcpu)
+{
+	int cpu = (unsigned long)hcpu;
+	struct device *dev = get_cpu_device(cpu);
+
+	if (!cpumask_test_cpu(cpu, &cpus_handled))
+		return NOTIFY_DONE;
+
+	switch (action) {
+	case CPU_STARTING:
+	case CPU_STARTING_FROZEN:
+		/*
+		 * Attach the cpu to its domain if the cpu is coming up
+		 * for the first time.
+		 * Called from the cpu that is coming up.
+		 */
+		if (!genpd_dev_pm_attach(dev))
+			do_cpu(NULL);
+		break;
+
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block hotplug_notifier = {
+	.notifier_call = cpu_online_notifier,
+};
+
+static struct notifier_block cpu_pm_notifier = {
+	.notifier_call = cpu_state_notifier,
+};
+
+static struct generic_pm_domain *get_cpu_domain(int cpu)
+{
+	struct device *dev = get_cpu_device(cpu);
+	struct of_phandle_args pd_args;
+	int ret;
+
+	/* Make sure we are a domain consumer */
+	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
+				"#power-domain-cells", 0, &pd_args);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* Attach cpus only for this domain */
+	return of_genpd_get_from_provider(&pd_args);
+}
+
+int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
+{
+	int cpu;
+	int ret;
+	cpumask_var_t tmpmask;
+	struct generic_pm_domain *cpupd;
+
+	if (!genpd || !dn)
+		return -EINVAL;
+
+	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
+		return -ENOMEM;
+
+	/* CPU genpds have to operate in IRQ safe mode */
+	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
+
+	pm_genpd_init(genpd, NULL, false);
+	ret = of_genpd_add_provider_simple(dn, genpd);
+	if (ret)
+		return ret;
+
+	/* Only add those cpus to whom we are the domain provider */
+	for_each_online_cpu(cpu) {
+		cpupd = get_cpu_domain(cpu);
+
+		if (IS_ERR(cpupd))
+			continue;
+
+		if (genpd == cpupd) {
+			cpuidle_genpd_device_init(cpu);
+			cpumask_set_cpu(cpu, tmpmask);
+		}
+	}
+
+	if (cpumask_empty(tmpmask))
+		goto done;
+
+	/*
+	 * Not all cpus may be online at this point. Use the hotplug
+	 * notifier to be notified of when the cpu comes online, then
+	 * attach it to the domain.
+	 *
+	 * Register hotplug and cpu_pm notification once for all
+	 * domains.
+	 */
+	if (cpumask_empty(&cpus_handled)) {
+		cpu_pm_register_notifier(&cpu_pm_notifier);
+		register_cpu_notifier(&hotplug_notifier);
+	}
+
+	cpumask_copy(&cpus_handled, tmpmask);
+
+done:
+	free_cpumask_var(tmpmask);
+	return 0;
+}
+EXPORT_SYMBOL(pm_cpu_domain_init);
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index dc7cb53..fc97ad8 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
 					void *data);
 
 int genpd_dev_pm_attach(struct device *dev);
+
 #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
 static inline int __of_genpd_add_provider(struct device_node *np,
 					genpd_xlate_t xlate, void *data)
@@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
 static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
 #endif
 
+#ifdef CONFIG_PM_CPU_DOMAIN
+extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn);
+#else
+static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
+			struct device_node *dn)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* _LINUX_PM_DOMAIN_H */
diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
index 7e01f78..55d49f6 100644
--- a/kernel/power/Kconfig
+++ b/kernel/power/Kconfig
@@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
 
 config CPU_PM
 	bool
+
+config PM_CPU_DOMAIN
+	def_bool y
+	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
+	help
+	  When cpuidle powers of the cpus in a domain, the domain can also be
+	  powered off.
+	  This config option allow for cpus to be registered with the domain
+	  provider specified in the DT and when the cpu is powered off, calls
+	  the runtime PM methods to do the reference counting. The last cpu
+	  going down powers the domain off as well.
+
-- 
2.1.4

Re: [PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> In order for domains to be powered on/off in irq locked context, the
> domain locks could either be a spinlock or a mutex, depending on the
> domain. In preparation for atomic support, allocate domain data outside
> the domain locks, so the allocation calls dont have to be context
> sensitive.
> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 25 ++++++++++++-------------
>  1 file changed, 12 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index 3b3367b..dfd7595 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>  int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>  			   struct generic_pm_domain *subdomain)
>  {
> -	struct gpd_link *link;
> +	struct gpd_link *link, *itr;
>  	int ret = 0;
>  
>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
>  	    || genpd == subdomain)
>  		return -EINVAL;
>  
> +	link = kzalloc(sizeof(*link), GFP_KERNEL);
> +	if (!link)
> +		return -ENOMEM;
> +
>  	mutex_lock(&genpd->lock);
>  	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>  
> @@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>  		goto out;
>  	}
>  
> -	list_for_each_entry(link, &genpd->master_links, master_node) {
> -		if (link->slave == subdomain && link->master == genpd) {
> +	list_for_each_entry(itr, &genpd->master_links, master_node) {
> +		if (itr->slave == subdomain && itr->master == genpd) {
>  			ret = -EINVAL;
>  			goto out;
>  		}
>  	}
>  
> -	link = kzalloc(sizeof(*link), GFP_KERNEL);
> -	if (!link) {
> -		ret = -ENOMEM;
> -		goto out;
> -	}
>  	link->master = genpd;
>  	list_add_tail(&link->master_node, &genpd->master_links);
>  	link->slave = subdomain;
> @@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>  	if (IS_ERR_OR_NULL(genpd) || state < 0)
>  		return -EINVAL;
>  
> +	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
> +	if (!cpuidle_data)
> +		return -ENOMEM;
> +
>  	mutex_lock(&genpd->lock);
>  
>  	if (genpd->cpuidle_data) {
>  		ret = -EEXIST;
>  		goto out;

I think you must update the error paths. In this case the kfree() is
missing.

Best regards,
Krzysztof

>  	}
> -	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
> -	if (!cpuidle_data) {
> -		ret = -ENOMEM;
> -		goto out;
> -	}
> +
>  	cpuidle_drv = cpuidle_driver_ref();
>  	if (!cpuidle_drv) {
>  		ret = -ENODEV;
> 

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> In order for domains to be powered on/off in irq locked context, the
> domain locks could either be a spinlock or a mutex, depending on the
> domain. In preparation for atomic support, allocate domain data outside
> the domain locks, so the allocation calls dont have to be context
> sensitive.
> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 25 ++++++++++++-------------
>  1 file changed, 12 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index 3b3367b..dfd7595 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>  int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>  			   struct generic_pm_domain *subdomain)
>  {
> -	struct gpd_link *link;
> +	struct gpd_link *link, *itr;
>  	int ret = 0;
>  
>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
>  	    || genpd == subdomain)
>  		return -EINVAL;
>  
> +	link = kzalloc(sizeof(*link), GFP_KERNEL);
> +	if (!link)
> +		return -ENOMEM;
> +
>  	mutex_lock(&genpd->lock);
>  	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>  
> @@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>  		goto out;
>  	}
>  
> -	list_for_each_entry(link, &genpd->master_links, master_node) {
> -		if (link->slave == subdomain && link->master == genpd) {
> +	list_for_each_entry(itr, &genpd->master_links, master_node) {
> +		if (itr->slave == subdomain && itr->master == genpd) {
>  			ret = -EINVAL;
>  			goto out;
>  		}
>  	}
>  
> -	link = kzalloc(sizeof(*link), GFP_KERNEL);
> -	if (!link) {
> -		ret = -ENOMEM;
> -		goto out;
> -	}
>  	link->master = genpd;
>  	list_add_tail(&link->master_node, &genpd->master_links);
>  	link->slave = subdomain;
> @@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>  	if (IS_ERR_OR_NULL(genpd) || state < 0)
>  		return -EINVAL;
>  
> +	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
> +	if (!cpuidle_data)
> +		return -ENOMEM;
> +
>  	mutex_lock(&genpd->lock);
>  
>  	if (genpd->cpuidle_data) {
>  		ret = -EEXIST;
>  		goto out;

I think you must update the error paths. In this case the kfree() is
missing.

Best regards,
Krzysztof

>  	}
> -	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
> -	if (!cpuidle_data) {
> -		ret = -ENOMEM;
> -		goto out;
> -	}
> +
>  	cpuidle_drv = cpuidle_driver_ref();
>  	if (!cpuidle_drv) {
>  		ret = -ENODEV;
> 

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> Power Domains currently support turning on/off only in process context.
> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.

I can find also other use case: currently the power domain with irq_safe
devices is always powered on. With the patch it could be powered off (of
course if the driver/mach code is irq-safe).

> 
> Genpd inherently provides support for devices, domain hierarchy and can
> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. Multiple such domains could be part of another
> domain. Because mutexes are used to protect and synchronize domain
> operations and cpu idle operations are inherently atomic, the use of
> genpd is not possible for runtime suspend and resume of the pm domain.
> Replacing the locks to spinlocks would allow cpu domain to be be powered
> off to save power, when all the cpus are powered off.
> 
> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
> 
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
> 
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.

So an irq-safe device can be put in irq-safe subdomain which can be a
child of non-irq-safe topdomain?

> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>  include/linux/pm_domain.h   |  11 ++-

Documentation should also be reflected.

>  2 files changed, 164 insertions(+), 47 deletions(-)
> 
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index dfd7595..8b89d15 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -50,6 +50,71 @@
>  static LIST_HEAD(gpd_list);
>  static DEFINE_MUTEX(gpd_list_lock);
>  
> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
> +					unsigned int subclass)
> +	__acquires(&genpd->slock)
> +{
> +	unsigned long flags;
> +
> +	if (unlikely(subclass > 0))
> +		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
> +	else
> +		spin_lock_irqsave(&genpd->slock, flags);
> +
> +	genpd->flags = flags;
> +
> +	return 0;
> +}
> +
> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
> +	__releases(&genpd->slock)
> +{
> +	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
> +	return 0;
> +}
> +
> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
> +					unsigned int subclass)
> +	__acquires(&genpd->mlock)
> +{
> +	if (unlikely(subclass > 0))
> +		mutex_lock_nested(&genpd->mlock, subclass);
> +	else
> +		mutex_lock(&genpd->mlock);
> +
> +	return 0;
> +}
> +
> +static inline int genpd_lock_domain_interruptible_irq(
> +				struct generic_pm_domain *genpd)
> +	__acquires(&genpd->mlock)
> +{
> +	return mutex_lock_interruptible(&genpd->mlock);
> +}
> +
> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
> +	__releases(&genpd->mlock)
> +{
> +	mutex_unlock(&genpd->mlock);
> +	return 0;
> +}
> +
> +#define genpd_lock_domain(genpd)				\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
> +			: genpd_lock_domain_irq(genpd, 0))
> +
> +#define genpd_lock_domain_nested(genpd)				\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
> +			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
> +
> +#define genpd_unlock_domain(genpd)				\
> +	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
> +			: genpd_unlock_domain_irq(genpd))
> +
> +#define genpd_lock_domain_interruptible(genpd)			\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
> +			: genpd_lock_domain_interruptible_irq(genpd))

Why macros? You are not using here benefits of a macro and they are
called just like ordinary functions.

You added "domain" prefix but genpd already contains this. genod_lock(),
genpd_lock_nested() etc. should be sufficient, unless there is a
conflict, similar name planned or you plan to lock something else
(genpd_lock_device?).


> +
>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>  {
>  	struct generic_pm_domain *genpd = NULL, *gpd;
> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>  {
>  	int ret;
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	ret = __pm_genpd_poweron(genpd);
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  	return ret;
>  }
>  
> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>  		spin_unlock_irq(&dev->power.lock);
>  
>  		if (!IS_ERR(genpd)) {
> -			mutex_lock(&genpd->lock);
> +			genpd_lock_domain(genpd);
>  			genpd->max_off_time_changed = true;
> -			mutex_unlock(&genpd->lock);
> +			genpd_unlock_domain(genpd);
>  		}
>  
>  		dev = dev->parent;
> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>  			return -EBUSY;
>  
>  		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
> -		    || pdd->dev->power.irq_safe))
> +			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>  			not_suspended++;
>  	}
>  
> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>  
>  	genpd = container_of(work, struct generic_pm_domain, power_off_work);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	pm_genpd_poweroff(genpd);

Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
At least in next-20150604 but maybe the patches, which this depends on,
changed it?


> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  }
>  
>  /**
> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return -EINVAL;
>  
> -	/*
> -	 * We can't allow to power off the PM domain if it holds an irq_safe
> -	 * device. That's beacuse we use mutexes to protect data while power
> -	 * off and on the PM domain, thus we can't execute in atomic context.
> -	 */
> -	if (dev->power.irq_safe)
> +	/* We can't allow to power off a domain that is also not irq safe. */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
>  		return -EBUSY;
>  
>  	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>  		return ret;
>  	}
>  
> -	mutex_lock(&genpd->lock);
> +	/*
> +	 * If power.irq_safe is set, this routine will be run with interrupts
> +	 * off, so suspend only if the power domain is irq_safe.
> +	 */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
> +		return 0;
> +
> +	genpd_lock_domain(genpd);
> +
>  	genpd->in_progress++;
>  	pm_genpd_poweroff(genpd);
>  	genpd->in_progress--;
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	return 0;
>  }
> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return -EINVAL;
>  
> -	/* If power.irq_safe, the PM domain is never powered off. */
> -	if (dev->power.irq_safe)
> +	/*
> +	 * If power.irq_safe and domain is not, then
> +	 * the PM domain is never powered off.
> +	 */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
>  		return genpd_start_dev_no_timing(genpd, dev);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	ret = __pm_genpd_poweron(genpd);
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (ret)
>  		return ret;
> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>  	if (resume_needed(dev, genpd))
>  		pm_runtime_resume(dev);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  
>  	if (genpd->prepared_count++ == 0) {
>  		genpd->suspended_count = 0;
>  		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>  	}
>  
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (genpd->suspend_power_off) {
>  		pm_runtime_put_noidle(dev);
> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>  
>  	ret = pm_generic_prepare(dev);
>  	if (ret) {
> -		mutex_lock(&genpd->lock);
> +		genpd_lock_domain(genpd);
>  
>  		if (--genpd->prepared_count == 0)
>  			genpd->suspend_power_off = false;
>  
> -		mutex_unlock(&genpd->lock);
> +		genpd_unlock_domain(genpd);
>  		pm_runtime_enable(dev);
>  	}
>  
> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return;
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  
>  	run_complete = !genpd->suspend_power_off;
>  	if (--genpd->prepared_count == 0)
>  		genpd->suspend_power_off = false;
>  
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (run_complete) {
>  		pm_generic_complete(dev);
> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>  		return -EINVAL;
>  
> +	/* Devices in an IRQ safe PM Domain have to be irq safe too */

Why? Can you add this information here? Previously there was a reason in
case of irq_safe devices which you removed leaving only policy.

Best regards,
Krzysztof

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> Power Domains currently support turning on/off only in process context.
> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.

I can find also other use case: currently the power domain with irq_safe
devices is always powered on. With the patch it could be powered off (of
course if the driver/mach code is irq-safe).

> 
> Genpd inherently provides support for devices, domain hierarchy and can
> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. Multiple such domains could be part of another
> domain. Because mutexes are used to protect and synchronize domain
> operations and cpu idle operations are inherently atomic, the use of
> genpd is not possible for runtime suspend and resume of the pm domain.
> Replacing the locks to spinlocks would allow cpu domain to be be powered
> off to save power, when all the cpus are powered off.
> 
> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
> 
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
> 
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.

So an irq-safe device can be put in irq-safe subdomain which can be a
child of non-irq-safe topdomain?

> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>  include/linux/pm_domain.h   |  11 ++-

Documentation should also be reflected.

>  2 files changed, 164 insertions(+), 47 deletions(-)
> 
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index dfd7595..8b89d15 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -50,6 +50,71 @@
>  static LIST_HEAD(gpd_list);
>  static DEFINE_MUTEX(gpd_list_lock);
>  
> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
> +					unsigned int subclass)
> +	__acquires(&genpd->slock)
> +{
> +	unsigned long flags;
> +
> +	if (unlikely(subclass > 0))
> +		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
> +	else
> +		spin_lock_irqsave(&genpd->slock, flags);
> +
> +	genpd->flags = flags;
> +
> +	return 0;
> +}
> +
> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
> +	__releases(&genpd->slock)
> +{
> +	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
> +	return 0;
> +}
> +
> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
> +					unsigned int subclass)
> +	__acquires(&genpd->mlock)
> +{
> +	if (unlikely(subclass > 0))
> +		mutex_lock_nested(&genpd->mlock, subclass);
> +	else
> +		mutex_lock(&genpd->mlock);
> +
> +	return 0;
> +}
> +
> +static inline int genpd_lock_domain_interruptible_irq(
> +				struct generic_pm_domain *genpd)
> +	__acquires(&genpd->mlock)
> +{
> +	return mutex_lock_interruptible(&genpd->mlock);
> +}
> +
> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
> +	__releases(&genpd->mlock)
> +{
> +	mutex_unlock(&genpd->mlock);
> +	return 0;
> +}
> +
> +#define genpd_lock_domain(genpd)				\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
> +			: genpd_lock_domain_irq(genpd, 0))
> +
> +#define genpd_lock_domain_nested(genpd)				\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
> +			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
> +
> +#define genpd_unlock_domain(genpd)				\
> +	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
> +			: genpd_unlock_domain_irq(genpd))
> +
> +#define genpd_lock_domain_interruptible(genpd)			\
> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
> +			: genpd_lock_domain_interruptible_irq(genpd))

Why macros? You are not using here benefits of a macro and they are
called just like ordinary functions.

You added "domain" prefix but genpd already contains this. genod_lock(),
genpd_lock_nested() etc. should be sufficient, unless there is a
conflict, similar name planned or you plan to lock something else
(genpd_lock_device?).


> +
>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>  {
>  	struct generic_pm_domain *genpd = NULL, *gpd;
> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>  {
>  	int ret;
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	ret = __pm_genpd_poweron(genpd);
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  	return ret;
>  }
>  
> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>  		spin_unlock_irq(&dev->power.lock);
>  
>  		if (!IS_ERR(genpd)) {
> -			mutex_lock(&genpd->lock);
> +			genpd_lock_domain(genpd);
>  			genpd->max_off_time_changed = true;
> -			mutex_unlock(&genpd->lock);
> +			genpd_unlock_domain(genpd);
>  		}
>  
>  		dev = dev->parent;
> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>  			return -EBUSY;
>  
>  		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
> -		    || pdd->dev->power.irq_safe))
> +			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>  			not_suspended++;
>  	}
>  
> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>  
>  	genpd = container_of(work, struct generic_pm_domain, power_off_work);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	pm_genpd_poweroff(genpd);

Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
At least in next-20150604 but maybe the patches, which this depends on,
changed it?


> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  }
>  
>  /**
> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return -EINVAL;
>  
> -	/*
> -	 * We can't allow to power off the PM domain if it holds an irq_safe
> -	 * device. That's beacuse we use mutexes to protect data while power
> -	 * off and on the PM domain, thus we can't execute in atomic context.
> -	 */
> -	if (dev->power.irq_safe)
> +	/* We can't allow to power off a domain that is also not irq safe. */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
>  		return -EBUSY;
>  
>  	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>  		return ret;
>  	}
>  
> -	mutex_lock(&genpd->lock);
> +	/*
> +	 * If power.irq_safe is set, this routine will be run with interrupts
> +	 * off, so suspend only if the power domain is irq_safe.
> +	 */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
> +		return 0;
> +
> +	genpd_lock_domain(genpd);
> +
>  	genpd->in_progress++;
>  	pm_genpd_poweroff(genpd);
>  	genpd->in_progress--;
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	return 0;
>  }
> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return -EINVAL;
>  
> -	/* If power.irq_safe, the PM domain is never powered off. */
> -	if (dev->power.irq_safe)
> +	/*
> +	 * If power.irq_safe and domain is not, then
> +	 * the PM domain is never powered off.
> +	 */
> +	if (dev->power.irq_safe && !genpd->irq_safe)
>  		return genpd_start_dev_no_timing(genpd, dev);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  	ret = __pm_genpd_poweron(genpd);
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (ret)
>  		return ret;
> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>  	if (resume_needed(dev, genpd))
>  		pm_runtime_resume(dev);
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  
>  	if (genpd->prepared_count++ == 0) {
>  		genpd->suspended_count = 0;
>  		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>  	}
>  
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (genpd->suspend_power_off) {
>  		pm_runtime_put_noidle(dev);
> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>  
>  	ret = pm_generic_prepare(dev);
>  	if (ret) {
> -		mutex_lock(&genpd->lock);
> +		genpd_lock_domain(genpd);
>  
>  		if (--genpd->prepared_count == 0)
>  			genpd->suspend_power_off = false;
>  
> -		mutex_unlock(&genpd->lock);
> +		genpd_unlock_domain(genpd);
>  		pm_runtime_enable(dev);
>  	}
>  
> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>  	if (IS_ERR(genpd))
>  		return;
>  
> -	mutex_lock(&genpd->lock);
> +	genpd_lock_domain(genpd);
>  
>  	run_complete = !genpd->suspend_power_off;
>  	if (--genpd->prepared_count == 0)
>  		genpd->suspend_power_off = false;
>  
> -	mutex_unlock(&genpd->lock);
> +	genpd_unlock_domain(genpd);
>  
>  	if (run_complete) {
>  		pm_generic_complete(dev);
> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>  		return -EINVAL;
>  
> +	/* Devices in an IRQ safe PM Domain have to be irq safe too */

Why? Can you add this information here? Previously there was a reason in
case of irq_safe devices which you removed leaving only policy.

Best regards,
Krzysztof

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> Generally cpus are grouped under a power domain in a SoC. When all cpus
> in the domain are in their power off state,

What do you exactly mean here by "CPU in power off state"? How does it
map to kernel understanding of CPU device (hotplug? cpuidle?)?

> the cpu domain can also be
> powered off. Genpd provides the framework for defining cpus as devices
> that are part of a cpu domain.

The problem which is solved looks to me like the same problem which
coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
off) can be entered when whole cluster is idle or other CPUs in cluster
are powered off completely.

It seems a little like duplicating the effort around coupled cpuidle.

Best regards,
Krzysztof

> 
> Introduce support for defining and adding a generic power domain for the
> cpus based on the DT specification of power domain providers and
> consumers.  SoC's that have the cpu domain defined in their DT, can
> setup a genpd with a name and the power_on/power_off callbacks. Calling
> pm_cpu_domain_init() will register the genpd and attach the cpus for
> this domain with the genpd.
> 
> CPU_PM notifications for are used to pm_runtime_get_sync() and
> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
> last cpu going down would call the genpd->power_off(). Correspondingly,
> the first cpu up would call the genpd->power_on() callback before
> resuming from idle.
> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile     |   1 +
>  drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
>  include/linux/pm_domain.h       |  12 +++
>  kernel/power/Kconfig            |  12 +++
>  4 files changed, 212 insertions(+)
>  create mode 100644 drivers/base/power/cpu_domain.c
> 
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 1cb8544..debfc74 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
>  obj-$(CONFIG_PM_OPP)	+= opp.o
>  obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
>  obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
> +obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
>  
>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
> diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
> new file mode 100644
> index 0000000..ee90094
> --- /dev/null
> +++ b/drivers/base/power/cpu_domain.c
> @@ -0,0 +1,187 @@
> +/*
> + * Generic CPU domain runtime power on/off support
> + *
> + * Copyright (C) 2015 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/cpu_pm.h>
> +#include <linux/device.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/pm_domain.h>
> +#include <linux/pm_runtime.h>
> +
> +static struct cpumask cpus_handled;
> +
> +static void do_cpu(void *unused)
> +{
> +	int cpu = smp_processor_id();
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	pm_runtime_get_sync(dev);
> +}
> +
> +static int cpuidle_genpd_device_init(int cpu)
> +{
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	/*
> +	 * CPU device have to be irq safe for use with cpuidle, which runs
> +	 * with irqs disabled.
> +	 */
> +	pm_runtime_irq_safe(dev);
> +	pm_runtime_enable(dev);
> +
> +	genpd_dev_pm_attach(dev);
> +
> +	/*
> +	 * Execute the below on 'that' cpu to ensure that the reference
> +	 * counting is correct. Its possible that while this code is
> +	 * executed, the cpu may be in idle but we may incorrectly
> +	 * increment the usage. By executing the do_cpu on 'that' cpu,
> +	 * we can ensure that the cpu and the usage count are matched.
> +	 */
> +	return smp_call_function_single(cpu, do_cpu, NULL, true);
> +}
> +
> +static int cpu_state_notifier(struct notifier_block *n,
> +			unsigned long action, void *hcpu)
> +{
> +	int cpu = smp_processor_id();
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
> +		return NOTIFY_DONE;
> +
> +	switch (action) {
> +	case CPU_PM_ENTER:
> +		pm_runtime_put_sync(dev);
> +		break;
> +
> +	case CPU_PM_ENTER_FAILED:
> +	case CPU_PM_EXIT:
> +		pm_runtime_get_sync(dev);
> +		break;
> +
> +	default:
> +		return NOTIFY_DONE;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static int cpu_online_notifier(struct notifier_block *n,
> +			unsigned long action, void *hcpu)
> +{
> +	int cpu = (unsigned long)hcpu;
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
> +		return NOTIFY_DONE;
> +
> +	switch (action) {
> +	case CPU_STARTING:
> +	case CPU_STARTING_FROZEN:
> +		/*
> +		 * Attach the cpu to its domain if the cpu is coming up
> +		 * for the first time.
> +		 * Called from the cpu that is coming up.
> +		 */
> +		if (!genpd_dev_pm_attach(dev))
> +			do_cpu(NULL);
> +		break;
> +
> +	default:
> +		return NOTIFY_DONE;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static struct notifier_block hotplug_notifier = {
> +	.notifier_call = cpu_online_notifier,
> +};
> +
> +static struct notifier_block cpu_pm_notifier = {
> +	.notifier_call = cpu_state_notifier,
> +};
> +
> +static struct generic_pm_domain *get_cpu_domain(int cpu)
> +{
> +	struct device *dev = get_cpu_device(cpu);
> +	struct of_phandle_args pd_args;
> +	int ret;
> +
> +	/* Make sure we are a domain consumer */
> +	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> +				"#power-domain-cells", 0, &pd_args);
> +	if (ret)
> +		return ERR_PTR(ret);
> +
> +	/* Attach cpus only for this domain */
> +	return of_genpd_get_from_provider(&pd_args);
> +}
> +
> +int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
> +{
> +	int cpu;
> +	int ret;
> +	cpumask_var_t tmpmask;
> +	struct generic_pm_domain *cpupd;
> +
> +	if (!genpd || !dn)
> +		return -EINVAL;
> +
> +	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	/* CPU genpds have to operate in IRQ safe mode */
> +	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
> +
> +	pm_genpd_init(genpd, NULL, false);
> +	ret = of_genpd_add_provider_simple(dn, genpd);
> +	if (ret)
> +		return ret;
> +
> +	/* Only add those cpus to whom we are the domain provider */
> +	for_each_online_cpu(cpu) {
> +		cpupd = get_cpu_domain(cpu);
> +
> +		if (IS_ERR(cpupd))
> +			continue;
> +
> +		if (genpd == cpupd) {
> +			cpuidle_genpd_device_init(cpu);
> +			cpumask_set_cpu(cpu, tmpmask);
> +		}
> +	}
> +
> +	if (cpumask_empty(tmpmask))
> +		goto done;
> +
> +	/*
> +	 * Not all cpus may be online at this point. Use the hotplug
> +	 * notifier to be notified of when the cpu comes online, then
> +	 * attach it to the domain.
> +	 *
> +	 * Register hotplug and cpu_pm notification once for all
> +	 * domains.
> +	 */
> +	if (cpumask_empty(&cpus_handled)) {
> +		cpu_pm_register_notifier(&cpu_pm_notifier);
> +		register_cpu_notifier(&hotplug_notifier);
> +	}
> +
> +	cpumask_copy(&cpus_handled, tmpmask);
> +
> +done:
> +	free_cpumask_var(tmpmask);
> +	return 0;
> +}
> +EXPORT_SYMBOL(pm_cpu_domain_init);
> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
> index dc7cb53..fc97ad8 100644
> --- a/include/linux/pm_domain.h
> +++ b/include/linux/pm_domain.h
> @@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
>  					void *data);
>  
>  int genpd_dev_pm_attach(struct device *dev);
> +
>  #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
>  static inline int __of_genpd_add_provider(struct device_node *np,
>  					genpd_xlate_t xlate, void *data)
> @@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
>  static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
>  #endif
>  
> +#ifdef CONFIG_PM_CPU_DOMAIN
> +extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
> +			struct device_node *dn);
> +#else
> +static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
> +			struct device_node *dn)
> +{
> +	return -ENODEV;
> +}
> +#endif
> +
>  #endif /* _LINUX_PM_DOMAIN_H */
> diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
> index 7e01f78..55d49f6 100644
> --- a/kernel/power/Kconfig
> +++ b/kernel/power/Kconfig
> @@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
>  
>  config CPU_PM
>  	bool
> +
> +config PM_CPU_DOMAIN
> +	def_bool y
> +	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
> +	help
> +	  When cpuidle powers of the cpus in a domain, the domain can also be
> +	  powered off.
> +	  This config option allow for cpus to be registered with the domain
> +	  provider specified in the DT and when the cpu is powered off, calls
> +	  the runtime PM methods to do the reference counting. The last cpu
> +	  going down powers the domain off as well.
> +
> 

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
> Generally cpus are grouped under a power domain in a SoC. When all cpus
> in the domain are in their power off state,

What do you exactly mean here by "CPU in power off state"? How does it
map to kernel understanding of CPU device (hotplug? cpuidle?)?

> the cpu domain can also be
> powered off. Genpd provides the framework for defining cpus as devices
> that are part of a cpu domain.

The problem which is solved looks to me like the same problem which
coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
off) can be entered when whole cluster is idle or other CPUs in cluster
are powered off completely.

It seems a little like duplicating the effort around coupled cpuidle.

Best regards,
Krzysztof

> 
> Introduce support for defining and adding a generic power domain for the
> cpus based on the DT specification of power domain providers and
> consumers.  SoC's that have the cpu domain defined in their DT, can
> setup a genpd with a name and the power_on/power_off callbacks. Calling
> pm_cpu_domain_init() will register the genpd and attach the cpus for
> this domain with the genpd.
> 
> CPU_PM notifications for are used to pm_runtime_get_sync() and
> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
> last cpu going down would call the genpd->power_off(). Correspondingly,
> the first cpu up would call the genpd->power_on() callback before
> resuming from idle.
> 
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/Makefile     |   1 +
>  drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
>  include/linux/pm_domain.h       |  12 +++
>  kernel/power/Kconfig            |  12 +++
>  4 files changed, 212 insertions(+)
>  create mode 100644 drivers/base/power/cpu_domain.c
> 
> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
> index 1cb8544..debfc74 100644
> --- a/drivers/base/power/Makefile
> +++ b/drivers/base/power/Makefile
> @@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
>  obj-$(CONFIG_PM_OPP)	+= opp.o
>  obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
>  obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
> +obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
>  
>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
> diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
> new file mode 100644
> index 0000000..ee90094
> --- /dev/null
> +++ b/drivers/base/power/cpu_domain.c
> @@ -0,0 +1,187 @@
> +/*
> + * Generic CPU domain runtime power on/off support
> + *
> + * Copyright (C) 2015 Linaro Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/cpu_pm.h>
> +#include <linux/device.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/pm_domain.h>
> +#include <linux/pm_runtime.h>
> +
> +static struct cpumask cpus_handled;
> +
> +static void do_cpu(void *unused)
> +{
> +	int cpu = smp_processor_id();
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	pm_runtime_get_sync(dev);
> +}
> +
> +static int cpuidle_genpd_device_init(int cpu)
> +{
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	/*
> +	 * CPU device have to be irq safe for use with cpuidle, which runs
> +	 * with irqs disabled.
> +	 */
> +	pm_runtime_irq_safe(dev);
> +	pm_runtime_enable(dev);
> +
> +	genpd_dev_pm_attach(dev);
> +
> +	/*
> +	 * Execute the below on 'that' cpu to ensure that the reference
> +	 * counting is correct. Its possible that while this code is
> +	 * executed, the cpu may be in idle but we may incorrectly
> +	 * increment the usage. By executing the do_cpu on 'that' cpu,
> +	 * we can ensure that the cpu and the usage count are matched.
> +	 */
> +	return smp_call_function_single(cpu, do_cpu, NULL, true);
> +}
> +
> +static int cpu_state_notifier(struct notifier_block *n,
> +			unsigned long action, void *hcpu)
> +{
> +	int cpu = smp_processor_id();
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
> +		return NOTIFY_DONE;
> +
> +	switch (action) {
> +	case CPU_PM_ENTER:
> +		pm_runtime_put_sync(dev);
> +		break;
> +
> +	case CPU_PM_ENTER_FAILED:
> +	case CPU_PM_EXIT:
> +		pm_runtime_get_sync(dev);
> +		break;
> +
> +	default:
> +		return NOTIFY_DONE;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static int cpu_online_notifier(struct notifier_block *n,
> +			unsigned long action, void *hcpu)
> +{
> +	int cpu = (unsigned long)hcpu;
> +	struct device *dev = get_cpu_device(cpu);
> +
> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
> +		return NOTIFY_DONE;
> +
> +	switch (action) {
> +	case CPU_STARTING:
> +	case CPU_STARTING_FROZEN:
> +		/*
> +		 * Attach the cpu to its domain if the cpu is coming up
> +		 * for the first time.
> +		 * Called from the cpu that is coming up.
> +		 */
> +		if (!genpd_dev_pm_attach(dev))
> +			do_cpu(NULL);
> +		break;
> +
> +	default:
> +		return NOTIFY_DONE;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static struct notifier_block hotplug_notifier = {
> +	.notifier_call = cpu_online_notifier,
> +};
> +
> +static struct notifier_block cpu_pm_notifier = {
> +	.notifier_call = cpu_state_notifier,
> +};
> +
> +static struct generic_pm_domain *get_cpu_domain(int cpu)
> +{
> +	struct device *dev = get_cpu_device(cpu);
> +	struct of_phandle_args pd_args;
> +	int ret;
> +
> +	/* Make sure we are a domain consumer */
> +	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> +				"#power-domain-cells", 0, &pd_args);
> +	if (ret)
> +		return ERR_PTR(ret);
> +
> +	/* Attach cpus only for this domain */
> +	return of_genpd_get_from_provider(&pd_args);
> +}
> +
> +int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
> +{
> +	int cpu;
> +	int ret;
> +	cpumask_var_t tmpmask;
> +	struct generic_pm_domain *cpupd;
> +
> +	if (!genpd || !dn)
> +		return -EINVAL;
> +
> +	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	/* CPU genpds have to operate in IRQ safe mode */
> +	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
> +
> +	pm_genpd_init(genpd, NULL, false);
> +	ret = of_genpd_add_provider_simple(dn, genpd);
> +	if (ret)
> +		return ret;
> +
> +	/* Only add those cpus to whom we are the domain provider */
> +	for_each_online_cpu(cpu) {
> +		cpupd = get_cpu_domain(cpu);
> +
> +		if (IS_ERR(cpupd))
> +			continue;
> +
> +		if (genpd == cpupd) {
> +			cpuidle_genpd_device_init(cpu);
> +			cpumask_set_cpu(cpu, tmpmask);
> +		}
> +	}
> +
> +	if (cpumask_empty(tmpmask))
> +		goto done;
> +
> +	/*
> +	 * Not all cpus may be online at this point. Use the hotplug
> +	 * notifier to be notified of when the cpu comes online, then
> +	 * attach it to the domain.
> +	 *
> +	 * Register hotplug and cpu_pm notification once for all
> +	 * domains.
> +	 */
> +	if (cpumask_empty(&cpus_handled)) {
> +		cpu_pm_register_notifier(&cpu_pm_notifier);
> +		register_cpu_notifier(&hotplug_notifier);
> +	}
> +
> +	cpumask_copy(&cpus_handled, tmpmask);
> +
> +done:
> +	free_cpumask_var(tmpmask);
> +	return 0;
> +}
> +EXPORT_SYMBOL(pm_cpu_domain_init);
> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
> index dc7cb53..fc97ad8 100644
> --- a/include/linux/pm_domain.h
> +++ b/include/linux/pm_domain.h
> @@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
>  					void *data);
>  
>  int genpd_dev_pm_attach(struct device *dev);
> +
>  #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
>  static inline int __of_genpd_add_provider(struct device_node *np,
>  					genpd_xlate_t xlate, void *data)
> @@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
>  static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
>  #endif
>  
> +#ifdef CONFIG_PM_CPU_DOMAIN
> +extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
> +			struct device_node *dn);
> +#else
> +static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
> +			struct device_node *dn)
> +{
> +	return -ENODEV;
> +}
> +#endif
> +
>  #endif /* _LINUX_PM_DOMAIN_H */
> diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
> index 7e01f78..55d49f6 100644
> --- a/kernel/power/Kconfig
> +++ b/kernel/power/Kconfig
> @@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
>  
>  config CPU_PM
>  	bool
> +
> +config PM_CPU_DOMAIN
> +	def_bool y
> +	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
> +	help
> +	  When cpuidle powers of the cpus in a domain, the domain can also be
> +	  powered off.
> +	  This config option allow for cpus to be registered with the domain
> +	  provider specified in the DT and when the cpu is powered off, calls
> +	  the runtime PM methods to do the reference counting. The last cpu
> +	  going down powers the domain off as well.
> +
> 

Re: [PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Lina Iyer]

On Sun, Jun 07 2015 at 02:35 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> In order for domains to be powered on/off in irq locked context, the
>> domain locks could either be a spinlock or a mutex, depending on the
>> domain. In preparation for atomic support, allocate domain data outside
>> the domain locks, so the allocation calls dont have to be context
>> sensitive.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 25 ++++++++++++-------------
>>  1 file changed, 12 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index 3b3367b..dfd7595 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>  int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>  			   struct generic_pm_domain *subdomain)
>>  {
>> -	struct gpd_link *link;
>> +	struct gpd_link *link, *itr;
>>  	int ret = 0;
>>
>>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
>>  	    || genpd == subdomain)
>>  		return -EINVAL;
>>
>> +	link = kzalloc(sizeof(*link), GFP_KERNEL);
>> +	if (!link)
>> +		return -ENOMEM;
>> +
>>  	mutex_lock(&genpd->lock);
>>  	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>>
>> @@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>  		goto out;
>>  	}
>>
>> -	list_for_each_entry(link, &genpd->master_links, master_node) {
>> -		if (link->slave == subdomain && link->master == genpd) {
>> +	list_for_each_entry(itr, &genpd->master_links, master_node) {
>> +		if (itr->slave == subdomain && itr->master == genpd) {
>>  			ret = -EINVAL;
>>  			goto out;
>>  		}
>>  	}
>>
>> -	link = kzalloc(sizeof(*link), GFP_KERNEL);
>> -	if (!link) {
>> -		ret = -ENOMEM;
>> -		goto out;
>> -	}
>>  	link->master = genpd;
>>  	list_add_tail(&link->master_node, &genpd->master_links);
>>  	link->slave = subdomain;
>> @@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>  	if (IS_ERR_OR_NULL(genpd) || state < 0)
>>  		return -EINVAL;
>>
>> +	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>> +	if (!cpuidle_data)
>> +		return -ENOMEM;
>> +
>>  	mutex_lock(&genpd->lock);
>>
>>  	if (genpd->cpuidle_data) {
>>  		ret = -EEXIST;
>>  		goto out;
>
>I think you must update the error paths. In this case the kfree() is
>missing.
>
Yes, thank you. Will fix.

-- Lina
>
>>  	}
>> -	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>> -	if (!cpuidle_data) {
>> -		ret = -ENOMEM;
>> -		goto out;
>> -	}
>> +
>>  	cpuidle_drv = cpuidle_driver_ref();
>>  	if (!cpuidle_drv) {
>>  		ret = -ENODEV;
>>
>

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Lina Iyer]

On Sun, Jun 07 2015 at 02:35 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> In order for domains to be powered on/off in irq locked context, the
>> domain locks could either be a spinlock or a mutex, depending on the
>> domain. In preparation for atomic support, allocate domain data outside
>> the domain locks, so the allocation calls dont have to be context
>> sensitive.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 25 ++++++++++++-------------
>>  1 file changed, 12 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index 3b3367b..dfd7595 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -1367,13 +1367,17 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>  int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>  			   struct generic_pm_domain *subdomain)
>>  {
>> -	struct gpd_link *link;
>> +	struct gpd_link *link, *itr;
>>  	int ret = 0;
>>
>>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain)
>>  	    || genpd == subdomain)
>>  		return -EINVAL;
>>
>> +	link = kzalloc(sizeof(*link), GFP_KERNEL);
>> +	if (!link)
>> +		return -ENOMEM;
>> +
>>  	mutex_lock(&genpd->lock);
>>  	mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>>
>> @@ -1383,18 +1387,13 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>  		goto out;
>>  	}
>>
>> -	list_for_each_entry(link, &genpd->master_links, master_node) {
>> -		if (link->slave == subdomain && link->master == genpd) {
>> +	list_for_each_entry(itr, &genpd->master_links, master_node) {
>> +		if (itr->slave == subdomain && itr->master == genpd) {
>>  			ret = -EINVAL;
>>  			goto out;
>>  		}
>>  	}
>>
>> -	link = kzalloc(sizeof(*link), GFP_KERNEL);
>> -	if (!link) {
>> -		ret = -ENOMEM;
>> -		goto out;
>> -	}
>>  	link->master = genpd;
>>  	list_add_tail(&link->master_node, &genpd->master_links);
>>  	link->slave = subdomain;
>> @@ -1496,17 +1495,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>  	if (IS_ERR_OR_NULL(genpd) || state < 0)
>>  		return -EINVAL;
>>
>> +	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>> +	if (!cpuidle_data)
>> +		return -ENOMEM;
>> +
>>  	mutex_lock(&genpd->lock);
>>
>>  	if (genpd->cpuidle_data) {
>>  		ret = -EEXIST;
>>  		goto out;
>
>I think you must update the error paths. In this case the kfree() is
>missing.
>
Yes, thank you. Will fix.

-- Lina
>
>>  	}
>> -	cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>> -	if (!cpuidle_data) {
>> -		ret = -ENOMEM;
>> -		goto out;
>> -	}
>> +
>>  	cpuidle_drv = cpuidle_driver_ref();
>>  	if (!cpuidle_drv) {
>>  		ret = -ENODEV;
>>
>

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Power Domains currently support turning on/off only in process context.
>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>
>I can find also other use case: currently the power domain with irq_safe
>devices is always powered on. With the patch it could be powered off (of
>course if the driver/mach code is irq-safe).
>
Yes, absolutely.

>>
>> Genpd inherently provides support for devices, domain hierarchy and can
>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware. Multiple such domains could be part of another
>> domain. Because mutexes are used to protect and synchronize domain
>> operations and cpu idle operations are inherently atomic, the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>> off to save power, when all the cpus are powered off.
>>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>
>So an irq-safe device can be put in irq-safe subdomain which can be a
>child of non-irq-safe topdomain?
>
Yes, the container need not be irq-safe but the contained need to be
irqsafe.

>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>>  include/linux/pm_domain.h   |  11 ++-
>
>Documentation should also be reflected.
>
Yes, will add.

>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index dfd7595..8b89d15 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -50,6 +50,71 @@
>>  static LIST_HEAD(gpd_list);
>>  static DEFINE_MUTEX(gpd_list_lock);
>>
>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
>> +					unsigned int subclass)
>> +	__acquires(&genpd->slock)
>> +{
>> +	unsigned long flags;
>> +
>> +	if (unlikely(subclass > 0))
>> +		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>> +	else
>> +		spin_lock_irqsave(&genpd->slock, flags);
>> +
>> +	genpd->flags = flags;
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
>> +	__releases(&genpd->slock)
>> +{
>> +	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
>> +					unsigned int subclass)
>> +	__acquires(&genpd->mlock)
>> +{
>> +	if (unlikely(subclass > 0))
>> +		mutex_lock_nested(&genpd->mlock, subclass);
>> +	else
>> +		mutex_lock(&genpd->mlock);
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_interruptible_irq(
>> +				struct generic_pm_domain *genpd)
>> +	__acquires(&genpd->mlock)
>> +{
>> +	return mutex_lock_interruptible(&genpd->mlock);
>> +}
>> +
>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
>> +	__releases(&genpd->mlock)
>> +{
>> +	mutex_unlock(&genpd->mlock);
>> +	return 0;
>> +}
>> +
>> +#define genpd_lock_domain(genpd)				\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
>> +			: genpd_lock_domain_irq(genpd, 0))
>> +
>> +#define genpd_lock_domain_nested(genpd)				\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
>> +			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>> +
>> +#define genpd_unlock_domain(genpd)				\
>> +	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
>> +			: genpd_unlock_domain_irq(genpd))
>> +
>> +#define genpd_lock_domain_interruptible(genpd)			\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
>> +			: genpd_lock_domain_interruptible_irq(genpd))
>
>Why macros? You are not using here benefits of a macro and they are
>called just like ordinary functions.
>
Well, I didnt see a need for a function that might show up in the stack.
But I have no strong preference either way.

>You added "domain" prefix but genpd already contains this. genod_lock(),
>genpd_lock_nested() etc. should be sufficient, unless there is a
>conflict, similar name planned or you plan to lock something else
>(genpd_lock_device?).
>
Sigh. Yes, you are right. Its redundant. Will remove.

>
>> +
>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>>  {
>>  	struct generic_pm_domain *genpd = NULL, *gpd;
>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>>  {
>>  	int ret;
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	ret = __pm_genpd_poweron(genpd);
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>  	return ret;
>>  }
>>
>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>>  		spin_unlock_irq(&dev->power.lock);
>>
>>  		if (!IS_ERR(genpd)) {
>> -			mutex_lock(&genpd->lock);
>> +			genpd_lock_domain(genpd);
>>  			genpd->max_off_time_changed = true;
>> -			mutex_unlock(&genpd->lock);
>> +			genpd_unlock_domain(genpd);
>>  		}
>>
>>  		dev = dev->parent;
>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>>  			return -EBUSY;
>>
>>  		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>> -		    || pdd->dev->power.irq_safe))
>> +			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>>  			not_suspended++;
>>  	}
>>
>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>>
>>  	genpd = container_of(work, struct generic_pm_domain, power_off_work);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	pm_genpd_poweroff(genpd);
>
>Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
>At least in next-20150604 but maybe the patches, which this depends on,
>changed it?
>
Yes. Ulf's patch remvoed that call.

>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>  }
>>
>>  /**
>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return -EINVAL;
>>
>> -	/*
>> -	 * We can't allow to power off the PM domain if it holds an irq_safe
>> -	 * device. That's beacuse we use mutexes to protect data while power
>> -	 * off and on the PM domain, thus we can't execute in atomic context.
>> -	 */
>> -	if (dev->power.irq_safe)
>> +	/* We can't allow to power off a domain that is also not irq safe. */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>>  		return -EBUSY;
>>
>>  	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>  		return ret;
>>  	}
>>
>> -	mutex_lock(&genpd->lock);
>> +	/*
>> +	 * If power.irq_safe is set, this routine will be run with interrupts
>> +	 * off, so suspend only if the power domain is irq_safe.
>> +	 */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>> +		return 0;
>> +
>> +	genpd_lock_domain(genpd);
>> +
>>  	genpd->in_progress++;
>>  	pm_genpd_poweroff(genpd);
>>  	genpd->in_progress--;
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	return 0;
>>  }
>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return -EINVAL;
>>
>> -	/* If power.irq_safe, the PM domain is never powered off. */
>> -	if (dev->power.irq_safe)
>> +	/*
>> +	 * If power.irq_safe and domain is not, then
>> +	 * the PM domain is never powered off.
>> +	 */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>>  		return genpd_start_dev_no_timing(genpd, dev);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	ret = __pm_genpd_poweron(genpd);
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (ret)
>>  		return ret;
>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>  	if (resume_needed(dev, genpd))
>>  		pm_runtime_resume(dev);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>
>>  	if (genpd->prepared_count++ == 0) {
>>  		genpd->suspended_count = 0;
>>  		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>>  	}
>>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (genpd->suspend_power_off) {
>>  		pm_runtime_put_noidle(dev);
>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>
>>  	ret = pm_generic_prepare(dev);
>>  	if (ret) {
>> -		mutex_lock(&genpd->lock);
>> +		genpd_lock_domain(genpd);
>>
>>  		if (--genpd->prepared_count == 0)
>>  			genpd->suspend_power_off = false;
>>
>> -		mutex_unlock(&genpd->lock);
>> +		genpd_unlock_domain(genpd);
>>  		pm_runtime_enable(dev);
>>  	}
>>
>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return;
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>
>>  	run_complete = !genpd->suspend_power_off;
>>  	if (--genpd->prepared_count == 0)
>>  		genpd->suspend_power_off = false;
>>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (run_complete) {
>>  		pm_generic_complete(dev);
>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>  		return -EINVAL;
>>
>> +	/* Devices in an IRQ safe PM Domain have to be irq safe too */
>
>Why? Can you add this information here? Previously there was a reason in
>case of irq_safe devices which you removed leaving only policy.
>
Sorry, your question is not clear to me.
I believe this is a new requirement that enforces the contained devices
of an irq-safe domain to be irq-safe as well.

Thanks for your review.

-- Lina

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Power Domains currently support turning on/off only in process context.
>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>
>I can find also other use case: currently the power domain with irq_safe
>devices is always powered on. With the patch it could be powered off (of
>course if the driver/mach code is irq-safe).
>
Yes, absolutely.

>>
>> Genpd inherently provides support for devices, domain hierarchy and can
>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware. Multiple such domains could be part of another
>> domain. Because mutexes are used to protect and synchronize domain
>> operations and cpu idle operations are inherently atomic, the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>> off to save power, when all the cpus are powered off.
>>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>
>So an irq-safe device can be put in irq-safe subdomain which can be a
>child of non-irq-safe topdomain?
>
Yes, the container need not be irq-safe but the contained need to be
irqsafe.

>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>>  include/linux/pm_domain.h   |  11 ++-
>
>Documentation should also be reflected.
>
Yes, will add.

>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index dfd7595..8b89d15 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -50,6 +50,71 @@
>>  static LIST_HEAD(gpd_list);
>>  static DEFINE_MUTEX(gpd_list_lock);
>>
>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
>> +					unsigned int subclass)
>> +	__acquires(&genpd->slock)
>> +{
>> +	unsigned long flags;
>> +
>> +	if (unlikely(subclass > 0))
>> +		spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>> +	else
>> +		spin_lock_irqsave(&genpd->slock, flags);
>> +
>> +	genpd->flags = flags;
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
>> +	__releases(&genpd->slock)
>> +{
>> +	spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
>> +					unsigned int subclass)
>> +	__acquires(&genpd->mlock)
>> +{
>> +	if (unlikely(subclass > 0))
>> +		mutex_lock_nested(&genpd->mlock, subclass);
>> +	else
>> +		mutex_lock(&genpd->mlock);
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_interruptible_irq(
>> +				struct generic_pm_domain *genpd)
>> +	__acquires(&genpd->mlock)
>> +{
>> +	return mutex_lock_interruptible(&genpd->mlock);
>> +}
>> +
>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
>> +	__releases(&genpd->mlock)
>> +{
>> +	mutex_unlock(&genpd->mlock);
>> +	return 0;
>> +}
>> +
>> +#define genpd_lock_domain(genpd)				\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
>> +			: genpd_lock_domain_irq(genpd, 0))
>> +
>> +#define genpd_lock_domain_nested(genpd)				\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
>> +			: genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>> +
>> +#define genpd_unlock_domain(genpd)				\
>> +	(genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)	\
>> +			: genpd_unlock_domain_irq(genpd))
>> +
>> +#define genpd_lock_domain_interruptible(genpd)			\
>> +	(genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)	\
>> +			: genpd_lock_domain_interruptible_irq(genpd))
>
>Why macros? You are not using here benefits of a macro and they are
>called just like ordinary functions.
>
Well, I didnt see a need for a function that might show up in the stack.
But I have no strong preference either way.

>You added "domain" prefix but genpd already contains this. genod_lock(),
>genpd_lock_nested() etc. should be sufficient, unless there is a
>conflict, similar name planned or you plan to lock something else
>(genpd_lock_device?).
>
Sigh. Yes, you are right. Its redundant. Will remove.

>
>> +
>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>>  {
>>  	struct generic_pm_domain *genpd = NULL, *gpd;
>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>>  {
>>  	int ret;
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	ret = __pm_genpd_poweron(genpd);
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>  	return ret;
>>  }
>>
>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>>  		spin_unlock_irq(&dev->power.lock);
>>
>>  		if (!IS_ERR(genpd)) {
>> -			mutex_lock(&genpd->lock);
>> +			genpd_lock_domain(genpd);
>>  			genpd->max_off_time_changed = true;
>> -			mutex_unlock(&genpd->lock);
>> +			genpd_unlock_domain(genpd);
>>  		}
>>
>>  		dev = dev->parent;
>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>>  			return -EBUSY;
>>
>>  		if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>> -		    || pdd->dev->power.irq_safe))
>> +			|| (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>>  			not_suspended++;
>>  	}
>>
>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>>
>>  	genpd = container_of(work, struct generic_pm_domain, power_off_work);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	pm_genpd_poweroff(genpd);
>
>Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
>At least in next-20150604 but maybe the patches, which this depends on,
>changed it?
>
Yes. Ulf's patch remvoed that call.

>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>  }
>>
>>  /**
>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return -EINVAL;
>>
>> -	/*
>> -	 * We can't allow to power off the PM domain if it holds an irq_safe
>> -	 * device. That's beacuse we use mutexes to protect data while power
>> -	 * off and on the PM domain, thus we can't execute in atomic context.
>> -	 */
>> -	if (dev->power.irq_safe)
>> +	/* We can't allow to power off a domain that is also not irq safe. */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>>  		return -EBUSY;
>>
>>  	stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>  		return ret;
>>  	}
>>
>> -	mutex_lock(&genpd->lock);
>> +	/*
>> +	 * If power.irq_safe is set, this routine will be run with interrupts
>> +	 * off, so suspend only if the power domain is irq_safe.
>> +	 */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>> +		return 0;
>> +
>> +	genpd_lock_domain(genpd);
>> +
>>  	genpd->in_progress++;
>>  	pm_genpd_poweroff(genpd);
>>  	genpd->in_progress--;
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	return 0;
>>  }
>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return -EINVAL;
>>
>> -	/* If power.irq_safe, the PM domain is never powered off. */
>> -	if (dev->power.irq_safe)
>> +	/*
>> +	 * If power.irq_safe and domain is not, then
>> +	 * the PM domain is never powered off.
>> +	 */
>> +	if (dev->power.irq_safe && !genpd->irq_safe)
>>  		return genpd_start_dev_no_timing(genpd, dev);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>  	ret = __pm_genpd_poweron(genpd);
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (ret)
>>  		return ret;
>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>  	if (resume_needed(dev, genpd))
>>  		pm_runtime_resume(dev);
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>
>>  	if (genpd->prepared_count++ == 0) {
>>  		genpd->suspended_count = 0;
>>  		genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>>  	}
>>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (genpd->suspend_power_off) {
>>  		pm_runtime_put_noidle(dev);
>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>
>>  	ret = pm_generic_prepare(dev);
>>  	if (ret) {
>> -		mutex_lock(&genpd->lock);
>> +		genpd_lock_domain(genpd);
>>
>>  		if (--genpd->prepared_count == 0)
>>  			genpd->suspend_power_off = false;
>>
>> -		mutex_unlock(&genpd->lock);
>> +		genpd_unlock_domain(genpd);
>>  		pm_runtime_enable(dev);
>>  	}
>>
>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>>  	if (IS_ERR(genpd))
>>  		return;
>>
>> -	mutex_lock(&genpd->lock);
>> +	genpd_lock_domain(genpd);
>>
>>  	run_complete = !genpd->suspend_power_off;
>>  	if (--genpd->prepared_count == 0)
>>  		genpd->suspend_power_off = false;
>>
>> -	mutex_unlock(&genpd->lock);
>> +	genpd_unlock_domain(genpd);
>>
>>  	if (run_complete) {
>>  		pm_generic_complete(dev);
>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>  	if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>  		return -EINVAL;
>>
>> +	/* Devices in an IRQ safe PM Domain have to be irq safe too */
>
>Why? Can you add this information here? Previously there was a reason in
>case of irq_safe devices which you removed leaving only policy.
>
Sorry, your question is not clear to me.
I believe this is a new requirement that enforces the contained devices
of an irq-safe domain to be irq-safe as well.

Thanks for your review.

-- Lina

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state,
>
>What do you exactly mean here by "CPU in power off state"? How does it
>map to kernel understanding of CPU device (hotplug? cpuidle?)?
>
Both cpuidle and hotplug could end with with core being powered down at
the platform driver or at PSCI (on V8). It does not matter which of
these two frameworks resulted in the cpu being powered off. But, if all
cpus in the domain are powered off, then the domain could be powered off
as well. This is the premise of this change. It is probably easier to
power off the domain when the cores in that domain/cluster have been
hotplugged off. It saves power to turn off the domain at that time, but
more power savings can be achieved if the domain could also be powered
off during cpuidle. Hotplug is not a common occurance, while cpuidle is.

>> the cpu domain can also be
>> powered off. Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>
>The problem which is solved looks to me like the same problem which
>coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>off) can be entered when whole cluster is idle or other CPUs in cluster
>are powered off completely.
>
>It seems a little like duplicating the effort around coupled cpuidle.
>
I see where are you are going with this, but genpd solution is not
exactly a duplicate of the solution.

Couple state is used to put the cpus in a deeper sleep state, which
could also result in powering off the domain. Coupled cpuidle is a
cpuidle mechanism for choosing a deeper sleep mode on certain hardware
that can only enter such a mode when all cpus cooperate.

This patch attempts to describe the backend of a cpu domain. CPUs are
responsible for individual cpuidle states, cpus do enter their
recommended deepest idle state at the time of no activity. A cpu-domain
could be comprised of cpus, and other devices like GIC, busses etc, that
all need to idle before the domain can be powered off. This patch does
not dictate which idle state any those devices should enter, or
coordinate the idle states between devices. But, if cpus, choose to
power down, then this patch recognizes that and reduces the reference
usage count on the domain. Only when all devices in the domain remove
their usage count, will the domain be powered off. 

There are two things this patch provides -

i. A generic way to initialize a genpd specifically for cpus. (The
platform specifies the relation between a cpu and its domain in the DT
and provides the memory for the genpd structure)

ii. On behalf of a platform, we track when the cpus power up and down
and use runtime_get and runtime_put on the genpd.

Unlike coupled cpuidle, individual cpu idle state is not manipulated.
Coupled cpuidle does not care if the domain is powered off, it is used
to allow a certain C-state for the cpu, based on the idleness of other
cpus in that cluster. The focus of the series is powering down the
domain when the devices (cpus included) are powered off. You could see
this patch as a cpu-pm and runtime-pm interface layer.

Hope that helps.

Thanks,
Lina

>
>>
>> Introduce support for defining and adding a generic power domain for the
>> cpus based on the DT specification of power domain providers and
>> consumers.  SoC's that have the cpu domain defined in their DT, can
>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>> this domain with the genpd.
>>
>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>> last cpu going down would call the genpd->power_off(). Correspondingly,
>> the first cpu up would call the genpd->power_on() callback before
>> resuming from idle.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/Makefile     |   1 +
>>  drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
>>  include/linux/pm_domain.h       |  12 +++
>>  kernel/power/Kconfig            |  12 +++
>>  4 files changed, 212 insertions(+)
>>  create mode 100644 drivers/base/power/cpu_domain.c
>>
>> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
>> index 1cb8544..debfc74 100644
>> --- a/drivers/base/power/Makefile
>> +++ b/drivers/base/power/Makefile
>> @@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
>>  obj-$(CONFIG_PM_OPP)	+= opp.o
>>  obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
>>  obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
>> +obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
>>
>>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
>> diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
>> new file mode 100644
>> index 0000000..ee90094
>> --- /dev/null
>> +++ b/drivers/base/power/cpu_domain.c
>> @@ -0,0 +1,187 @@
>> +/*
>> + * Generic CPU domain runtime power on/off support
>> + *
>> + * Copyright (C) 2015 Linaro Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + */
>> +
>> +#include <linux/cpu.h>
>> +#include <linux/cpu_pm.h>
>> +#include <linux/device.h>
>> +#include <linux/kernel.h>
>> +#include <linux/module.h>
>> +#include <linux/of.h>
>> +#include <linux/pm_domain.h>
>> +#include <linux/pm_runtime.h>
>> +
>> +static struct cpumask cpus_handled;
>> +
>> +static void do_cpu(void *unused)
>> +{
>> +	int cpu = smp_processor_id();
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	pm_runtime_get_sync(dev);
>> +}
>> +
>> +static int cpuidle_genpd_device_init(int cpu)
>> +{
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	/*
>> +	 * CPU device have to be irq safe for use with cpuidle, which runs
>> +	 * with irqs disabled.
>> +	 */
>> +	pm_runtime_irq_safe(dev);
>> +	pm_runtime_enable(dev);
>> +
>> +	genpd_dev_pm_attach(dev);
>> +
>> +	/*
>> +	 * Execute the below on 'that' cpu to ensure that the reference
>> +	 * counting is correct. Its possible that while this code is
>> +	 * executed, the cpu may be in idle but we may incorrectly
>> +	 * increment the usage. By executing the do_cpu on 'that' cpu,
>> +	 * we can ensure that the cpu and the usage count are matched.
>> +	 */
>> +	return smp_call_function_single(cpu, do_cpu, NULL, true);
>> +}
>> +
>> +static int cpu_state_notifier(struct notifier_block *n,
>> +			unsigned long action, void *hcpu)
>> +{
>> +	int cpu = smp_processor_id();
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
>> +		return NOTIFY_DONE;
>> +
>> +	switch (action) {
>> +	case CPU_PM_ENTER:
>> +		pm_runtime_put_sync(dev);
>> +		break;
>> +
>> +	case CPU_PM_ENTER_FAILED:
>> +	case CPU_PM_EXIT:
>> +		pm_runtime_get_sync(dev);
>> +		break;
>> +
>> +	default:
>> +		return NOTIFY_DONE;
>> +	}
>> +
>> +	return NOTIFY_OK;
>> +}
>> +
>> +static int cpu_online_notifier(struct notifier_block *n,
>> +			unsigned long action, void *hcpu)
>> +{
>> +	int cpu = (unsigned long)hcpu;
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
>> +		return NOTIFY_DONE;
>> +
>> +	switch (action) {
>> +	case CPU_STARTING:
>> +	case CPU_STARTING_FROZEN:
>> +		/*
>> +		 * Attach the cpu to its domain if the cpu is coming up
>> +		 * for the first time.
>> +		 * Called from the cpu that is coming up.
>> +		 */
>> +		if (!genpd_dev_pm_attach(dev))
>> +			do_cpu(NULL);
>> +		break;
>> +
>> +	default:
>> +		return NOTIFY_DONE;
>> +	}
>> +
>> +	return NOTIFY_OK;
>> +}
>> +
>> +static struct notifier_block hotplug_notifier = {
>> +	.notifier_call = cpu_online_notifier,
>> +};
>> +
>> +static struct notifier_block cpu_pm_notifier = {
>> +	.notifier_call = cpu_state_notifier,
>> +};
>> +
>> +static struct generic_pm_domain *get_cpu_domain(int cpu)
>> +{
>> +	struct device *dev = get_cpu_device(cpu);
>> +	struct of_phandle_args pd_args;
>> +	int ret;
>> +
>> +	/* Make sure we are a domain consumer */
>> +	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
>> +				"#power-domain-cells", 0, &pd_args);
>> +	if (ret)
>> +		return ERR_PTR(ret);
>> +
>> +	/* Attach cpus only for this domain */
>> +	return of_genpd_get_from_provider(&pd_args);
>> +}
>> +
>> +int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
>> +{
>> +	int cpu;
>> +	int ret;
>> +	cpumask_var_t tmpmask;
>> +	struct generic_pm_domain *cpupd;
>> +
>> +	if (!genpd || !dn)
>> +		return -EINVAL;
>> +
>> +	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
>> +		return -ENOMEM;
>> +
>> +	/* CPU genpds have to operate in IRQ safe mode */
>> +	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
>> +
>> +	pm_genpd_init(genpd, NULL, false);
>> +	ret = of_genpd_add_provider_simple(dn, genpd);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/* Only add those cpus to whom we are the domain provider */
>> +	for_each_online_cpu(cpu) {
>> +		cpupd = get_cpu_domain(cpu);
>> +
>> +		if (IS_ERR(cpupd))
>> +			continue;
>> +
>> +		if (genpd == cpupd) {
>> +			cpuidle_genpd_device_init(cpu);
>> +			cpumask_set_cpu(cpu, tmpmask);
>> +		}
>> +	}
>> +
>> +	if (cpumask_empty(tmpmask))
>> +		goto done;
>> +
>> +	/*
>> +	 * Not all cpus may be online at this point. Use the hotplug
>> +	 * notifier to be notified of when the cpu comes online, then
>> +	 * attach it to the domain.
>> +	 *
>> +	 * Register hotplug and cpu_pm notification once for all
>> +	 * domains.
>> +	 */
>> +	if (cpumask_empty(&cpus_handled)) {
>> +		cpu_pm_register_notifier(&cpu_pm_notifier);
>> +		register_cpu_notifier(&hotplug_notifier);
>> +	}
>> +
>> +	cpumask_copy(&cpus_handled, tmpmask);
>> +
>> +done:
>> +	free_cpumask_var(tmpmask);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL(pm_cpu_domain_init);
>> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
>> index dc7cb53..fc97ad8 100644
>> --- a/include/linux/pm_domain.h
>> +++ b/include/linux/pm_domain.h
>> @@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
>>  					void *data);
>>
>>  int genpd_dev_pm_attach(struct device *dev);
>> +
>>  #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
>>  static inline int __of_genpd_add_provider(struct device_node *np,
>>  					genpd_xlate_t xlate, void *data)
>> @@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
>>  static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
>>  #endif
>>
>> +#ifdef CONFIG_PM_CPU_DOMAIN
>> +extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
>> +			struct device_node *dn);
>> +#else
>> +static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
>> +			struct device_node *dn)
>> +{
>> +	return -ENODEV;
>> +}
>> +#endif
>> +
>>  #endif /* _LINUX_PM_DOMAIN_H */
>> diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
>> index 7e01f78..55d49f6 100644
>> --- a/kernel/power/Kconfig
>> +++ b/kernel/power/Kconfig
>> @@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
>>
>>  config CPU_PM
>>  	bool
>> +
>> +config PM_CPU_DOMAIN
>> +	def_bool y
>> +	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
>> +	help
>> +	  When cpuidle powers of the cpus in a domain, the domain can also be
>> +	  powered off.
>> +	  This config option allow for cpus to be registered with the domain
>> +	  provider specified in the DT and when the cpu is powered off, calls
>> +	  the runtime PM methods to do the reference counting. The last cpu
>> +	  going down powers the domain off as well.
>> +
>>
>

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state,
>
>What do you exactly mean here by "CPU in power off state"? How does it
>map to kernel understanding of CPU device (hotplug? cpuidle?)?
>
Both cpuidle and hotplug could end with with core being powered down at
the platform driver or at PSCI (on V8). It does not matter which of
these two frameworks resulted in the cpu being powered off. But, if all
cpus in the domain are powered off, then the domain could be powered off
as well. This is the premise of this change. It is probably easier to
power off the domain when the cores in that domain/cluster have been
hotplugged off. It saves power to turn off the domain at that time, but
more power savings can be achieved if the domain could also be powered
off during cpuidle. Hotplug is not a common occurance, while cpuidle is.

>> the cpu domain can also be
>> powered off. Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>
>The problem which is solved looks to me like the same problem which
>coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>off) can be entered when whole cluster is idle or other CPUs in cluster
>are powered off completely.
>
>It seems a little like duplicating the effort around coupled cpuidle.
>
I see where are you are going with this, but genpd solution is not
exactly a duplicate of the solution.

Couple state is used to put the cpus in a deeper sleep state, which
could also result in powering off the domain. Coupled cpuidle is a
cpuidle mechanism for choosing a deeper sleep mode on certain hardware
that can only enter such a mode when all cpus cooperate.

This patch attempts to describe the backend of a cpu domain. CPUs are
responsible for individual cpuidle states, cpus do enter their
recommended deepest idle state at the time of no activity. A cpu-domain
could be comprised of cpus, and other devices like GIC, busses etc, that
all need to idle before the domain can be powered off. This patch does
not dictate which idle state any those devices should enter, or
coordinate the idle states between devices. But, if cpus, choose to
power down, then this patch recognizes that and reduces the reference
usage count on the domain. Only when all devices in the domain remove
their usage count, will the domain be powered off. 

There are two things this patch provides -

i. A generic way to initialize a genpd specifically for cpus. (The
platform specifies the relation between a cpu and its domain in the DT
and provides the memory for the genpd structure)

ii. On behalf of a platform, we track when the cpus power up and down
and use runtime_get and runtime_put on the genpd.

Unlike coupled cpuidle, individual cpu idle state is not manipulated.
Coupled cpuidle does not care if the domain is powered off, it is used
to allow a certain C-state for the cpu, based on the idleness of other
cpus in that cluster. The focus of the series is powering down the
domain when the devices (cpus included) are powered off. You could see
this patch as a cpu-pm and runtime-pm interface layer.

Hope that helps.

Thanks,
Lina

>
>>
>> Introduce support for defining and adding a generic power domain for the
>> cpus based on the DT specification of power domain providers and
>> consumers.  SoC's that have the cpu domain defined in their DT, can
>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>> this domain with the genpd.
>>
>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>> last cpu going down would call the genpd->power_off(). Correspondingly,
>> the first cpu up would call the genpd->power_on() callback before
>> resuming from idle.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/Makefile     |   1 +
>>  drivers/base/power/cpu_domain.c | 187 ++++++++++++++++++++++++++++++++++++++++
>>  include/linux/pm_domain.h       |  12 +++
>>  kernel/power/Kconfig            |  12 +++
>>  4 files changed, 212 insertions(+)
>>  create mode 100644 drivers/base/power/cpu_domain.c
>>
>> diff --git a/drivers/base/power/Makefile b/drivers/base/power/Makefile
>> index 1cb8544..debfc74 100644
>> --- a/drivers/base/power/Makefile
>> +++ b/drivers/base/power/Makefile
>> @@ -4,5 +4,6 @@ obj-$(CONFIG_PM_TRACE_RTC)	+= trace.o
>>  obj-$(CONFIG_PM_OPP)	+= opp.o
>>  obj-$(CONFIG_PM_GENERIC_DOMAINS)	+=  domain.o domain_governor.o
>>  obj-$(CONFIG_HAVE_CLK)	+= clock_ops.o
>> +obj-$(CONFIG_PM_CPU_DOMAIN)	+= cpu_domain.o
>>
>>  ccflags-$(CONFIG_DEBUG_DRIVER) := -DDEBUG
>> diff --git a/drivers/base/power/cpu_domain.c b/drivers/base/power/cpu_domain.c
>> new file mode 100644
>> index 0000000..ee90094
>> --- /dev/null
>> +++ b/drivers/base/power/cpu_domain.c
>> @@ -0,0 +1,187 @@
>> +/*
>> + * Generic CPU domain runtime power on/off support
>> + *
>> + * Copyright (C) 2015 Linaro Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + */
>> +
>> +#include <linux/cpu.h>
>> +#include <linux/cpu_pm.h>
>> +#include <linux/device.h>
>> +#include <linux/kernel.h>
>> +#include <linux/module.h>
>> +#include <linux/of.h>
>> +#include <linux/pm_domain.h>
>> +#include <linux/pm_runtime.h>
>> +
>> +static struct cpumask cpus_handled;
>> +
>> +static void do_cpu(void *unused)
>> +{
>> +	int cpu = smp_processor_id();
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	pm_runtime_get_sync(dev);
>> +}
>> +
>> +static int cpuidle_genpd_device_init(int cpu)
>> +{
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	/*
>> +	 * CPU device have to be irq safe for use with cpuidle, which runs
>> +	 * with irqs disabled.
>> +	 */
>> +	pm_runtime_irq_safe(dev);
>> +	pm_runtime_enable(dev);
>> +
>> +	genpd_dev_pm_attach(dev);
>> +
>> +	/*
>> +	 * Execute the below on 'that' cpu to ensure that the reference
>> +	 * counting is correct. Its possible that while this code is
>> +	 * executed, the cpu may be in idle but we may incorrectly
>> +	 * increment the usage. By executing the do_cpu on 'that' cpu,
>> +	 * we can ensure that the cpu and the usage count are matched.
>> +	 */
>> +	return smp_call_function_single(cpu, do_cpu, NULL, true);
>> +}
>> +
>> +static int cpu_state_notifier(struct notifier_block *n,
>> +			unsigned long action, void *hcpu)
>> +{
>> +	int cpu = smp_processor_id();
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
>> +		return NOTIFY_DONE;
>> +
>> +	switch (action) {
>> +	case CPU_PM_ENTER:
>> +		pm_runtime_put_sync(dev);
>> +		break;
>> +
>> +	case CPU_PM_ENTER_FAILED:
>> +	case CPU_PM_EXIT:
>> +		pm_runtime_get_sync(dev);
>> +		break;
>> +
>> +	default:
>> +		return NOTIFY_DONE;
>> +	}
>> +
>> +	return NOTIFY_OK;
>> +}
>> +
>> +static int cpu_online_notifier(struct notifier_block *n,
>> +			unsigned long action, void *hcpu)
>> +{
>> +	int cpu = (unsigned long)hcpu;
>> +	struct device *dev = get_cpu_device(cpu);
>> +
>> +	if (!cpumask_test_cpu(cpu, &cpus_handled))
>> +		return NOTIFY_DONE;
>> +
>> +	switch (action) {
>> +	case CPU_STARTING:
>> +	case CPU_STARTING_FROZEN:
>> +		/*
>> +		 * Attach the cpu to its domain if the cpu is coming up
>> +		 * for the first time.
>> +		 * Called from the cpu that is coming up.
>> +		 */
>> +		if (!genpd_dev_pm_attach(dev))
>> +			do_cpu(NULL);
>> +		break;
>> +
>> +	default:
>> +		return NOTIFY_DONE;
>> +	}
>> +
>> +	return NOTIFY_OK;
>> +}
>> +
>> +static struct notifier_block hotplug_notifier = {
>> +	.notifier_call = cpu_online_notifier,
>> +};
>> +
>> +static struct notifier_block cpu_pm_notifier = {
>> +	.notifier_call = cpu_state_notifier,
>> +};
>> +
>> +static struct generic_pm_domain *get_cpu_domain(int cpu)
>> +{
>> +	struct device *dev = get_cpu_device(cpu);
>> +	struct of_phandle_args pd_args;
>> +	int ret;
>> +
>> +	/* Make sure we are a domain consumer */
>> +	ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
>> +				"#power-domain-cells", 0, &pd_args);
>> +	if (ret)
>> +		return ERR_PTR(ret);
>> +
>> +	/* Attach cpus only for this domain */
>> +	return of_genpd_get_from_provider(&pd_args);
>> +}
>> +
>> +int pm_cpu_domain_init(struct generic_pm_domain *genpd, struct device_node *dn)
>> +{
>> +	int cpu;
>> +	int ret;
>> +	cpumask_var_t tmpmask;
>> +	struct generic_pm_domain *cpupd;
>> +
>> +	if (!genpd || !dn)
>> +		return -EINVAL;
>> +
>> +	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
>> +		return -ENOMEM;
>> +
>> +	/* CPU genpds have to operate in IRQ safe mode */
>> +	genpd->flags |= GENPD_FLAG_IRQ_SAFE;
>> +
>> +	pm_genpd_init(genpd, NULL, false);
>> +	ret = of_genpd_add_provider_simple(dn, genpd);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/* Only add those cpus to whom we are the domain provider */
>> +	for_each_online_cpu(cpu) {
>> +		cpupd = get_cpu_domain(cpu);
>> +
>> +		if (IS_ERR(cpupd))
>> +			continue;
>> +
>> +		if (genpd == cpupd) {
>> +			cpuidle_genpd_device_init(cpu);
>> +			cpumask_set_cpu(cpu, tmpmask);
>> +		}
>> +	}
>> +
>> +	if (cpumask_empty(tmpmask))
>> +		goto done;
>> +
>> +	/*
>> +	 * Not all cpus may be online at this point. Use the hotplug
>> +	 * notifier to be notified of when the cpu comes online, then
>> +	 * attach it to the domain.
>> +	 *
>> +	 * Register hotplug and cpu_pm notification once for all
>> +	 * domains.
>> +	 */
>> +	if (cpumask_empty(&cpus_handled)) {
>> +		cpu_pm_register_notifier(&cpu_pm_notifier);
>> +		register_cpu_notifier(&hotplug_notifier);
>> +	}
>> +
>> +	cpumask_copy(&cpus_handled, tmpmask);
>> +
>> +done:
>> +	free_cpumask_var(tmpmask);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL(pm_cpu_domain_init);
>> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
>> index dc7cb53..fc97ad8 100644
>> --- a/include/linux/pm_domain.h
>> +++ b/include/linux/pm_domain.h
>> @@ -280,6 +280,7 @@ struct generic_pm_domain *__of_genpd_xlate_onecell(
>>  					void *data);
>>
>>  int genpd_dev_pm_attach(struct device *dev);
>> +
>>  #else /* !CONFIG_PM_GENERIC_DOMAINS_OF */
>>  static inline int __of_genpd_add_provider(struct device_node *np,
>>  					genpd_xlate_t xlate, void *data)
>> @@ -325,4 +326,15 @@ static inline int dev_pm_domain_attach(struct device *dev, bool power_on)
>>  static inline void dev_pm_domain_detach(struct device *dev, bool power_off) {}
>>  #endif
>>
>> +#ifdef CONFIG_PM_CPU_DOMAIN
>> +extern int pm_cpu_domain_init(struct generic_pm_domain *genpd,
>> +			struct device_node *dn);
>> +#else
>> +static inline int pm_cpu_domain_init(struct generic_pm_domain *genpd,
>> +			struct device_node *dn)
>> +{
>> +	return -ENODEV;
>> +}
>> +#endif
>> +
>>  #endif /* _LINUX_PM_DOMAIN_H */
>> diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
>> index 7e01f78..55d49f6 100644
>> --- a/kernel/power/Kconfig
>> +++ b/kernel/power/Kconfig
>> @@ -301,3 +301,15 @@ config PM_GENERIC_DOMAINS_OF
>>
>>  config CPU_PM
>>  	bool
>> +
>> +config PM_CPU_DOMAIN
>> +	def_bool y
>> +	depends on PM_GENERIC_DOMAINS_OF && CPU_PM
>> +	help
>> +	  When cpuidle powers of the cpus in a domain, the domain can also be
>> +	  powered off.
>> +	  This config option allow for cpus to be registered with the domain
>> +	  provider specified in the DT and when the cpu is powered off, calls
>> +	  the runtime PM methods to do the reference counting. The last cpu
>> +	  going down powers the domain off as well.
>> +
>>
>

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Krzysztof Kozlowski <k.kozlowski@samsung.com> writes:

> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state,
>
> What do you exactly mean here by "CPU in power off state"? How does it
> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>
>> the cpu domain can also be
>> powered off. Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>
> The problem which is solved looks to me like the same problem which
> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
> off) can be entered when whole cluster is idle or other CPUs in cluster
> are powered off completely.
>
> It seems a little like duplicating the effort around coupled cpuidle.

Yes, it duplicates some aspects of coupled idle states, but coupled
states have their own limitations:

- only handles CPUs, not other devices sharing a power rail (e.g. L2$,
  GIC, floating point unit, CoreSight, etc. etc.)

- not scaling well past 2 CPUs

- doesn't handle clusters: While this series only addresses CPUs
  currently, the approach can be extended.  Because genpd handles nested
  domains, the could be used to model clusters as well.

Kevin

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Krzysztof Kozlowski <k.kozlowski@samsung.com> writes:

> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state,
>
> What do you exactly mean here by "CPU in power off state"? How does it
> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>
>> the cpu domain can also be
>> powered off. Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>
> The problem which is solved looks to me like the same problem which
> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
> off) can be entered when whole cluster is idle or other CPUs in cluster
> are powered off completely.
>
> It seems a little like duplicating the effort around coupled cpuidle.

Yes, it duplicates some aspects of coupled idle states, but coupled
states have their own limitations:

- only handles CPUs, not other devices sharing a power rail (e.g. L2$,
  GIC, floating point unit, CoreSight, etc. etc.)

- not scaling well past 2 CPUs

- doesn't handle clusters: While this series only addresses CPUs
  currently, the approach can be extended.  Because genpd handles nested
  domains, the could be used to model clusters as well.

Kevin

Re: [PATCH RFC 0/3] PM / Domains: Generic PM domains for cpus

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> This is an attempt to provide a generic PM domain for cpus, so as to allow the
> cpu domain to be powered off, when all the cpus are in power down state during
> cpuidle. 

You don't use the word "cluster" here, but it might be helpful to
summarize this as using a genpd to model a cluster.  A cluster contains
CPUs, but also additional devices/logic like L2$, GIC, PMUs, floating
point units, CoreSight, etc. etc.

> The rationale behind the change is that newer SoCs can power down the
> cpus for very short sleep times to save on leakage power. The domain which
> usually has the cpus, a second level cache and some peripheral hardware is
> powered by a rail that can also be turned off when the cpus are not in
> use.

This last sentence doesn't read well.  I think you meant something like:

In the domain which has the CPUs, a second level cache and some
peripheral hardware might share the power rail with the CPUs so could
also be turned off when the cpus are not in use.

> Devices for each cpu, L2 and other related blocks could be attached to the
> domain and when there are no uses of these devices (device idle) the domain
> could also be powered off. Generic PM domains provides all the backend needed
> for such an organization.
>
> In the first 2 patches, I make genpd usable in atomic context as well. CPUIdle

s/as well//

> runs with irqs disabled and therefore use of mutexes in the current genpd
> implementation is a limitation. But, not all PM domains need to operate in irq
> safe context, those that need to can specify explicitly the irq safe
> requirement of the genpd at init. Devices and sub-domains that attach to an irq
> safe genpd also have to be irq safe.
>
> The third patch, adds a generic PM domain for the cpus. GenPD provider can be

drop the ',' 

A GenPD provider...

> specified in the DT and individual cpus that are part of the domain would be
> the domain consumers. A new API pm_cpu_domain_init() has been introduced that
> would initialize the genpd and attach cpus specified as consumers in the DT to
> the domain. 

Hmm, doesn't the of_genpd_* stuff already handle this?

What about non-CPU consumers in the same domain?  

> When the cpus enter their idle state cpu_pm notifications are sent
> out for that cpu. The last cpu to send cpu_pm notification would trigger the
> domain power_off callback and the first cpu to come up would trigger the genpd
> power_on callback.  Generally, the power_off callback is where the caches are
> flushed in preparation for a power down and the domain hardware configured to
> power down when the cpu finishes execution. In the power_on callback, the
> domain hardware is reset to a state to allow cpus to be active or entire idle
> states individually.
>
> A future addition to this feature, could be a new genpd governor for the
> specific case of the cpu domain.  The governor may look up the time available
> to sleep between the last cpu down and the first cpu up, in determining if it
> would be more efficient to just keep the domain powered on.
>
> This patch is based on Ulf's patch for simplifying domain power down states
> [1], which removes a bunch of complexity in genpd, simplifying atomic genpd.

Kevin

[PATCH RFC 0/3] PM / Domains: Generic PM domains for cpus

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> This is an attempt to provide a generic PM domain for cpus, so as to allow the
> cpu domain to be powered off, when all the cpus are in power down state during
> cpuidle. 

You don't use the word "cluster" here, but it might be helpful to
summarize this as using a genpd to model a cluster.  A cluster contains
CPUs, but also additional devices/logic like L2$, GIC, PMUs, floating
point units, CoreSight, etc. etc.

> The rationale behind the change is that newer SoCs can power down the
> cpus for very short sleep times to save on leakage power. The domain which
> usually has the cpus, a second level cache and some peripheral hardware is
> powered by a rail that can also be turned off when the cpus are not in
> use.

This last sentence doesn't read well.  I think you meant something like:

In the domain which has the CPUs, a second level cache and some
peripheral hardware might share the power rail with the CPUs so could
also be turned off when the cpus are not in use.

> Devices for each cpu, L2 and other related blocks could be attached to the
> domain and when there are no uses of these devices (device idle) the domain
> could also be powered off. Generic PM domains provides all the backend needed
> for such an organization.
>
> In the first 2 patches, I make genpd usable in atomic context as well. CPUIdle

s/as well//

> runs with irqs disabled and therefore use of mutexes in the current genpd
> implementation is a limitation. But, not all PM domains need to operate in irq
> safe context, those that need to can specify explicitly the irq safe
> requirement of the genpd at init. Devices and sub-domains that attach to an irq
> safe genpd also have to be irq safe.
>
> The third patch, adds a generic PM domain for the cpus. GenPD provider can be

drop the ',' 

A GenPD provider...

> specified in the DT and individual cpus that are part of the domain would be
> the domain consumers. A new API pm_cpu_domain_init() has been introduced that
> would initialize the genpd and attach cpus specified as consumers in the DT to
> the domain. 

Hmm, doesn't the of_genpd_* stuff already handle this?

What about non-CPU consumers in the same domain?  

> When the cpus enter their idle state cpu_pm notifications are sent
> out for that cpu. The last cpu to send cpu_pm notification would trigger the
> domain power_off callback and the first cpu to come up would trigger the genpd
> power_on callback.  Generally, the power_off callback is where the caches are
> flushed in preparation for a power down and the domain hardware configured to
> power down when the cpu finishes execution. In the power_on callback, the
> domain hardware is reset to a state to allow cpus to be active or entire idle
> states individually.
>
> A future addition to this feature, could be a new genpd governor for the
> specific case of the cpu domain.  The governor may look up the time available
> to sleep between the last cpu down and the first cpu up, in determining if it
> would be more efficient to just keep the domain powered on.
>
> This patch is based on Ulf's patch for simplifying domain power down states
> [1], which removes a bunch of complexity in genpd, simplifying atomic genpd.

Kevin

Re: [PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> In order for domains to be powered on/off in irq locked context, the
> domain locks could either be a spinlock or a mutex, depending on the
> domain. 

I'd drop that sentence.

> In preparation for atomic support, allocate domain data outside

In preparation for supporting IRQ-safe domainss, ...

> the domain locks, so the allocation calls dont have to be context
> sensitive.

OK, but *why* don't these allocations need to be protected by the
locks.  Presumably they were put inside the locks for a reason, so you
should make the case for why the lock protection isn't needed.

Kevin

[PATCH RFC 1/3] PM / Domains: Allocate memory outside domain locks

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> In order for domains to be powered on/off in irq locked context, the
> domain locks could either be a spinlock or a mutex, depending on the
> domain. 

I'd drop that sentence.

> In preparation for atomic support, allocate domain data outside

In preparation for supporting IRQ-safe domainss, ...

> the domain locks, so the allocation calls dont have to be context
> sensitive.

OK, but *why* don't these allocations need to be protected by the
locks.  Presumably they were put inside the locks for a reason, so you
should make the case for why the lock protection isn't needed.

Kevin

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> Power Domains currently support turning on/off only in process context.

Generic Power Domains...

Also Re: $SUBJECT.  s/atomic/IRQ safe/

> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.

That sentence reads the opposite of what you mean.  Rather than "This
restricts X to Y", I think you menant "This prevents X for Y".

> Genpd inherently provides support for devices, domain hierarchy and can

s/domain heirarchy/nesting/

> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. 

s/domain/power domain/

> Multiple such domains could be part of another domain. 

OK, but not important to this change IMO.

> Because mutexes are used to protect and synchronize domain

s/domain/genpd/

> operations and cpu idle operations are inherently atomic, 

Be more specific about "CPU idle operations are inherently atomic", as
it's not obvious that it's true.  I think what you mean is that
the CPUidle paths for entering of low-power idle states is inherently
atomic because interrupts are disabled.

> the use of
> genpd is not possible for runtime suspend and resume of the pm domain.

... so, the use of genpd during the idle path of CPUs is not currently
possible because interrups are disabled in the idle path.

> Replacing the locks to spinlocks would allow cpu domain to be be powered
                     ^^^^
s/to/with/

> off to save power, when all the cpus are powered off.

More accuratly, replacing the locks doesn't allow the domain to be
powered off, rather it allows genpd to be used in the idle path, which 
would allow genpd to be used

> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
>
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
>
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.

This might need some corresponding updates to Documentation/ as well.

Kevin

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> Power Domains currently support turning on/off only in process context.

Generic Power Domains...

Also Re: $SUBJECT.  s/atomic/IRQ safe/

> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.

That sentence reads the opposite of what you mean.  Rather than "This
restricts X to Y", I think you menant "This prevents X for Y".

> Genpd inherently provides support for devices, domain hierarchy and can

s/domain heirarchy/nesting/

> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. 

s/domain/power domain/

> Multiple such domains could be part of another domain. 

OK, but not important to this change IMO.

> Because mutexes are used to protect and synchronize domain

s/domain/genpd/

> operations and cpu idle operations are inherently atomic, 

Be more specific about "CPU idle operations are inherently atomic", as
it's not obvious that it's true.  I think what you mean is that
the CPUidle paths for entering of low-power idle states is inherently
atomic because interrupts are disabled.

> the use of
> genpd is not possible for runtime suspend and resume of the pm domain.

... so, the use of genpd during the idle path of CPUs is not currently
possible because interrups are disabled in the idle path.

> Replacing the locks to spinlocks would allow cpu domain to be be powered
                     ^^^^
s/to/with/

> off to save power, when all the cpus are powered off.

More accuratly, replacing the locks doesn't allow the domain to be
powered off, rather it allows genpd to be used in the idle path, which 
would allow genpd to be used

> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
>
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
>
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.

This might need some corresponding updates to Documentation/ as well.

Kevin

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Wed, Jun 10 2015 at 12:04 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> Power Domains currently support turning on/off only in process context.
>
>Generic Power Domains...
>
>Also Re: $SUBJECT.  s/atomic/IRQ safe/
>
Okay. 

>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>
>That sentence reads the opposite of what you mean.  Rather than "This
>restricts X to Y", I think you menant "This prevents X for Y".
>
Will reword.

>> Genpd inherently provides support for devices, domain hierarchy and can
>
>s/domain heirarchy/nesting/
>
Ok

>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware.
>
>s/domain/power domain/
>
OK

>> Multiple such domains could be part of another domain.
>
>OK, but not important to this change IMO.
>
>> Because mutexes are used to protect and synchronize domain
>
>s/domain/genpd/
>
Ya, genpd is better word.

>> operations and cpu idle operations are inherently atomic,
>
>Be more specific about "CPU idle operations are inherently atomic", as
>it's not obvious that it's true.  I think what you mean is that
>the CPUidle paths for entering of low-power idle states is inherently
>atomic because interrupts are disabled.
>
Will explain

>> the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>
>... so, the use of genpd during the idle path of CPUs is not currently
>possible because interrups are disabled in the idle path.
>
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>                     ^^^^
>s/to/with/
>
Ok

>> off to save power, when all the cpus are powered off.
>
>More accuratly, replacing the locks doesn't allow the domain to be
>powered off, rather it allows genpd to be used in the idle path, which
>would allow genpd to be used
>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>
>This might need some corresponding updates to Documentation/ as well.
>
Agreed. Will add documentation in the next spin.

Thanks for the review Kevin.

-- Lina

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Wed, Jun 10 2015 at 12:04 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> Power Domains currently support turning on/off only in process context.
>
>Generic Power Domains...
>
>Also Re: $SUBJECT.  s/atomic/IRQ safe/
>
Okay. 

>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>
>That sentence reads the opposite of what you mean.  Rather than "This
>restricts X to Y", I think you menant "This prevents X for Y".
>
Will reword.

>> Genpd inherently provides support for devices, domain hierarchy and can
>
>s/domain heirarchy/nesting/
>
Ok

>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware.
>
>s/domain/power domain/
>
OK

>> Multiple such domains could be part of another domain.
>
>OK, but not important to this change IMO.
>
>> Because mutexes are used to protect and synchronize domain
>
>s/domain/genpd/
>
Ya, genpd is better word.

>> operations and cpu idle operations are inherently atomic,
>
>Be more specific about "CPU idle operations are inherently atomic", as
>it's not obvious that it's true.  I think what you mean is that
>the CPUidle paths for entering of low-power idle states is inherently
>atomic because interrupts are disabled.
>
Will explain

>> the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>
>... so, the use of genpd during the idle path of CPUs is not currently
>possible because interrups are disabled in the idle path.
>
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>                     ^^^^
>s/to/with/
>
Ok

>> off to save power, when all the cpus are powered off.
>
>More accuratly, replacing the locks doesn't allow the domain to be
>powered off, rather it allows genpd to be used in the idle path, which
>would allow genpd to be used
>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>
>This might need some corresponding updates to Documentation/ as well.
>
Agreed. Will add documentation in the next spin.

Thanks for the review Kevin.

-- Lina

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> Generally cpus are grouped under a power domain in a SoC. When all cpus
> in the domain are in their power off state, the cpu domain can also be
> powered off. 

How does this relate to a cluster, and why aren't you using that terminolgy?

> Genpd provides the framework for defining cpus as devices
> that are part of a cpu domain.
>
> Introduce support for defining and adding a generic power domain for the
> cpus based on the DT specification of power domain providers and
> consumers.  SoC's that have the cpu domain defined in their DT, can
> setup a genpd with a name and the power_on/power_off callbacks. Calling
> pm_cpu_domain_init() will register the genpd and attach the cpus for
> this domain with the genpd.
>
> CPU_PM notifications for are used to pm_runtime_get_sync() and
> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
> last cpu going down would call the genpd->power_off(). Correspondingly,
> the first cpu up would call the genpd->power_on() callback before
> resuming from idle.

Other patches also mention this genpd being useful to gate power to
non-CPU peripherals on the same power rail.  How are those devices to be
added?

Without seeing the DTs and the init code that might call
pm_cpu_domain_init(), it's hard for me to see how this is intended to be
used.  Could you also include a patch that shows how this is initialized
and the DT additions?  Ideally, it should also show how a non-CPU device
would be included.

Kevin

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> Generally cpus are grouped under a power domain in a SoC. When all cpus
> in the domain are in their power off state, the cpu domain can also be
> powered off. 

How does this relate to a cluster, and why aren't you using that terminolgy?

> Genpd provides the framework for defining cpus as devices
> that are part of a cpu domain.
>
> Introduce support for defining and adding a generic power domain for the
> cpus based on the DT specification of power domain providers and
> consumers.  SoC's that have the cpu domain defined in their DT, can
> setup a genpd with a name and the power_on/power_off callbacks. Calling
> pm_cpu_domain_init() will register the genpd and attach the cpus for
> this domain with the genpd.
>
> CPU_PM notifications for are used to pm_runtime_get_sync() and
> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
> last cpu going down would call the genpd->power_off(). Correspondingly,
> the first cpu up would call the genpd->power_on() callback before
> resuming from idle.

Other patches also mention this genpd being useful to gate power to
non-CPU peripherals on the same power rail.  How are those devices to be
added?

Without seeing the DTs and the init code that might call
pm_cpu_domain_init(), it's hard for me to see how this is intended to be
used.  Could you also include a patch that shows how this is initialized
and the DT additions?  Ideally, it should also show how a non-CPU device
would be included.

Kevin

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Krzysztof Kozlowski]

On 11.06.2015 01:13, Lina Iyer wrote:
> On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Power Domains currently support turning on/off only in process context.
>>> This restricts the usage of PM domains to devices and domains that
>>> could be powered on/off in irq disabled contexts as the mutexes used in
>>> GenPD allows for cpu sleep while waiting for locks.
>>
>> I can find also other use case: currently the power domain with irq_safe
>> devices is always powered on. With the patch it could be powered off (of
>> course if the driver/mach code is irq-safe).
>>
> Yes, absolutely.
> 
>>>
>>> Genpd inherently provides support for devices, domain hierarchy and can
>>> be used to represent cpu clusters like in ARM's big.Little, where, each
>>> cpu cluster is in its domain, with supporting caches and other
>>> peripheral hardware. Multiple such domains could be part of another
>>> domain. Because mutexes are used to protect and synchronize domain
>>> operations and cpu idle operations are inherently atomic, the use of
>>> genpd is not possible for runtime suspend and resume of the pm domain.
>>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>>> off to save power, when all the cpus are powered off.
>>>
>>> However, not all domains can operate in irq-safe contexts and usually
>>> would need to sleep during domain operations. So genpd has to support
>>> both the cases, where the domain is or is not irq-safe. The irq-safe
>>> attribute is therefore domain specific.
>>>
>>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>>> while defining the domain. This determines if the domain should use a
>>> spinlock instead of a mutex. Locking is abstracted through
>>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>>> flag to determine the locking to be used for this domain.
>>>
>>> The restriction this imposes on the domain hierarchy is that subdomains
>>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>>> may continue to have irq-safe devices, but not the other way around.
>>
>> So an irq-safe device can be put in irq-safe subdomain which can be a
>> child of non-irq-safe topdomain?
>>
> Yes, the container need not be irq-safe but the contained need to be
> irqsafe.

I had doubts about possibility of grabbing any lock from irq-safe
subdomain before grabbing lock from non-irq-safe parent. That would mean
that spin_lock is acquired and then mutex. But now after giving some
more thoughts I can't find such case so it seems fine.

Thanks for explanation.

> 
>>>
>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Kevin Hilman <khilman@linaro.org>
>>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>>> ---
>>>  drivers/base/power/domain.c | 200
>>> ++++++++++++++++++++++++++++++++++----------
>>>  include/linux/pm_domain.h   |  11 ++-
>>
>> Documentation should also be reflected.
>>
> Yes, will add.
> 
>>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>>
>>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>>> index dfd7595..8b89d15 100644
>>> --- a/drivers/base/power/domain.c
>>> +++ b/drivers/base/power/domain.c
>>> @@ -50,6 +50,71 @@
>>>  static LIST_HEAD(gpd_list);
>>>  static DEFINE_MUTEX(gpd_list_lock);
>>>
>>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain
>>> *genpd,
>>> +                    unsigned int subclass)
>>> +    __acquires(&genpd->slock)
>>> +{
>>> +    unsigned long flags;
>>> +
>>> +    if (unlikely(subclass > 0))
>>> +        spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>>> +    else
>>> +        spin_lock_irqsave(&genpd->slock, flags);
>>> +
>>> +    genpd->flags = flags;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain
>>> *genpd)
>>> +    __releases(&genpd->slock)
>>> +{
>>> +    spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain
>>> *genpd,
>>> +                    unsigned int subclass)
>>> +    __acquires(&genpd->mlock)
>>> +{
>>> +    if (unlikely(subclass > 0))
>>> +        mutex_lock_nested(&genpd->mlock, subclass);
>>> +    else
>>> +        mutex_lock(&genpd->mlock);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_lock_domain_interruptible_irq(
>>> +                struct generic_pm_domain *genpd)
>>> +    __acquires(&genpd->mlock)
>>> +{
>>> +    return mutex_lock_interruptible(&genpd->mlock);
>>> +}
>>> +
>>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain
>>> *genpd)
>>> +    __releases(&genpd->mlock)
>>> +{
>>> +    mutex_unlock(&genpd->mlock);
>>> +    return 0;
>>> +}
>>> +
>>> +#define genpd_lock_domain(genpd)                \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>>> +            : genpd_lock_domain_irq(genpd, 0))
>>> +
>>> +#define genpd_lock_domain_nested(genpd)                \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd,
>>> SINGLE_DEPTH_NESTING)\
>>> +            : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>>> +
>>> +#define genpd_unlock_domain(genpd)                \
>>> +    (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)    \
>>> +            : genpd_unlock_domain_irq(genpd))
>>> +
>>> +#define genpd_lock_domain_interruptible(genpd)            \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>>> +            : genpd_lock_domain_interruptible_irq(genpd))
>>
>> Why macros? You are not using here benefits of a macro and they are
>> called just like ordinary functions.
>>
> Well, I didnt see a need for a function that might show up in the stack.
> But I have no strong preference either way.

That is actually a valid reason. It is just my personal dislike of
macros so I am fine with your idea.

> 
>> You added "domain" prefix but genpd already contains this. genod_lock(),
>> genpd_lock_nested() etc. should be sufficient, unless there is a
>> conflict, similar name planned or you plan to lock something else
>> (genpd_lock_device?).
>>
> Sigh. Yes, you are right. Its redundant. Will remove.
> 
>>
>>> +
>>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char
>>> *domain_name)
>>>  {
>>>      struct generic_pm_domain *genpd = NULL, *gpd;
>>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain
>>> *genpd)
>>>  {
>>>      int ret;
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      ret = __pm_genpd_poweron(genpd);
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>      return ret;
>>>  }
>>>
>>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct
>>> notifier_block *nb,
>>>          spin_unlock_irq(&dev->power.lock);
>>>
>>>          if (!IS_ERR(genpd)) {
>>> -            mutex_lock(&genpd->lock);
>>> +            genpd_lock_domain(genpd);
>>>              genpd->max_off_time_changed = true;
>>> -            mutex_unlock(&genpd->lock);
>>> +            genpd_unlock_domain(genpd);
>>>          }
>>>
>>>          dev = dev->parent;
>>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct
>>> generic_pm_domain *genpd)
>>>              return -EBUSY;
>>>
>>>          if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>>> -            || pdd->dev->power.irq_safe))
>>> +            || (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>>>              not_suspended++;
>>>      }
>>>
>>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct
>>> work_struct *work)
>>>
>>>      genpd = container_of(work, struct generic_pm_domain,
>>> power_off_work);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      pm_genpd_poweroff(genpd);
>>
>> Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
>> At least in next-20150604 but maybe the patches, which this depends on,
>> changed it?
>>
> Yes. Ulf's patch remvoed that call.

OK, thanks.

> 
>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>  }
>>>
>>>  /**
>>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct
>>> device *dev)
>>>      if (IS_ERR(genpd))
>>>          return -EINVAL;
>>>
>>> -    /*
>>> -     * We can't allow to power off the PM domain if it holds an
>>> irq_safe
>>> -     * device. That's beacuse we use mutexes to protect data while
>>> power
>>> -     * off and on the PM domain, thus we can't execute in atomic
>>> context.
>>> -     */
>>> -    if (dev->power.irq_safe)
>>> +    /* We can't allow to power off a domain that is also not irq
>>> safe. */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>>          return -EBUSY;
>>>
>>>      stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct
>>> device *dev)
>>>          return ret;
>>>      }
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    /*
>>> +     * If power.irq_safe is set, this routine will be run with
>>> interrupts
>>> +     * off, so suspend only if the power domain is irq_safe.
>>> +     */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>> +        return 0;
>>> +
>>> +    genpd_lock_domain(genpd);
>>> +
>>>      genpd->in_progress++;
>>>      pm_genpd_poweroff(genpd);
>>>      genpd->in_progress--;
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      return 0;
>>>  }
>>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct
>>> device *dev)
>>>      if (IS_ERR(genpd))
>>>          return -EINVAL;
>>>
>>> -    /* If power.irq_safe, the PM domain is never powered off. */
>>> -    if (dev->power.irq_safe)
>>> +    /*
>>> +     * If power.irq_safe and domain is not, then
>>> +     * the PM domain is never powered off.
>>> +     */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>>          return genpd_start_dev_no_timing(genpd, dev);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      ret = __pm_genpd_poweron(genpd);
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (ret)
>>>          return ret;
>>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>>      if (resume_needed(dev, genpd))
>>>          pm_runtime_resume(dev);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>
>>>      if (genpd->prepared_count++ == 0) {
>>>          genpd->suspended_count = 0;
>>>          genpd->suspend_power_off = genpd->status ==
>>> GPD_STATE_POWER_OFF;
>>>      }
>>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (genpd->suspend_power_off) {
>>>          pm_runtime_put_noidle(dev);
>>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>>
>>>      ret = pm_generic_prepare(dev);
>>>      if (ret) {
>>> -        mutex_lock(&genpd->lock);
>>> +        genpd_lock_domain(genpd);
>>>
>>>          if (--genpd->prepared_count == 0)
>>>              genpd->suspend_power_off = false;
>>>
>>> -        mutex_unlock(&genpd->lock);
>>> +        genpd_unlock_domain(genpd);
>>>          pm_runtime_enable(dev);
>>>      }
>>>
>>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device
>>> *dev)
>>>      if (IS_ERR(genpd))
>>>          return;
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>
>>>      run_complete = !genpd->suspend_power_off;
>>>      if (--genpd->prepared_count == 0)
>>>          genpd->suspend_power_off = false;
>>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (run_complete) {
>>>          pm_generic_complete(dev);
>>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct
>>> generic_pm_domain *genpd, struct device *dev,
>>>      if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>>          return -EINVAL;
>>>
>>> +    /* Devices in an IRQ safe PM Domain have to be irq safe too */
>>
>> Why? Can you add this information here? Previously there was a reason in
>> case of irq_safe devices which you removed leaving only policy.
>>
> Sorry, your question is not clear to me.
> I believe this is a new requirement that enforces the contained devices
> of an irq-safe domain to be irq-safe as well.

What I wanted to say is that it would be nice if comment explained why
domain have to be IRQ safe too. Without this "WHY" answer the comment is
quite redundant - the "if" statement is obvious. But the "WHY" is not
such obvious.

Previous comments in few places mentioned the answer:
/*
 * We can't allow to power off the PM domain if it holds an irq_safe
 * device. That's beacuse we use mutexes to protect data while power
 * off and on the PM domain, thus we can't execute in atomic context.
 */

Best regards,
Krzysztof

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Krzysztof Kozlowski]

On 11.06.2015 01:13, Lina Iyer wrote:
> On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Power Domains currently support turning on/off only in process context.
>>> This restricts the usage of PM domains to devices and domains that
>>> could be powered on/off in irq disabled contexts as the mutexes used in
>>> GenPD allows for cpu sleep while waiting for locks.
>>
>> I can find also other use case: currently the power domain with irq_safe
>> devices is always powered on. With the patch it could be powered off (of
>> course if the driver/mach code is irq-safe).
>>
> Yes, absolutely.
> 
>>>
>>> Genpd inherently provides support for devices, domain hierarchy and can
>>> be used to represent cpu clusters like in ARM's big.Little, where, each
>>> cpu cluster is in its domain, with supporting caches and other
>>> peripheral hardware. Multiple such domains could be part of another
>>> domain. Because mutexes are used to protect and synchronize domain
>>> operations and cpu idle operations are inherently atomic, the use of
>>> genpd is not possible for runtime suspend and resume of the pm domain.
>>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>>> off to save power, when all the cpus are powered off.
>>>
>>> However, not all domains can operate in irq-safe contexts and usually
>>> would need to sleep during domain operations. So genpd has to support
>>> both the cases, where the domain is or is not irq-safe. The irq-safe
>>> attribute is therefore domain specific.
>>>
>>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>>> while defining the domain. This determines if the domain should use a
>>> spinlock instead of a mutex. Locking is abstracted through
>>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>>> flag to determine the locking to be used for this domain.
>>>
>>> The restriction this imposes on the domain hierarchy is that subdomains
>>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>>> may continue to have irq-safe devices, but not the other way around.
>>
>> So an irq-safe device can be put in irq-safe subdomain which can be a
>> child of non-irq-safe topdomain?
>>
> Yes, the container need not be irq-safe but the contained need to be
> irqsafe.

I had doubts about possibility of grabbing any lock from irq-safe
subdomain before grabbing lock from non-irq-safe parent. That would mean
that spin_lock is acquired and then mutex. But now after giving some
more thoughts I can't find such case so it seems fine.

Thanks for explanation.

> 
>>>
>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Kevin Hilman <khilman@linaro.org>
>>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>>> ---
>>>  drivers/base/power/domain.c | 200
>>> ++++++++++++++++++++++++++++++++++----------
>>>  include/linux/pm_domain.h   |  11 ++-
>>
>> Documentation should also be reflected.
>>
> Yes, will add.
> 
>>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>>
>>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>>> index dfd7595..8b89d15 100644
>>> --- a/drivers/base/power/domain.c
>>> +++ b/drivers/base/power/domain.c
>>> @@ -50,6 +50,71 @@
>>>  static LIST_HEAD(gpd_list);
>>>  static DEFINE_MUTEX(gpd_list_lock);
>>>
>>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain
>>> *genpd,
>>> +                    unsigned int subclass)
>>> +    __acquires(&genpd->slock)
>>> +{
>>> +    unsigned long flags;
>>> +
>>> +    if (unlikely(subclass > 0))
>>> +        spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>>> +    else
>>> +        spin_lock_irqsave(&genpd->slock, flags);
>>> +
>>> +    genpd->flags = flags;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain
>>> *genpd)
>>> +    __releases(&genpd->slock)
>>> +{
>>> +    spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain
>>> *genpd,
>>> +                    unsigned int subclass)
>>> +    __acquires(&genpd->mlock)
>>> +{
>>> +    if (unlikely(subclass > 0))
>>> +        mutex_lock_nested(&genpd->mlock, subclass);
>>> +    else
>>> +        mutex_lock(&genpd->mlock);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static inline int genpd_lock_domain_interruptible_irq(
>>> +                struct generic_pm_domain *genpd)
>>> +    __acquires(&genpd->mlock)
>>> +{
>>> +    return mutex_lock_interruptible(&genpd->mlock);
>>> +}
>>> +
>>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain
>>> *genpd)
>>> +    __releases(&genpd->mlock)
>>> +{
>>> +    mutex_unlock(&genpd->mlock);
>>> +    return 0;
>>> +}
>>> +
>>> +#define genpd_lock_domain(genpd)                \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>>> +            : genpd_lock_domain_irq(genpd, 0))
>>> +
>>> +#define genpd_lock_domain_nested(genpd)                \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd,
>>> SINGLE_DEPTH_NESTING)\
>>> +            : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>>> +
>>> +#define genpd_unlock_domain(genpd)                \
>>> +    (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)    \
>>> +            : genpd_unlock_domain_irq(genpd))
>>> +
>>> +#define genpd_lock_domain_interruptible(genpd)            \
>>> +    (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>>> +            : genpd_lock_domain_interruptible_irq(genpd))
>>
>> Why macros? You are not using here benefits of a macro and they are
>> called just like ordinary functions.
>>
> Well, I didnt see a need for a function that might show up in the stack.
> But I have no strong preference either way.

That is actually a valid reason. It is just my personal dislike of
macros so I am fine with your idea.

> 
>> You added "domain" prefix but genpd already contains this. genod_lock(),
>> genpd_lock_nested() etc. should be sufficient, unless there is a
>> conflict, similar name planned or you plan to lock something else
>> (genpd_lock_device?).
>>
> Sigh. Yes, you are right. Its redundant. Will remove.
> 
>>
>>> +
>>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char
>>> *domain_name)
>>>  {
>>>      struct generic_pm_domain *genpd = NULL, *gpd;
>>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain
>>> *genpd)
>>>  {
>>>      int ret;
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      ret = __pm_genpd_poweron(genpd);
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>      return ret;
>>>  }
>>>
>>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct
>>> notifier_block *nb,
>>>          spin_unlock_irq(&dev->power.lock);
>>>
>>>          if (!IS_ERR(genpd)) {
>>> -            mutex_lock(&genpd->lock);
>>> +            genpd_lock_domain(genpd);
>>>              genpd->max_off_time_changed = true;
>>> -            mutex_unlock(&genpd->lock);
>>> +            genpd_unlock_domain(genpd);
>>>          }
>>>
>>>          dev = dev->parent;
>>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct
>>> generic_pm_domain *genpd)
>>>              return -EBUSY;
>>>
>>>          if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>>> -            || pdd->dev->power.irq_safe))
>>> +            || (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>>>              not_suspended++;
>>>      }
>>>
>>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct
>>> work_struct *work)
>>>
>>>      genpd = container_of(work, struct generic_pm_domain,
>>> power_off_work);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      pm_genpd_poweroff(genpd);
>>
>> Ipm_genpd_poweroff() calls __pm_genpd_save_device() which grabs mutex.
>> At least in next-20150604 but maybe the patches, which this depends on,
>> changed it?
>>
> Yes. Ulf's patch remvoed that call.

OK, thanks.

> 
>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>  }
>>>
>>>  /**
>>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct
>>> device *dev)
>>>      if (IS_ERR(genpd))
>>>          return -EINVAL;
>>>
>>> -    /*
>>> -     * We can't allow to power off the PM domain if it holds an
>>> irq_safe
>>> -     * device. That's beacuse we use mutexes to protect data while
>>> power
>>> -     * off and on the PM domain, thus we can't execute in atomic
>>> context.
>>> -     */
>>> -    if (dev->power.irq_safe)
>>> +    /* We can't allow to power off a domain that is also not irq
>>> safe. */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>>          return -EBUSY;
>>>
>>>      stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct
>>> device *dev)
>>>          return ret;
>>>      }
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    /*
>>> +     * If power.irq_safe is set, this routine will be run with
>>> interrupts
>>> +     * off, so suspend only if the power domain is irq_safe.
>>> +     */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>> +        return 0;
>>> +
>>> +    genpd_lock_domain(genpd);
>>> +
>>>      genpd->in_progress++;
>>>      pm_genpd_poweroff(genpd);
>>>      genpd->in_progress--;
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      return 0;
>>>  }
>>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct
>>> device *dev)
>>>      if (IS_ERR(genpd))
>>>          return -EINVAL;
>>>
>>> -    /* If power.irq_safe, the PM domain is never powered off. */
>>> -    if (dev->power.irq_safe)
>>> +    /*
>>> +     * If power.irq_safe and domain is not, then
>>> +     * the PM domain is never powered off.
>>> +     */
>>> +    if (dev->power.irq_safe && !genpd->irq_safe)
>>>          return genpd_start_dev_no_timing(genpd, dev);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>      ret = __pm_genpd_poweron(genpd);
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (ret)
>>>          return ret;
>>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>>      if (resume_needed(dev, genpd))
>>>          pm_runtime_resume(dev);
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>
>>>      if (genpd->prepared_count++ == 0) {
>>>          genpd->suspended_count = 0;
>>>          genpd->suspend_power_off = genpd->status ==
>>> GPD_STATE_POWER_OFF;
>>>      }
>>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (genpd->suspend_power_off) {
>>>          pm_runtime_put_noidle(dev);
>>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>>
>>>      ret = pm_generic_prepare(dev);
>>>      if (ret) {
>>> -        mutex_lock(&genpd->lock);
>>> +        genpd_lock_domain(genpd);
>>>
>>>          if (--genpd->prepared_count == 0)
>>>              genpd->suspend_power_off = false;
>>>
>>> -        mutex_unlock(&genpd->lock);
>>> +        genpd_unlock_domain(genpd);
>>>          pm_runtime_enable(dev);
>>>      }
>>>
>>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device
>>> *dev)
>>>      if (IS_ERR(genpd))
>>>          return;
>>>
>>> -    mutex_lock(&genpd->lock);
>>> +    genpd_lock_domain(genpd);
>>>
>>>      run_complete = !genpd->suspend_power_off;
>>>      if (--genpd->prepared_count == 0)
>>>          genpd->suspend_power_off = false;
>>>
>>> -    mutex_unlock(&genpd->lock);
>>> +    genpd_unlock_domain(genpd);
>>>
>>>      if (run_complete) {
>>>          pm_generic_complete(dev);
>>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct
>>> generic_pm_domain *genpd, struct device *dev,
>>>      if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>>          return -EINVAL;
>>>
>>> +    /* Devices in an IRQ safe PM Domain have to be irq safe too */
>>
>> Why? Can you add this information here? Previously there was a reason in
>> case of irq_safe devices which you removed leaving only policy.
>>
> Sorry, your question is not clear to me.
> I believe this is a new requirement that enforces the contained devices
> of an irq-safe domain to be irq-safe as well.

What I wanted to say is that it would be nice if comment explained why
domain have to be IRQ safe too. Without this "WHY" answer the comment is
quite redundant - the "if" statement is obvious. But the "WHY" is not
such obvious.

Previous comments in few places mentioned the answer:
/*
 * We can't allow to power off the PM domain if it holds an irq_safe
 * device. That's beacuse we use mutexes to protect data while power
 * off and on the PM domain, thus we can't execute in atomic context.
 */

Best regards,
Krzysztof

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

On 11.06.2015 01:57, Lina Iyer wrote:
> On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state,
>>
>> What do you exactly mean here by "CPU in power off state"? How does it
>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>
> Both cpuidle and hotplug could end with with core being powered down at
> the platform driver or at PSCI (on V8). It does not matter which of
> these two frameworks resulted in the cpu being powered off. But, if all
> cpus in the domain are powered off, then the domain could be powered off
> as well. This is the premise of this change. It is probably easier to
> power off the domain when the cores in that domain/cluster have been
> hotplugged off. It saves power to turn off the domain at that time, but
> more power savings can be achieved if the domain could also be powered
> off during cpuidle. Hotplug is not a common occurance, while cpuidle is.

OK, it answers my questions, thanks.

> 
>>> the cpu domain can also be
>>> powered off. Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>
>> The problem which is solved looks to me like the same problem which
>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>> off) can be entered when whole cluster is idle or other CPUs in cluster
>> are powered off completely.
>>
>> It seems a little like duplicating the effort around coupled cpuidle.
>>
> I see where are you are going with this, but genpd solution is not
> exactly a duplicate of the solution.
> 
> Couple state is used to put the cpus in a deeper sleep state, which
> could also result in powering off the domain. Coupled cpuidle is a
> cpuidle mechanism for choosing a deeper sleep mode on certain hardware
> that can only enter such a mode when all cpus cooperate.
> 
> This patch attempts to describe the backend of a cpu domain. CPUs are
> responsible for individual cpuidle states, cpus do enter their
> recommended deepest idle state at the time of no activity. A cpu-domain
> could be comprised of cpus, and other devices like GIC, busses etc, that
> all need to idle before the domain can be powered off. This patch does
> not dictate which idle state any those devices should enter, or
> coordinate the idle states between devices. But, if cpus, choose to
> power down, then this patch recognizes that and reduces the reference
> usage count on the domain. Only when all devices in the domain remove
> their usage count, will the domain be powered off.

It would be nice to see the usage of this patch in cpuidle driver or
platform code but I think I get the idea.

Actually I like the approach.
I am thinking how to utilize it to replace coupled cpuidle for our case.
In our case we use coupled cpuidle because the SoC can be put in low
power mode only if non-boot CPUs are powered down.

However in our case:
1. Some other devices (buses, clocks) also should be idle. This would
perfectly match with this patch and with runtime PM.

2. Some non-boot idle CPU could power itself down but it cannot wake up.
Only the alive CPU can wake others. This probably means that we cannot
provide a cpuidle driver which will power off unused cores and then, if
boot CPU is idle, disable the CPU power domain by entering to low power
mode.

Anyway, as I said, I like the approach.


> There are two things this patch provides -
> 
> i. A generic way to initialize a genpd specifically for cpus. (The
> platform specifies the relation between a cpu and its domain in the DT
> and provides the memory for the genpd structure)
> 
> ii. On behalf of a platform, we track when the cpus power up and down
> and use runtime_get and runtime_put on the genpd.
> 
> Unlike coupled cpuidle, individual cpu idle state is not manipulated.
> Coupled cpuidle does not care if the domain is powered off, it is used
> to allow a certain C-state for the cpu, based on the idleness of other
> cpus in that cluster. The focus of the series is powering down the
> domain when the devices (cpus included) are powered off. You could see
> this patch as a cpu-pm and runtime-pm interface layer.
> 
> Hope that helps.
> 
> Thanks,
> Lina
> 

Best regards,
Krzysztof

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

On 11.06.2015 01:57, Lina Iyer wrote:
> On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state,
>>
>> What do you exactly mean here by "CPU in power off state"? How does it
>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>
> Both cpuidle and hotplug could end with with core being powered down at
> the platform driver or at PSCI (on V8). It does not matter which of
> these two frameworks resulted in the cpu being powered off. But, if all
> cpus in the domain are powered off, then the domain could be powered off
> as well. This is the premise of this change. It is probably easier to
> power off the domain when the cores in that domain/cluster have been
> hotplugged off. It saves power to turn off the domain at that time, but
> more power savings can be achieved if the domain could also be powered
> off during cpuidle. Hotplug is not a common occurance, while cpuidle is.

OK, it answers my questions, thanks.

> 
>>> the cpu domain can also be
>>> powered off. Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>
>> The problem which is solved looks to me like the same problem which
>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>> off) can be entered when whole cluster is idle or other CPUs in cluster
>> are powered off completely.
>>
>> It seems a little like duplicating the effort around coupled cpuidle.
>>
> I see where are you are going with this, but genpd solution is not
> exactly a duplicate of the solution.
> 
> Couple state is used to put the cpus in a deeper sleep state, which
> could also result in powering off the domain. Coupled cpuidle is a
> cpuidle mechanism for choosing a deeper sleep mode on certain hardware
> that can only enter such a mode when all cpus cooperate.
> 
> This patch attempts to describe the backend of a cpu domain. CPUs are
> responsible for individual cpuidle states, cpus do enter their
> recommended deepest idle state at the time of no activity. A cpu-domain
> could be comprised of cpus, and other devices like GIC, busses etc, that
> all need to idle before the domain can be powered off. This patch does
> not dictate which idle state any those devices should enter, or
> coordinate the idle states between devices. But, if cpus, choose to
> power down, then this patch recognizes that and reduces the reference
> usage count on the domain. Only when all devices in the domain remove
> their usage count, will the domain be powered off.

It would be nice to see the usage of this patch in cpuidle driver or
platform code but I think I get the idea.

Actually I like the approach.
I am thinking how to utilize it to replace coupled cpuidle for our case.
In our case we use coupled cpuidle because the SoC can be put in low
power mode only if non-boot CPUs are powered down.

However in our case:
1. Some other devices (buses, clocks) also should be idle. This would
perfectly match with this patch and with runtime PM.

2. Some non-boot idle CPU could power itself down but it cannot wake up.
Only the alive CPU can wake others. This probably means that we cannot
provide a cpuidle driver which will power off unused cores and then, if
boot CPU is idle, disable the CPU power domain by entering to low power
mode.

Anyway, as I said, I like the approach.


> There are two things this patch provides -
> 
> i. A generic way to initialize a genpd specifically for cpus. (The
> platform specifies the relation between a cpu and its domain in the DT
> and provides the memory for the genpd structure)
> 
> ii. On behalf of a platform, we track when the cpus power up and down
> and use runtime_get and runtime_put on the genpd.
> 
> Unlike coupled cpuidle, individual cpu idle state is not manipulated.
> Coupled cpuidle does not care if the domain is powered off, it is used
> to allow a certain C-state for the cpu, based on the idleness of other
> cpus in that cluster. The focus of the series is powering down the
> domain when the devices (cpus included) are powered off. You could see
> this patch as a cpu-pm and runtime-pm interface layer.
> 
> Hope that helps.
> 
> Thanks,
> Lina
> 

Best regards,
Krzysztof

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

On 11.06.2015 02:01, Kevin Hilman wrote:
> Krzysztof Kozlowski <k.kozlowski@samsung.com> writes:
> 
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state,
>>
>> What do you exactly mean here by "CPU in power off state"? How does it
>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>
>>> the cpu domain can also be
>>> powered off. Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>
>> The problem which is solved looks to me like the same problem which
>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>> off) can be entered when whole cluster is idle or other CPUs in cluster
>> are powered off completely.
>>
>> It seems a little like duplicating the effort around coupled cpuidle.
> 
> Yes, it duplicates some aspects of coupled idle states, but coupled
> states have their own limitations:
> 
> - only handles CPUs, not other devices sharing a power rail (e.g. L2$,
>   GIC, floating point unit, CoreSight, etc. etc.)
> 
> - not scaling well past 2 CPUs
> 
> - doesn't handle clusters: While this series only addresses CPUs
>   currently, the approach can be extended.  Because genpd handles nested
>   domains, the could be used to model clusters as well.

Right. I agree with your explanation. I am just thinking how to utilize
this for Exynos deep sleep modes which now we implement using coupled
cpuidle.

Anyway I like the idea!

Best regards,
Krzysztof

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Krzysztof Kozlowski]

On 11.06.2015 02:01, Kevin Hilman wrote:
> Krzysztof Kozlowski <k.kozlowski@samsung.com> writes:
> 
>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state,
>>
>> What do you exactly mean here by "CPU in power off state"? How does it
>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>
>>> the cpu domain can also be
>>> powered off. Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>
>> The problem which is solved looks to me like the same problem which
>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>> off) can be entered when whole cluster is idle or other CPUs in cluster
>> are powered off completely.
>>
>> It seems a little like duplicating the effort around coupled cpuidle.
> 
> Yes, it duplicates some aspects of coupled idle states, but coupled
> states have their own limitations:
> 
> - only handles CPUs, not other devices sharing a power rail (e.g. L2$,
>   GIC, floating point unit, CoreSight, etc. etc.)
> 
> - not scaling well past 2 CPUs
> 
> - doesn't handle clusters: While this series only addresses CPUs
>   currently, the approach can be extended.  Because genpd handles nested
>   domains, the could be used to model clusters as well.

Right. I agree with your explanation. I am just thinking how to utilize
this for Exynos deep sleep modes which now we implement using coupled
cpuidle.

Anyway I like the idea!

Best regards,
Krzysztof

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Ulf Hansson]

On 5 June 2015 at 00:29, Lina Iyer <lina.iyer@linaro.org> wrote:
> Power Domains currently support turning on/off only in process context.
> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.
>
> Genpd inherently provides support for devices, domain hierarchy and can
> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. Multiple such domains could be part of another
> domain. Because mutexes are used to protect and synchronize domain
> operations and cpu idle operations are inherently atomic, the use of
> genpd is not possible for runtime suspend and resume of the pm domain.
> Replacing the locks to spinlocks would allow cpu domain to be be powered
> off to save power, when all the cpus are powered off.
>
> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
>
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
>
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.
>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>  include/linux/pm_domain.h   |  11 ++-
>  2 files changed, 164 insertions(+), 47 deletions(-)
>
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index dfd7595..8b89d15 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -50,6 +50,71 @@
>  static LIST_HEAD(gpd_list);
>  static DEFINE_MUTEX(gpd_list_lock);
>
> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,

Nitpick:

I would prefer a bit shorter names of all these new locking functions.
In general, I think you can remove "domain" from the names. In this
case it would mean genpd_lock_noirq().

> +                                       unsigned int subclass)
> +       __acquires(&genpd->slock)
> +{
> +       unsigned long flags;
> +
> +       if (unlikely(subclass > 0))
> +               spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
> +       else
> +               spin_lock_irqsave(&genpd->slock, flags);
> +
> +       genpd->flags = flags;

This is wrong, it should be genpd->lock_flags that you assign.

BTW, can't you assign the genpd->lock_flags immediately when call
spin_lock_irqsave*(), instead of keeping a local copy?

> +
> +       return 0;
> +}
> +
> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
> +       __releases(&genpd->slock)
> +{
> +       spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
> +       return 0;
> +}
> +
> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
> +                                       unsigned int subclass)
> +       __acquires(&genpd->mlock)
> +{
> +       if (unlikely(subclass > 0))
> +               mutex_lock_nested(&genpd->mlock, subclass);
> +       else
> +               mutex_lock(&genpd->mlock);
> +
> +       return 0;
> +}
> +
> +static inline int genpd_lock_domain_interruptible_irq(
> +                               struct generic_pm_domain *genpd)
> +       __acquires(&genpd->mlock)
> +{
> +       return mutex_lock_interruptible(&genpd->mlock);
> +}
> +
> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
> +       __releases(&genpd->mlock)
> +{
> +       mutex_unlock(&genpd->mlock);
> +       return 0;
> +}
> +
> +#define genpd_lock_domain(genpd)                               \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
> +                       : genpd_lock_domain_irq(genpd, 0))
> +
> +#define genpd_lock_domain_nested(genpd)                                \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
> +                       : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
> +
> +#define genpd_unlock_domain(genpd)                             \
> +       (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)     \
> +                       : genpd_unlock_domain_irq(genpd))
> +
> +#define genpd_lock_domain_interruptible(genpd)                 \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
> +                       : genpd_lock_domain_interruptible_irq(genpd))

In general I don't like macros (as Krzysztof).

In this case I also don't see the benfit, could you consider to
convert to in-line functions instead?

> +
>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>  {
>         struct generic_pm_domain *genpd = NULL, *gpd;
> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>  {
>         int ret;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         ret = __pm_genpd_poweron(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>  }
>
> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>                 spin_unlock_irq(&dev->power.lock);
>
>                 if (!IS_ERR(genpd)) {
> -                       mutex_lock(&genpd->lock);
> +                       genpd_lock_domain(genpd);
>                         genpd->max_off_time_changed = true;
> -                       mutex_unlock(&genpd->lock);
> +                       genpd_unlock_domain(genpd);
>                 }
>
>                 dev = dev->parent;
> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>                         return -EBUSY;
>
>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
> -                   || pdd->dev->power.irq_safe))
> +                       || (pdd->dev->power.irq_safe && !genpd->irq_safe)))

This deserves a comment in the code.

>                         not_suspended++;
>         }
>
> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>
>         genpd = container_of(work, struct generic_pm_domain, power_off_work);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         pm_genpd_poweroff(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>  }
>
>  /**
> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>         if (IS_ERR(genpd))
>                 return -EINVAL;
>
> -       /*
> -        * We can't allow to power off the PM domain if it holds an irq_safe
> -        * device. That's beacuse we use mutexes to protect data while power
> -        * off and on the PM domain, thus we can't execute in atomic context.
> -        */
> -       if (dev->power.irq_safe)
> +       /* We can't allow to power off a domain that is also not irq safe. */

I wouldn't mind to keep some more pieces of the earlier comment, since
it explains a bit more of the *why*. Could you try to rephrase this
comment in that regard?

> +       if (dev->power.irq_safe && !genpd->irq_safe)

This looks correct...

>                 return -EBUSY;
>
>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>                 return ret;
>         }
>
> -       mutex_lock(&genpd->lock);
> +       /*
> +        * If power.irq_safe is set, this routine will be run with interrupts
> +        * off, so suspend only if the power domain is irq_safe.
> +        */
> +       if (dev->power.irq_safe && !genpd->irq_safe)
> +               return 0;

... but this doesn't. You have already returned -EBUSY above for this case.

> +
> +       genpd_lock_domain(genpd);
> +
>         genpd->in_progress++;
>         pm_genpd_poweroff(genpd);
>         genpd->in_progress--;
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return 0;
>  }
> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>         if (IS_ERR(genpd))
>                 return -EINVAL;
>
> -       /* If power.irq_safe, the PM domain is never powered off. */
> -       if (dev->power.irq_safe)
> +       /*
> +        * If power.irq_safe and domain is not, then

Let's take the opportunity to rephrase this comment a bit.

Perhaps something along the lines: "As we don't power off a non IRQ
safe PM domain which holds an IRQ safe device, we don't need to
restore the power to it."

> +        * the PM domain is never powered off.
> +        */
> +       if (dev->power.irq_safe && !genpd->irq_safe)
>                 return genpd_start_dev_no_timing(genpd, dev);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         ret = __pm_genpd_poweron(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (ret)
>                 return ret;
> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>         if (resume_needed(dev, genpd))
>                 pm_runtime_resume(dev);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count++ == 0) {
>                 genpd->suspended_count = 0;
>                 genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>         }
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (genpd->suspend_power_off) {
>                 pm_runtime_put_noidle(dev);
> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>
>         ret = pm_generic_prepare(dev);
>         if (ret) {
> -               mutex_lock(&genpd->lock);
> +               genpd_lock_domain(genpd);
>
>                 if (--genpd->prepared_count == 0)
>                         genpd->suspend_power_off = false;
>
> -               mutex_unlock(&genpd->lock);
> +               genpd_unlock_domain(genpd);
>                 pm_runtime_enable(dev);
>         }
>
> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>         if (IS_ERR(genpd))
>                 return;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         run_complete = !genpd->suspend_power_off;
>         if (--genpd->prepared_count == 0)
>                 genpd->suspend_power_off = false;
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (run_complete) {
>                 pm_generic_complete(dev);
> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>                 return -EINVAL;
>
> +       /* Devices in an IRQ safe PM Domain have to be irq safe too */

Let's be consistent and stick with IRQ in upper case letters for the
comments. I think there are other places in the patch which needs
update as well.

> +       if (genpd->irq_safe && !dev->power.irq_safe) {
> +               dev_warn(dev,

dev_err(), or perhaps even WARN() ?

> +                       "Devices in an irq-safe domain have to be irq safe.\n");
> +               return -EINVAL;
> +       }
> +
>         gpd_data = genpd_alloc_dev_data(dev, genpd, td);
>         if (IS_ERR(gpd_data))
>                 return PTR_ERR(gpd_data);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count > 0) {
>                 ret = -EAGAIN;
> @@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>         list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (ret)
>                 genpd_free_dev_data(dev, gpd_data);
> @@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>         gpd_data = to_gpd_data(pdd);
>         dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count > 0) {
>                 ret = -EAGAIN;
> @@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>
>         list_del_init(&pdd->list_node);
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         genpd_free_dev_data(dev, gpd_data);
>
>         return 0;
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         dev_pm_qos_add_notifier(dev, &gpd_data->nb);
>
>         return ret;
> @@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>             || genpd == subdomain)
>                 return -EINVAL;
>
> +       /*
> +        * If the domain can be powered on/off in an irq safe
> +        * context, ensure that the subdomain can also be
> +        * powered on/off in that context.
> +        */

Have you considered if/how to address this limitation?

Just to be clear, for now I think this limitation is perfectly okay,
but going forward we might see a need to mix non IRQ safe domains with
IRQ safe domains, via sub-domains.

> +       if (genpd->irq_safe && !subdomain->irq_safe) {
> +               pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
> +                               subdomain->name, genpd->name);

Similar comment as earlier. Perhaps pr_err() or WARN() instead.

> +               return -EINVAL;
> +       }
> +
>         link = kzalloc(sizeof(*link), GFP_KERNEL);
>         if (!link)
>                 return -ENOMEM;
>
> -       mutex_lock(&genpd->lock);
> -       mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
> +       genpd_lock_domain(genpd);
> +       genpd_lock_domain_nested(subdomain);
>
>         if (genpd->status == GPD_STATE_POWER_OFF
>             &&  subdomain->status != GPD_STATE_POWER_OFF) {
> @@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>                 genpd_sd_counter_inc(genpd);
>
>   out:
> -       mutex_unlock(&subdomain->lock);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(subdomain);
> +       genpd_unlock_domain(genpd);
>
>         return ret;
>  }
> @@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
>                 return -EINVAL;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         list_for_each_entry(link, &genpd->master_links, master_node) {
>                 if (link->slave != subdomain)
>                         continue;
>
> -               mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
> +               genpd_lock_domain_nested(subdomain);
>
>                 list_del(&link->master_node);
>                 list_del(&link->slave_node);
> @@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>                 if (subdomain->status != GPD_STATE_POWER_OFF)
>                         genpd_sd_counter_dec(genpd);
>
> -               mutex_unlock(&subdomain->lock);
> +               genpd_unlock_domain(subdomain);
>
>                 ret = 0;
>                 break;
>         }
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return ret;
>  }
> @@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>         if (IS_ERR_OR_NULL(genpd) || state < 0)
>                 return -EINVAL;
>
> +       if (genpd->irq_safe) {
> +               pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
> +                               genpd->name);

Here you have pr_err(), which is okay but not consistent. Also, I
wonder if we should use a WARN() instead?

> +               return -EINVAL;
> +       }
> +
>         cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>         if (!cpuidle_data)
>                 return -ENOMEM;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->cpuidle_data) {
>                 ret = -EEXIST;
> @@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>         genpd_recalc_cpu_exit_latency(genpd);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>
>   err:
> @@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>         if (IS_ERR_OR_NULL(genpd))
>                 return -EINVAL;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         cpuidle_data = genpd->cpuidle_data;
>         if (!cpuidle_data) {
> @@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>         kfree(cpuidle_data);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>  }
>
> @@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
>         return cb ? cb(dev) : 0;
>  }
>
> +static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
> +{
> +       if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
> +               spin_lock_init(&genpd->slock);
> +               genpd->irq_safe = true;

Copying the result from "genpd->flags & GENPD_FLAG_IRQ_SAFE" into the
irq_safe bool, seems like a "micro" optimization. How about only using
the genpd->flags instead?

> +       } else {
> +               mutex_init(&genpd->mlock);
> +               genpd->irq_safe = false;
> +       }
> +}
> +
>  /**
>   * pm_genpd_init - Initialize a generic I/O PM domain object.
>   * @genpd: PM domain object to initialize.
> @@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
>         INIT_LIST_HEAD(&genpd->master_links);
>         INIT_LIST_HEAD(&genpd->slave_links);
>         INIT_LIST_HEAD(&genpd->dev_list);
> -       mutex_init(&genpd->lock);
> +       genpd_lock_domain_init(genpd);
>         genpd->gov = gov;
>         INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
>         genpd->in_progress = 0;
> @@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>         struct gpd_link *link;
>         int ret;
>
> -       ret = mutex_lock_interruptible(&genpd->lock);
> +       ret = genpd_lock_domain_interruptible(genpd);
>         if (ret)
>                 return -ERESTARTSYS;
>
> @@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
>         }
>
>         list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
> -               kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
> +               kobj_path = kobject_get_path(&pm_data->dev->kobj,
> +                               genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
>                 if (kobj_path == NULL)
>                         continue;
>
> @@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>
>         seq_puts(s, "\n");
>  exit:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return 0;
>  }
> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
> index b2725e6..dc7cb53 100644
> --- a/include/linux/pm_domain.h
> +++ b/include/linux/pm_domain.h
> @@ -16,9 +16,11 @@
>  #include <linux/of.h>
>  #include <linux/notifier.h>
>  #include <linux/cpuidle.h>
> +#include <linux/spinlock.h>
>
>  /* Defines used for the flags field in the struct generic_pm_domain */
>  #define GENPD_FLAG_PM_CLK      (1U << 0) /* PM domain uses PM clk */
> +#define GENPD_FLAG_IRQ_SAFE    (1U << 1) /* PM domain operates in atomic */
>
>  enum gpd_status {
>         GPD_STATE_ACTIVE = 0,   /* PM domain is active */
> @@ -49,7 +51,6 @@ struct generic_pm_domain {
>         struct list_head master_links;  /* Links with PM domain as a master */
>         struct list_head slave_links;   /* Links with PM domain as a slave */
>         struct list_head dev_list;      /* List of devices */
> -       struct mutex lock;
>         struct dev_power_governor *gov;
>         struct work_struct power_off_work;
>         const char *name;
> @@ -74,6 +75,14 @@ struct generic_pm_domain {
>         void (*detach_dev)(struct generic_pm_domain *domain,
>                            struct device *dev);
>         unsigned int flags;             /* Bit field of configs for genpd */
> +       bool irq_safe;
> +       union {
> +               struct mutex mlock;
> +               struct {
> +                       spinlock_t slock;
> +                       unsigned long lock_flags;
> +               };
> +       };
>  };
>
>  static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
> --
> 2.1.4
>

Kind regards
Uffe

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Ulf Hansson]

On 5 June 2015 at 00:29, Lina Iyer <lina.iyer@linaro.org> wrote:
> Power Domains currently support turning on/off only in process context.
> This restricts the usage of PM domains to devices and domains that
> could be powered on/off in irq disabled contexts as the mutexes used in
> GenPD allows for cpu sleep while waiting for locks.
>
> Genpd inherently provides support for devices, domain hierarchy and can
> be used to represent cpu clusters like in ARM's big.Little, where, each
> cpu cluster is in its domain, with supporting caches and other
> peripheral hardware. Multiple such domains could be part of another
> domain. Because mutexes are used to protect and synchronize domain
> operations and cpu idle operations are inherently atomic, the use of
> genpd is not possible for runtime suspend and resume of the pm domain.
> Replacing the locks to spinlocks would allow cpu domain to be be powered
> off to save power, when all the cpus are powered off.
>
> However, not all domains can operate in irq-safe contexts and usually
> would need to sleep during domain operations. So genpd has to support
> both the cases, where the domain is or is not irq-safe. The irq-safe
> attribute is therefore domain specific.
>
> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
> while defining the domain. This determines if the domain should use a
> spinlock instead of a mutex. Locking is abstracted through
> genpd_lock_domain() and genpd_unlock_domain() functions that use the
> flag to determine the locking to be used for this domain.
>
> The restriction this imposes on the domain hierarchy is that subdomains
> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
> may continue to have irq-safe devices, but not the other way around.
>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Kevin Hilman <khilman@linaro.org>
> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
> ---
>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>  include/linux/pm_domain.h   |  11 ++-
>  2 files changed, 164 insertions(+), 47 deletions(-)
>
> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
> index dfd7595..8b89d15 100644
> --- a/drivers/base/power/domain.c
> +++ b/drivers/base/power/domain.c
> @@ -50,6 +50,71 @@
>  static LIST_HEAD(gpd_list);
>  static DEFINE_MUTEX(gpd_list_lock);
>
> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,

Nitpick:

I would prefer a bit shorter names of all these new locking functions.
In general, I think you can remove "domain" from the names. In this
case it would mean genpd_lock_noirq().

> +                                       unsigned int subclass)
> +       __acquires(&genpd->slock)
> +{
> +       unsigned long flags;
> +
> +       if (unlikely(subclass > 0))
> +               spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
> +       else
> +               spin_lock_irqsave(&genpd->slock, flags);
> +
> +       genpd->flags = flags;

This is wrong, it should be genpd->lock_flags that you assign.

BTW, can't you assign the genpd->lock_flags immediately when call
spin_lock_irqsave*(), instead of keeping a local copy?

> +
> +       return 0;
> +}
> +
> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
> +       __releases(&genpd->slock)
> +{
> +       spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
> +       return 0;
> +}
> +
> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
> +                                       unsigned int subclass)
> +       __acquires(&genpd->mlock)
> +{
> +       if (unlikely(subclass > 0))
> +               mutex_lock_nested(&genpd->mlock, subclass);
> +       else
> +               mutex_lock(&genpd->mlock);
> +
> +       return 0;
> +}
> +
> +static inline int genpd_lock_domain_interruptible_irq(
> +                               struct generic_pm_domain *genpd)
> +       __acquires(&genpd->mlock)
> +{
> +       return mutex_lock_interruptible(&genpd->mlock);
> +}
> +
> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
> +       __releases(&genpd->mlock)
> +{
> +       mutex_unlock(&genpd->mlock);
> +       return 0;
> +}
> +
> +#define genpd_lock_domain(genpd)                               \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
> +                       : genpd_lock_domain_irq(genpd, 0))
> +
> +#define genpd_lock_domain_nested(genpd)                                \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
> +                       : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
> +
> +#define genpd_unlock_domain(genpd)                             \
> +       (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)     \
> +                       : genpd_unlock_domain_irq(genpd))
> +
> +#define genpd_lock_domain_interruptible(genpd)                 \
> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
> +                       : genpd_lock_domain_interruptible_irq(genpd))

In general I don't like macros (as Krzysztof).

In this case I also don't see the benfit, could you consider to
convert to in-line functions instead?

> +
>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>  {
>         struct generic_pm_domain *genpd = NULL, *gpd;
> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>  {
>         int ret;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         ret = __pm_genpd_poweron(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>  }
>
> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>                 spin_unlock_irq(&dev->power.lock);
>
>                 if (!IS_ERR(genpd)) {
> -                       mutex_lock(&genpd->lock);
> +                       genpd_lock_domain(genpd);
>                         genpd->max_off_time_changed = true;
> -                       mutex_unlock(&genpd->lock);
> +                       genpd_unlock_domain(genpd);
>                 }
>
>                 dev = dev->parent;
> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>                         return -EBUSY;
>
>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
> -                   || pdd->dev->power.irq_safe))
> +                       || (pdd->dev->power.irq_safe && !genpd->irq_safe)))

This deserves a comment in the code.

>                         not_suspended++;
>         }
>
> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>
>         genpd = container_of(work, struct generic_pm_domain, power_off_work);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         pm_genpd_poweroff(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>  }
>
>  /**
> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>         if (IS_ERR(genpd))
>                 return -EINVAL;
>
> -       /*
> -        * We can't allow to power off the PM domain if it holds an irq_safe
> -        * device. That's beacuse we use mutexes to protect data while power
> -        * off and on the PM domain, thus we can't execute in atomic context.
> -        */
> -       if (dev->power.irq_safe)
> +       /* We can't allow to power off a domain that is also not irq safe. */

I wouldn't mind to keep some more pieces of the earlier comment, since
it explains a bit more of the *why*. Could you try to rephrase this
comment in that regard?

> +       if (dev->power.irq_safe && !genpd->irq_safe)

This looks correct...

>                 return -EBUSY;
>
>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>                 return ret;
>         }
>
> -       mutex_lock(&genpd->lock);
> +       /*
> +        * If power.irq_safe is set, this routine will be run with interrupts
> +        * off, so suspend only if the power domain is irq_safe.
> +        */
> +       if (dev->power.irq_safe && !genpd->irq_safe)
> +               return 0;

... but this doesn't. You have already returned -EBUSY above for this case.

> +
> +       genpd_lock_domain(genpd);
> +
>         genpd->in_progress++;
>         pm_genpd_poweroff(genpd);
>         genpd->in_progress--;
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return 0;
>  }
> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>         if (IS_ERR(genpd))
>                 return -EINVAL;
>
> -       /* If power.irq_safe, the PM domain is never powered off. */
> -       if (dev->power.irq_safe)
> +       /*
> +        * If power.irq_safe and domain is not, then

Let's take the opportunity to rephrase this comment a bit.

Perhaps something along the lines: "As we don't power off a non IRQ
safe PM domain which holds an IRQ safe device, we don't need to
restore the power to it."

> +        * the PM domain is never powered off.
> +        */
> +       if (dev->power.irq_safe && !genpd->irq_safe)
>                 return genpd_start_dev_no_timing(genpd, dev);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>         ret = __pm_genpd_poweron(genpd);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (ret)
>                 return ret;
> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>         if (resume_needed(dev, genpd))
>                 pm_runtime_resume(dev);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count++ == 0) {
>                 genpd->suspended_count = 0;
>                 genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>         }
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (genpd->suspend_power_off) {
>                 pm_runtime_put_noidle(dev);
> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>
>         ret = pm_generic_prepare(dev);
>         if (ret) {
> -               mutex_lock(&genpd->lock);
> +               genpd_lock_domain(genpd);
>
>                 if (--genpd->prepared_count == 0)
>                         genpd->suspend_power_off = false;
>
> -               mutex_unlock(&genpd->lock);
> +               genpd_unlock_domain(genpd);
>                 pm_runtime_enable(dev);
>         }
>
> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>         if (IS_ERR(genpd))
>                 return;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         run_complete = !genpd->suspend_power_off;
>         if (--genpd->prepared_count == 0)
>                 genpd->suspend_power_off = false;
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (run_complete) {
>                 pm_generic_complete(dev);
> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>                 return -EINVAL;
>
> +       /* Devices in an IRQ safe PM Domain have to be irq safe too */

Let's be consistent and stick with IRQ in upper case letters for the
comments. I think there are other places in the patch which needs
update as well.

> +       if (genpd->irq_safe && !dev->power.irq_safe) {
> +               dev_warn(dev,

dev_err(), or perhaps even WARN() ?

> +                       "Devices in an irq-safe domain have to be irq safe.\n");
> +               return -EINVAL;
> +       }
> +
>         gpd_data = genpd_alloc_dev_data(dev, genpd, td);
>         if (IS_ERR(gpd_data))
>                 return PTR_ERR(gpd_data);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count > 0) {
>                 ret = -EAGAIN;
> @@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>         list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         if (ret)
>                 genpd_free_dev_data(dev, gpd_data);
> @@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>         gpd_data = to_gpd_data(pdd);
>         dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->prepared_count > 0) {
>                 ret = -EAGAIN;
> @@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>
>         list_del_init(&pdd->list_node);
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         genpd_free_dev_data(dev, gpd_data);
>
>         return 0;
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         dev_pm_qos_add_notifier(dev, &gpd_data->nb);
>
>         return ret;
> @@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>             || genpd == subdomain)
>                 return -EINVAL;
>
> +       /*
> +        * If the domain can be powered on/off in an irq safe
> +        * context, ensure that the subdomain can also be
> +        * powered on/off in that context.
> +        */

Have you considered if/how to address this limitation?

Just to be clear, for now I think this limitation is perfectly okay,
but going forward we might see a need to mix non IRQ safe domains with
IRQ safe domains, via sub-domains.

> +       if (genpd->irq_safe && !subdomain->irq_safe) {
> +               pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
> +                               subdomain->name, genpd->name);

Similar comment as earlier. Perhaps pr_err() or WARN() instead.

> +               return -EINVAL;
> +       }
> +
>         link = kzalloc(sizeof(*link), GFP_KERNEL);
>         if (!link)
>                 return -ENOMEM;
>
> -       mutex_lock(&genpd->lock);
> -       mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
> +       genpd_lock_domain(genpd);
> +       genpd_lock_domain_nested(subdomain);
>
>         if (genpd->status == GPD_STATE_POWER_OFF
>             &&  subdomain->status != GPD_STATE_POWER_OFF) {
> @@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>                 genpd_sd_counter_inc(genpd);
>
>   out:
> -       mutex_unlock(&subdomain->lock);
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(subdomain);
> +       genpd_unlock_domain(genpd);
>
>         return ret;
>  }
> @@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
>                 return -EINVAL;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         list_for_each_entry(link, &genpd->master_links, master_node) {
>                 if (link->slave != subdomain)
>                         continue;
>
> -               mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
> +               genpd_lock_domain_nested(subdomain);
>
>                 list_del(&link->master_node);
>                 list_del(&link->slave_node);
> @@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>                 if (subdomain->status != GPD_STATE_POWER_OFF)
>                         genpd_sd_counter_dec(genpd);
>
> -               mutex_unlock(&subdomain->lock);
> +               genpd_unlock_domain(subdomain);
>
>                 ret = 0;
>                 break;
>         }
>
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return ret;
>  }
> @@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>         if (IS_ERR_OR_NULL(genpd) || state < 0)
>                 return -EINVAL;
>
> +       if (genpd->irq_safe) {
> +               pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
> +                               genpd->name);

Here you have pr_err(), which is okay but not consistent. Also, I
wonder if we should use a WARN() instead?

> +               return -EINVAL;
> +       }
> +
>         cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>         if (!cpuidle_data)
>                 return -ENOMEM;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         if (genpd->cpuidle_data) {
>                 ret = -EEXIST;
> @@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>         genpd_recalc_cpu_exit_latency(genpd);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>
>   err:
> @@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>         if (IS_ERR_OR_NULL(genpd))
>                 return -EINVAL;
>
> -       mutex_lock(&genpd->lock);
> +       genpd_lock_domain(genpd);
>
>         cpuidle_data = genpd->cpuidle_data;
>         if (!cpuidle_data) {
> @@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>         kfree(cpuidle_data);
>
>   out:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>         return ret;
>  }
>
> @@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
>         return cb ? cb(dev) : 0;
>  }
>
> +static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
> +{
> +       if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
> +               spin_lock_init(&genpd->slock);
> +               genpd->irq_safe = true;

Copying the result from "genpd->flags & GENPD_FLAG_IRQ_SAFE" into the
irq_safe bool, seems like a "micro" optimization. How about only using
the genpd->flags instead?

> +       } else {
> +               mutex_init(&genpd->mlock);
> +               genpd->irq_safe = false;
> +       }
> +}
> +
>  /**
>   * pm_genpd_init - Initialize a generic I/O PM domain object.
>   * @genpd: PM domain object to initialize.
> @@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
>         INIT_LIST_HEAD(&genpd->master_links);
>         INIT_LIST_HEAD(&genpd->slave_links);
>         INIT_LIST_HEAD(&genpd->dev_list);
> -       mutex_init(&genpd->lock);
> +       genpd_lock_domain_init(genpd);
>         genpd->gov = gov;
>         INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
>         genpd->in_progress = 0;
> @@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>         struct gpd_link *link;
>         int ret;
>
> -       ret = mutex_lock_interruptible(&genpd->lock);
> +       ret = genpd_lock_domain_interruptible(genpd);
>         if (ret)
>                 return -ERESTARTSYS;
>
> @@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
>         }
>
>         list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
> -               kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
> +               kobj_path = kobject_get_path(&pm_data->dev->kobj,
> +                               genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
>                 if (kobj_path == NULL)
>                         continue;
>
> @@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>
>         seq_puts(s, "\n");
>  exit:
> -       mutex_unlock(&genpd->lock);
> +       genpd_unlock_domain(genpd);
>
>         return 0;
>  }
> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
> index b2725e6..dc7cb53 100644
> --- a/include/linux/pm_domain.h
> +++ b/include/linux/pm_domain.h
> @@ -16,9 +16,11 @@
>  #include <linux/of.h>
>  #include <linux/notifier.h>
>  #include <linux/cpuidle.h>
> +#include <linux/spinlock.h>
>
>  /* Defines used for the flags field in the struct generic_pm_domain */
>  #define GENPD_FLAG_PM_CLK      (1U << 0) /* PM domain uses PM clk */
> +#define GENPD_FLAG_IRQ_SAFE    (1U << 1) /* PM domain operates in atomic */
>
>  enum gpd_status {
>         GPD_STATE_ACTIVE = 0,   /* PM domain is active */
> @@ -49,7 +51,6 @@ struct generic_pm_domain {
>         struct list_head master_links;  /* Links with PM domain as a master */
>         struct list_head slave_links;   /* Links with PM domain as a slave */
>         struct list_head dev_list;      /* List of devices */
> -       struct mutex lock;
>         struct dev_power_governor *gov;
>         struct work_struct power_off_work;
>         const char *name;
> @@ -74,6 +75,14 @@ struct generic_pm_domain {
>         void (*detach_dev)(struct generic_pm_domain *domain,
>                            struct device *dev);
>         unsigned int flags;             /* Bit field of configs for genpd */
> +       bool irq_safe;
> +       union {
> +               struct mutex mlock;
> +               struct {
> +                       spinlock_t slock;
> +                       unsigned long lock_flags;
> +               };
> +       };
>  };
>
>  static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
> --
> 2.1.4
>

Kind regards
Uffe

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Thu, Jun 11 2015 at 18:13 -0600, Krzysztof Kozlowski wrote:
>On 11.06.2015 01:13, Lina Iyer wrote:
>> On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:

...

>>>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct
>>>> generic_pm_domain *genpd, struct device *dev,
>>>>      if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>>>          return -EINVAL;
>>>>
>>>> +    /* Devices in an IRQ safe PM Domain have to be irq safe too */
>>>
>>> Why? Can you add this information here? Previously there was a reason in
>>> case of irq_safe devices which you removed leaving only policy.
>>>
>> Sorry, your question is not clear to me.
>> I believe this is a new requirement that enforces the contained devices
>> of an irq-safe domain to be irq-safe as well.
>
>What I wanted to say is that it would be nice if comment explained why
>domain have to be IRQ safe too. Without this "WHY" answer the comment is
>quite redundant - the "if" statement is obvious. But the "WHY" is not
>such obvious.
>
>Previous comments in few places mentioned the answer:
>/*
> * We can't allow to power off the PM domain if it holds an irq_safe
> * device. That's beacuse we use mutexes to protect data while power
> * off and on the PM domain, thus we can't execute in atomic context.
> */
>
Oh, yes. Will fix it.

Thanks,
Lina

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Thu, Jun 11 2015 at 18:13 -0600, Krzysztof Kozlowski wrote:
>On 11.06.2015 01:13, Lina Iyer wrote:
>> On Sun, Jun 07 2015 at 03:21 -0600, Krzysztof Kozlowski wrote:
>>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:

...

>>>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct
>>>> generic_pm_domain *genpd, struct device *dev,
>>>>      if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>>>          return -EINVAL;
>>>>
>>>> +    /* Devices in an IRQ safe PM Domain have to be irq safe too */
>>>
>>> Why? Can you add this information here? Previously there was a reason in
>>> case of irq_safe devices which you removed leaving only policy.
>>>
>> Sorry, your question is not clear to me.
>> I believe this is a new requirement that enforces the contained devices
>> of an irq-safe domain to be irq-safe as well.
>
>What I wanted to say is that it would be nice if comment explained why
>domain have to be IRQ safe too. Without this "WHY" answer the comment is
>quite redundant - the "if" statement is obvious. But the "WHY" is not
>such obvious.
>
>Previous comments in few places mentioned the answer:
>/*
> * We can't allow to power off the PM domain if it holds an irq_safe
> * device. That's beacuse we use mutexes to protect data while power
> * off and on the PM domain, thus we can't execute in atomic context.
> */
>
Oh, yes. Will fix it.

Thanks,
Lina

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Thu, Jun 11 2015 at 18:27 -0600, Krzysztof Kozlowski wrote:
>On 11.06.2015 01:57, Lina Iyer wrote:
>> On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>>> in the domain are in their power off state,
>>>
>>> What do you exactly mean here by "CPU in power off state"? How does it
>>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>>
>> Both cpuidle and hotplug could end with with core being powered down at
>> the platform driver or at PSCI (on V8). It does not matter which of
>> these two frameworks resulted in the cpu being powered off. But, if all
>> cpus in the domain are powered off, then the domain could be powered off
>> as well. This is the premise of this change. It is probably easier to
>> power off the domain when the cores in that domain/cluster have been
>> hotplugged off. It saves power to turn off the domain at that time, but
>> more power savings can be achieved if the domain could also be powered
>> off during cpuidle. Hotplug is not a common occurance, while cpuidle is.
>
>OK, it answers my questions, thanks.
>
>>
>>>> the cpu domain can also be
>>>> powered off. Genpd provides the framework for defining cpus as devices
>>>> that are part of a cpu domain.
>>>
>>> The problem which is solved looks to me like the same problem which
>>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>>> off) can be entered when whole cluster is idle or other CPUs in cluster
>>> are powered off completely.
>>>
>>> It seems a little like duplicating the effort around coupled cpuidle.
>>>
>> I see where are you are going with this, but genpd solution is not
>> exactly a duplicate of the solution.
>>
>> Couple state is used to put the cpus in a deeper sleep state, which
>> could also result in powering off the domain. Coupled cpuidle is a
>> cpuidle mechanism for choosing a deeper sleep mode on certain hardware
>> that can only enter such a mode when all cpus cooperate.
>>
>> This patch attempts to describe the backend of a cpu domain. CPUs are
>> responsible for individual cpuidle states, cpus do enter their
>> recommended deepest idle state at the time of no activity. A cpu-domain
>> could be comprised of cpus, and other devices like GIC, busses etc, that
>> all need to idle before the domain can be powered off. This patch does
>> not dictate which idle state any those devices should enter, or
>> coordinate the idle states between devices. But, if cpus, choose to
>> power down, then this patch recognizes that and reduces the reference
>> usage count on the domain. Only when all devices in the domain remove
>> their usage count, will the domain be powered off.
>
>It would be nice to see the usage of this patch in cpuidle driver or
>platform code but I think I get the idea.
>
Ok, my next spin, will include the platform driver changes for the QCOM
SoC that I tested this on.


>Actually I like the approach.
>I am thinking how to utilize it to replace coupled cpuidle for our case.
>In our case we use coupled cpuidle because the SoC can be put in low
>power mode only if non-boot CPUs are powered down.
>
>However in our case:
>1. Some other devices (buses, clocks) also should be idle. This would
>perfectly match with this patch and with runtime PM.
>
>2. Some non-boot idle CPU could power itself down but it cannot wake up.
>Only the alive CPU can wake others. This probably means that we cannot
>provide a cpuidle driver which will power off unused cores and then, if
>boot CPU is idle, disable the CPU power domain by entering to low power
>mode.
>
>Anyway, as I said, I like the approach.
>
It was Kevin's idea, that I implemented.

Thanks,
Lina
>
>> There are two things this patch provides -
>>
>> i. A generic way to initialize a genpd specifically for cpus. (The
>> platform specifies the relation between a cpu and its domain in the DT
>> and provides the memory for the genpd structure)
>>
>> ii. On behalf of a platform, we track when the cpus power up and down
>> and use runtime_get and runtime_put on the genpd.
>>
>> Unlike coupled cpuidle, individual cpu idle state is not manipulated.
>> Coupled cpuidle does not care if the domain is powered off, it is used
>> to allow a certain C-state for the cpu, based on the idleness of other
>> cpus in that cluster. The focus of the series is powering down the
>> domain when the devices (cpus included) are powered off. You could see
>> this patch as a cpu-pm and runtime-pm interface layer.
>>
>> Hope that helps.
>>
>> Thanks,
>> Lina
>>
>
>Best regards,
>Krzysztof
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-pm" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Thu, Jun 11 2015 at 18:27 -0600, Krzysztof Kozlowski wrote:
>On 11.06.2015 01:57, Lina Iyer wrote:
>> On Sun, Jun 07 2015 at 03:43 -0600, Krzysztof Kozlowski wrote:
>>> W dniu 05.06.2015 o 07:29, Lina Iyer pisze:
>>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>>> in the domain are in their power off state,
>>>
>>> What do you exactly mean here by "CPU in power off state"? How does it
>>> map to kernel understanding of CPU device (hotplug? cpuidle?)?
>>>
>> Both cpuidle and hotplug could end with with core being powered down at
>> the platform driver or at PSCI (on V8). It does not matter which of
>> these two frameworks resulted in the cpu being powered off. But, if all
>> cpus in the domain are powered off, then the domain could be powered off
>> as well. This is the premise of this change. It is probably easier to
>> power off the domain when the cores in that domain/cluster have been
>> hotplugged off. It saves power to turn off the domain at that time, but
>> more power savings can be achieved if the domain could also be powered
>> off during cpuidle. Hotplug is not a common occurance, while cpuidle is.
>
>OK, it answers my questions, thanks.
>
>>
>>>> the cpu domain can also be
>>>> powered off. Genpd provides the framework for defining cpus as devices
>>>> that are part of a cpu domain.
>>>
>>> The problem which is solved looks to me like the same problem which
>>> coupled cpuidle tried to solve: a certain deep sleep mode (e.g. power
>>> off) can be entered when whole cluster is idle or other CPUs in cluster
>>> are powered off completely.
>>>
>>> It seems a little like duplicating the effort around coupled cpuidle.
>>>
>> I see where are you are going with this, but genpd solution is not
>> exactly a duplicate of the solution.
>>
>> Couple state is used to put the cpus in a deeper sleep state, which
>> could also result in powering off the domain. Coupled cpuidle is a
>> cpuidle mechanism for choosing a deeper sleep mode on certain hardware
>> that can only enter such a mode when all cpus cooperate.
>>
>> This patch attempts to describe the backend of a cpu domain. CPUs are
>> responsible for individual cpuidle states, cpus do enter their
>> recommended deepest idle state at the time of no activity. A cpu-domain
>> could be comprised of cpus, and other devices like GIC, busses etc, that
>> all need to idle before the domain can be powered off. This patch does
>> not dictate which idle state any those devices should enter, or
>> coordinate the idle states between devices. But, if cpus, choose to
>> power down, then this patch recognizes that and reduces the reference
>> usage count on the domain. Only when all devices in the domain remove
>> their usage count, will the domain be powered off.
>
>It would be nice to see the usage of this patch in cpuidle driver or
>platform code but I think I get the idea.
>
Ok, my next spin, will include the platform driver changes for the QCOM
SoC that I tested this on.


>Actually I like the approach.
>I am thinking how to utilize it to replace coupled cpuidle for our case.
>In our case we use coupled cpuidle because the SoC can be put in low
>power mode only if non-boot CPUs are powered down.
>
>However in our case:
>1. Some other devices (buses, clocks) also should be idle. This would
>perfectly match with this patch and with runtime PM.
>
>2. Some non-boot idle CPU could power itself down but it cannot wake up.
>Only the alive CPU can wake others. This probably means that we cannot
>provide a cpuidle driver which will power off unused cores and then, if
>boot CPU is idle, disable the CPU power domain by entering to low power
>mode.
>
>Anyway, as I said, I like the approach.
>
It was Kevin's idea, that I implemented.

Thanks,
Lina
>
>> There are two things this patch provides -
>>
>> i. A generic way to initialize a genpd specifically for cpus. (The
>> platform specifies the relation between a cpu and its domain in the DT
>> and provides the memory for the genpd structure)
>>
>> ii. On behalf of a platform, we track when the cpus power up and down
>> and use runtime_get and runtime_put on the genpd.
>>
>> Unlike coupled cpuidle, individual cpu idle state is not manipulated.
>> Coupled cpuidle does not care if the domain is powered off, it is used
>> to allow a certain C-state for the cpu, based on the idleness of other
>> cpus in that cluster. The focus of the series is powering down the
>> domain when the devices (cpus included) are powered off. You could see
>> this patch as a cpu-pm and runtime-pm interface layer.
>>
>> Hope that helps.
>>
>> Thanks,
>> Lina
>>
>
>Best regards,
>Krzysztof
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-pm" in
>the body of a message to majordomo at vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state, the cpu domain can also be
>> powered off.
>
>How does this relate to a cluster, and why aren't you using that terminolgy?
>
>> Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>>
>> Introduce support for defining and adding a generic power domain for the
>> cpus based on the DT specification of power domain providers and
>> consumers.  SoC's that have the cpu domain defined in their DT, can
>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>> this domain with the genpd.
>>
>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>> last cpu going down would call the genpd->power_off(). Correspondingly,
>> the first cpu up would call the genpd->power_on() callback before
>> resuming from idle.
>
>Other patches also mention this genpd being useful to gate power to
>non-CPU peripherals on the same power rail.  How are those devices to be
>added?
>
I am not investigating DT nodes to figure out which node is a consumer
for this domain provider. That could be a good way to do it, but
practically speaking, there may be platform dependencies and specifics
that may need to be met, before the device can be added to the CPU
genpd.

So that is not generalized here. I didn't see a better way to do that,
generically. Do you have ideas on that?

The platform is the owner of the genpd and therefore can add those
non-cpu devices to the genpd as and when appropritate.

In this regard, I also have a question, who initializes the genpd. My
assumption is that CPUs will get probed before most other devices and
therefore the domain provider could be initialized by this file. But, I
could be wrong here.




>Without seeing the DTs and the init code that might call
>pm_cpu_domain_init(), it's hard for me to see how this is intended to be
>used.  Could you also include a patch that shows how this is initialized
>and the DT additions?  Ideally, it should also show how a non-CPU device
>would be included.
>
I am sorry, you are right. This is better explained with the patches for
platform driver. I will add them in the next spin. I thought the
examples provided in the cover letter is pretty close to the genpd
related changes that I made in my platform coder. But I agree, it doesnt
give the complete picture.

Thanks,
Lina

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>> in the domain are in their power off state, the cpu domain can also be
>> powered off.
>
>How does this relate to a cluster, and why aren't you using that terminolgy?
>
>> Genpd provides the framework for defining cpus as devices
>> that are part of a cpu domain.
>>
>> Introduce support for defining and adding a generic power domain for the
>> cpus based on the DT specification of power domain providers and
>> consumers.  SoC's that have the cpu domain defined in their DT, can
>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>> this domain with the genpd.
>>
>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>> last cpu going down would call the genpd->power_off(). Correspondingly,
>> the first cpu up would call the genpd->power_on() callback before
>> resuming from idle.
>
>Other patches also mention this genpd being useful to gate power to
>non-CPU peripherals on the same power rail.  How are those devices to be
>added?
>
I am not investigating DT nodes to figure out which node is a consumer
for this domain provider. That could be a good way to do it, but
practically speaking, there may be platform dependencies and specifics
that may need to be met, before the device can be added to the CPU
genpd.

So that is not generalized here. I didn't see a better way to do that,
generically. Do you have ideas on that?

The platform is the owner of the genpd and therefore can add those
non-cpu devices to the genpd as and when appropritate.

In this regard, I also have a question, who initializes the genpd. My
assumption is that CPUs will get probed before most other devices and
therefore the domain provider could be initialized by this file. But, I
could be wrong here.




>Without seeing the DTs and the init code that might call
>pm_cpu_domain_init(), it's hard for me to see how this is intended to be
>used.  Could you also include a patch that shows how this is initialized
>and the DT additions?  Ideally, it should also show how a non-CPU device
>would be included.
>
I am sorry, you are right. This is better explained with the patches for
platform driver. I will add them in the next spin. I thought the
examples provided in the cover letter is pretty close to the genpd
related changes that I made in my platform coder. But I agree, it doesnt
give the complete picture.

Thanks,
Lina

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Thu, Jun 11 2015 at 03:41 -0600, Ulf Hansson wrote:
>On 5 June 2015 at 00:29, Lina Iyer <lina.iyer@linaro.org> wrote:
>> Power Domains currently support turning on/off only in process context.
>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>>
>> Genpd inherently provides support for devices, domain hierarchy and can
>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware. Multiple such domains could be part of another
>> domain. Because mutexes are used to protect and synchronize domain
>> operations and cpu idle operations are inherently atomic, the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>> off to save power, when all the cpus are powered off.
>>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>>  include/linux/pm_domain.h   |  11 ++-
>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index dfd7595..8b89d15 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -50,6 +50,71 @@
>>  static LIST_HEAD(gpd_list);
>>  static DEFINE_MUTEX(gpd_list_lock);
>>
>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
>
>Nitpick:
>
>I would prefer a bit shorter names of all these new locking functions.
>In general, I think you can remove "domain" from the names. In this
>case it would mean genpd_lock_noirq().
>
Done.

>> +                                       unsigned int subclass)
>> +       __acquires(&genpd->slock)
>> +{
>> +       unsigned long flags;
>> +
>> +       if (unlikely(subclass > 0))
>> +               spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>> +       else
>> +               spin_lock_irqsave(&genpd->slock, flags);
>> +
>> +       genpd->flags = flags;
>
>This is wrong, it should be genpd->lock_flags that you assign.
>
Argh. My bad. Fixed.

>BTW, can't you assign the genpd->lock_flags immediately when call
>spin_lock_irqsave*(), instead of keeping a local copy?
>
I vaguely remember reading that in some architecture you needed to use a
stack variable to save the IRQ state . I cant seem to find it now.
I can change it.

>> +
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
>> +       __releases(&genpd->slock)
>> +{
>> +       spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
>> +                                       unsigned int subclass)
>> +       __acquires(&genpd->mlock)
>> +{
>> +       if (unlikely(subclass > 0))
>> +               mutex_lock_nested(&genpd->mlock, subclass);
>> +       else
>> +               mutex_lock(&genpd->mlock);
>> +
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_interruptible_irq(
>> +                               struct generic_pm_domain *genpd)
>> +       __acquires(&genpd->mlock)
>> +{
>> +       return mutex_lock_interruptible(&genpd->mlock);
>> +}
>> +
>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
>> +       __releases(&genpd->mlock)
>> +{
>> +       mutex_unlock(&genpd->mlock);
>> +       return 0;
>> +}
>> +
>> +#define genpd_lock_domain(genpd)                               \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>> +                       : genpd_lock_domain_irq(genpd, 0))
>> +
>> +#define genpd_lock_domain_nested(genpd)                                \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
>> +                       : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>> +
>> +#define genpd_unlock_domain(genpd)                             \
>> +       (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)     \
>> +                       : genpd_unlock_domain_irq(genpd))
>> +
>> +#define genpd_lock_domain_interruptible(genpd)                 \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>> +                       : genpd_lock_domain_interruptible_irq(genpd))
>
>In general I don't like macros (as Krzysztof).
>
>In this case I also don't see the benfit, could you consider to
>convert to in-line functions instead?
>
Done. Since I have no strong preference either way, I will change this
to inline functions.

>> +
>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>>  {
>>         struct generic_pm_domain *genpd = NULL, *gpd;
>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>>  {
>>         int ret;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         ret = __pm_genpd_poweron(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>  }
>>
>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>>                 spin_unlock_irq(&dev->power.lock);
>>
>>                 if (!IS_ERR(genpd)) {
>> -                       mutex_lock(&genpd->lock);
>> +                       genpd_lock_domain(genpd);
>>                         genpd->max_off_time_changed = true;
>> -                       mutex_unlock(&genpd->lock);
>> +                       genpd_unlock_domain(genpd);
>>                 }
>>
>>                 dev = dev->parent;
>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>>                         return -EBUSY;
>>
>>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>> -                   || pdd->dev->power.irq_safe))
>> +                       || (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>
>This deserves a comment in the code.
>
This check should simplify to
		if (!pm_runtime_suspended(pdd->dev))

The other connditions seem unnecessary just to determine if the devices are
suspended.

Do you see any problem with that?

>>                         not_suspended++;
>>         }
>>
>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>>
>>         genpd = container_of(work, struct generic_pm_domain, power_off_work);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         pm_genpd_poweroff(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>  }
>>
>>  /**
>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return -EINVAL;
>>
>> -       /*
>> -        * We can't allow to power off the PM domain if it holds an irq_safe
>> -        * device. That's beacuse we use mutexes to protect data while power
>> -        * off and on the PM domain, thus we can't execute in atomic context.
>> -        */
>> -       if (dev->power.irq_safe)
>> +       /* We can't allow to power off a domain that is also not irq safe. */
>
>I wouldn't mind to keep some more pieces of the earlier comment, since
>it explains a bit more of the *why*. Could you try to rephrase this
>comment in that regard?
>
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>
>This looks correct...
>
Apparently not the best place to check this. We should still allow
runtime suspend of the device, irrespective of whether the domain is IRQ
safe or not and this check here is incorrect.

>>                 return -EBUSY;
>>
>>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>                 return ret;
>>         }
>>
>> -       mutex_lock(&genpd->lock);
>> +       /*
>> +        * If power.irq_safe is set, this routine will be run with interrupts
>> +        * off, so suspend only if the power domain is irq_safe.
>> +        */
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>> +               return 0;
>
>... but this doesn't. You have already returned -EBUSY above for this case.
>
This is probably more appropriate, as we want to skip locking the
domain, because it may be in atomic context and the domain may use
mutexes to lock.

I feel like, we miss out an opportunity here to power off the domain, if
device is IRQ safe while the domain is not, but the function is called
from a process context. I am not sure how to check that correctly, yet.

>> +
>> +       genpd_lock_domain(genpd);
>> +
>>         genpd->in_progress++;
>>         pm_genpd_poweroff(genpd);
>>         genpd->in_progress--;
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return 0;
>>  }
>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return -EINVAL;
>>
>> -       /* If power.irq_safe, the PM domain is never powered off. */
>> -       if (dev->power.irq_safe)
>> +       /*
>> +        * If power.irq_safe and domain is not, then
>
>Let's take the opportunity to rephrase this comment a bit.
>
>Perhaps something along the lines: "As we don't power off a non IRQ
>safe PM domain which holds an IRQ safe device, we don't need to
>restore the power to it."
>
Done. Makes sense.

>> +        * the PM domain is never powered off.
>> +        */
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>                 return genpd_start_dev_no_timing(genpd, dev);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         ret = __pm_genpd_poweron(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (ret)
>>                 return ret;
>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>         if (resume_needed(dev, genpd))
>>                 pm_runtime_resume(dev);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count++ == 0) {
>>                 genpd->suspended_count = 0;
>>                 genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>>         }
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (genpd->suspend_power_off) {
>>                 pm_runtime_put_noidle(dev);
>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>
>>         ret = pm_generic_prepare(dev);
>>         if (ret) {
>> -               mutex_lock(&genpd->lock);
>> +               genpd_lock_domain(genpd);
>>
>>                 if (--genpd->prepared_count == 0)
>>                         genpd->suspend_power_off = false;
>>
>> -               mutex_unlock(&genpd->lock);
>> +               genpd_unlock_domain(genpd);
>>                 pm_runtime_enable(dev);
>>         }
>>
>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         run_complete = !genpd->suspend_power_off;
>>         if (--genpd->prepared_count == 0)
>>                 genpd->suspend_power_off = false;
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (run_complete) {
>>                 pm_generic_complete(dev);
>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>                 return -EINVAL;
>>
>> +       /* Devices in an IRQ safe PM Domain have to be irq safe too */
>
>Let's be consistent and stick with IRQ in upper case letters for the
>comments. I think there are other places in the patch which needs
>update as well.
>
Sorry. I will stick to IRQ and CPU (thanks Kevin) henceforth. 

>> +       if (genpd->irq_safe && !dev->power.irq_safe) {
>> +               dev_warn(dev,
>
>dev_err(), or perhaps even WARN() ?
>
OK, they are now WARN.

>> +                       "Devices in an irq-safe domain have to be irq safe.\n");
>> +               return -EINVAL;
>> +       }
>> +
>>         gpd_data = genpd_alloc_dev_data(dev, genpd, td);
>>         if (IS_ERR(gpd_data))
>>                 return PTR_ERR(gpd_data);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count > 0) {
>>                 ret = -EAGAIN;
>> @@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>         list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (ret)
>>                 genpd_free_dev_data(dev, gpd_data);
>> @@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>         gpd_data = to_gpd_data(pdd);
>>         dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count > 0) {
>>                 ret = -EAGAIN;
>> @@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>
>>         list_del_init(&pdd->list_node);
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         genpd_free_dev_data(dev, gpd_data);
>>
>>         return 0;
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         dev_pm_qos_add_notifier(dev, &gpd_data->nb);
>>
>>         return ret;
>> @@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>             || genpd == subdomain)
>>                 return -EINVAL;
>>
>> +       /*
>> +        * If the domain can be powered on/off in an irq safe
>> +        * context, ensure that the subdomain can also be
>> +        * powered on/off in that context.
>> +        */
>
>Have you considered if/how to address this limitation?
>
>Just to be clear, for now I think this limitation is perfectly okay,
>but going forward we might see a need to mix non IRQ safe domains with
>IRQ safe domains, via sub-domains.
>
Yeah, if we could be cognizant of the context we are running in, we
can be better about operating on domain and devices. Then this
limitation would go away.

>> +       if (genpd->irq_safe && !subdomain->irq_safe) {
>> +               pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
>> +                               subdomain->name, genpd->name);
>
>Similar comment as earlier. Perhaps pr_err() or WARN() instead.
>
Done.

>> +               return -EINVAL;
>> +       }
>> +
>>         link = kzalloc(sizeof(*link), GFP_KERNEL);
>>         if (!link)
>>                 return -ENOMEM;
>>
>> -       mutex_lock(&genpd->lock);
>> -       mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>> +       genpd_lock_domain(genpd);
>> +       genpd_lock_domain_nested(subdomain);
>>
>>         if (genpd->status == GPD_STATE_POWER_OFF
>>             &&  subdomain->status != GPD_STATE_POWER_OFF) {
>> @@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>                 genpd_sd_counter_inc(genpd);
>>
>>   out:
>> -       mutex_unlock(&subdomain->lock);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(subdomain);
>> +       genpd_unlock_domain(genpd);
>>
>>         return ret;
>>  }
>> @@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
>>                 return -EINVAL;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         list_for_each_entry(link, &genpd->master_links, master_node) {
>>                 if (link->slave != subdomain)
>>                         continue;
>>
>> -               mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>> +               genpd_lock_domain_nested(subdomain);
>>
>>                 list_del(&link->master_node);
>>                 list_del(&link->slave_node);
>> @@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>>                 if (subdomain->status != GPD_STATE_POWER_OFF)
>>                         genpd_sd_counter_dec(genpd);
>>
>> -               mutex_unlock(&subdomain->lock);
>> +               genpd_unlock_domain(subdomain);
>>
>>                 ret = 0;
>>                 break;
>>         }
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return ret;
>>  }
>> @@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>         if (IS_ERR_OR_NULL(genpd) || state < 0)
>>                 return -EINVAL;
>>
>> +       if (genpd->irq_safe) {
>> +               pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
>> +                               genpd->name);
>
>Here you have pr_err(), which is okay but not consistent. Also, I
>wonder if we should use a WARN() instead?
>
Done.

>> +               return -EINVAL;
>> +       }
>> +
>>         cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>>         if (!cpuidle_data)
>>                 return -ENOMEM;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->cpuidle_data) {
>>                 ret = -EEXIST;
>> @@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>         genpd_recalc_cpu_exit_latency(genpd);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>
>>   err:
>> @@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>>         if (IS_ERR_OR_NULL(genpd))
>>                 return -EINVAL;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         cpuidle_data = genpd->cpuidle_data;
>>         if (!cpuidle_data) {
>> @@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>>         kfree(cpuidle_data);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>  }
>>
>> @@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
>>         return cb ? cb(dev) : 0;
>>  }
>>
>> +static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
>> +{
>> +       if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
>> +               spin_lock_init(&genpd->slock);
>> +               genpd->irq_safe = true;
>
>Copying the result from "genpd->flags & GENPD_FLAG_IRQ_SAFE" into the
>irq_safe bool, seems like a "micro" optimization. How about only using
>the genpd->flags instead?
>
Actually, I would prefer using the flag, there are a few places in the
code where I need to know if the domain is IRQ safe or not. Checking for
flags is cumbersome and not every aestheic.

>> +       } else {
>> +               mutex_init(&genpd->mlock);
>> +               genpd->irq_safe = false;
>> +       }
>> +}
>> +
>>  /**
>>   * pm_genpd_init - Initialize a generic I/O PM domain object.
>>   * @genpd: PM domain object to initialize.
>> @@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
>>         INIT_LIST_HEAD(&genpd->master_links);
>>         INIT_LIST_HEAD(&genpd->slave_links);
>>         INIT_LIST_HEAD(&genpd->dev_list);
>> -       mutex_init(&genpd->lock);
>> +       genpd_lock_domain_init(genpd);
>>         genpd->gov = gov;
>>         INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
>>         genpd->in_progress = 0;
>> @@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>         struct gpd_link *link;
>>         int ret;
>>
>> -       ret = mutex_lock_interruptible(&genpd->lock);
>> +       ret = genpd_lock_domain_interruptible(genpd);
>>         if (ret)
>>                 return -ERESTARTSYS;
>>
>> @@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>         }
>>
>>         list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
>> -               kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
>> +               kobj_path = kobject_get_path(&pm_data->dev->kobj,
>> +                               genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
>>                 if (kobj_path == NULL)
>>                         continue;
>>
>> @@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>
>>         seq_puts(s, "\n");
>>  exit:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return 0;
>>  }
>> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
>> index b2725e6..dc7cb53 100644
>> --- a/include/linux/pm_domain.h
>> +++ b/include/linux/pm_domain.h
>> @@ -16,9 +16,11 @@
>>  #include <linux/of.h>
>>  #include <linux/notifier.h>
>>  #include <linux/cpuidle.h>
>> +#include <linux/spinlock.h>
>>
>>  /* Defines used for the flags field in the struct generic_pm_domain */
>>  #define GENPD_FLAG_PM_CLK      (1U << 0) /* PM domain uses PM clk */
>> +#define GENPD_FLAG_IRQ_SAFE    (1U << 1) /* PM domain operates in atomic */
>>
>>  enum gpd_status {
>>         GPD_STATE_ACTIVE = 0,   /* PM domain is active */
>> @@ -49,7 +51,6 @@ struct generic_pm_domain {
>>         struct list_head master_links;  /* Links with PM domain as a master */
>>         struct list_head slave_links;   /* Links with PM domain as a slave */
>>         struct list_head dev_list;      /* List of devices */
>> -       struct mutex lock;
>>         struct dev_power_governor *gov;
>>         struct work_struct power_off_work;
>>         const char *name;
>> @@ -74,6 +75,14 @@ struct generic_pm_domain {
>>         void (*detach_dev)(struct generic_pm_domain *domain,
>>                            struct device *dev);
>>         unsigned int flags;             /* Bit field of configs for genpd */
>> +       bool irq_safe;
>> +       union {
>> +               struct mutex mlock;
>> +               struct {
>> +                       spinlock_t slock;
>> +                       unsigned long lock_flags;
>> +               };
>> +       };
>>  };
>>
>>  static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
>> --
>> 2.1.4
>>

Thank you so much Ulf for the review.

--Lina

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Lina Iyer]

On Thu, Jun 11 2015 at 03:41 -0600, Ulf Hansson wrote:
>On 5 June 2015 at 00:29, Lina Iyer <lina.iyer@linaro.org> wrote:
>> Power Domains currently support turning on/off only in process context.
>> This restricts the usage of PM domains to devices and domains that
>> could be powered on/off in irq disabled contexts as the mutexes used in
>> GenPD allows for cpu sleep while waiting for locks.
>>
>> Genpd inherently provides support for devices, domain hierarchy and can
>> be used to represent cpu clusters like in ARM's big.Little, where, each
>> cpu cluster is in its domain, with supporting caches and other
>> peripheral hardware. Multiple such domains could be part of another
>> domain. Because mutexes are used to protect and synchronize domain
>> operations and cpu idle operations are inherently atomic, the use of
>> genpd is not possible for runtime suspend and resume of the pm domain.
>> Replacing the locks to spinlocks would allow cpu domain to be be powered
>> off to save power, when all the cpus are powered off.
>>
>> However, not all domains can operate in irq-safe contexts and usually
>> would need to sleep during domain operations. So genpd has to support
>> both the cases, where the domain is or is not irq-safe. The irq-safe
>> attribute is therefore domain specific.
>>
>> To achieve domain specific locking, set the GENPD_FLAG_IRQ_SAFE flag
>> while defining the domain. This determines if the domain should use a
>> spinlock instead of a mutex. Locking is abstracted through
>> genpd_lock_domain() and genpd_unlock_domain() functions that use the
>> flag to determine the locking to be used for this domain.
>>
>> The restriction this imposes on the domain hierarchy is that subdomains
>> and all devices in the hierarchy also be irq-safe. Non irq-safe domains
>> may continue to have irq-safe devices, but not the other way around.
>>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Kevin Hilman <khilman@linaro.org>
>> Signed-off-by: Lina Iyer <lina.iyer@linaro.org>
>> ---
>>  drivers/base/power/domain.c | 200 ++++++++++++++++++++++++++++++++++----------
>>  include/linux/pm_domain.h   |  11 ++-
>>  2 files changed, 164 insertions(+), 47 deletions(-)
>>
>> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
>> index dfd7595..8b89d15 100644
>> --- a/drivers/base/power/domain.c
>> +++ b/drivers/base/power/domain.c
>> @@ -50,6 +50,71 @@
>>  static LIST_HEAD(gpd_list);
>>  static DEFINE_MUTEX(gpd_list_lock);
>>
>> +static inline int genpd_lock_domain_noirq(struct generic_pm_domain *genpd,
>
>Nitpick:
>
>I would prefer a bit shorter names of all these new locking functions.
>In general, I think you can remove "domain" from the names. In this
>case it would mean genpd_lock_noirq().
>
Done.

>> +                                       unsigned int subclass)
>> +       __acquires(&genpd->slock)
>> +{
>> +       unsigned long flags;
>> +
>> +       if (unlikely(subclass > 0))
>> +               spin_lock_irqsave_nested(&genpd->slock, flags, subclass);
>> +       else
>> +               spin_lock_irqsave(&genpd->slock, flags);
>> +
>> +       genpd->flags = flags;
>
>This is wrong, it should be genpd->lock_flags that you assign.
>
Argh. My bad. Fixed.

>BTW, can't you assign the genpd->lock_flags immediately when call
>spin_lock_irqsave*(), instead of keeping a local copy?
>
I vaguely remember reading that in some architecture you needed to use a
stack variable to save the IRQ state . I cant seem to find it now.
I can change it.

>> +
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_unlock_domain_noirq(struct generic_pm_domain *genpd)
>> +       __releases(&genpd->slock)
>> +{
>> +       spin_unlock_irqrestore(&genpd->slock, genpd->lock_flags);
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_irq(struct generic_pm_domain *genpd,
>> +                                       unsigned int subclass)
>> +       __acquires(&genpd->mlock)
>> +{
>> +       if (unlikely(subclass > 0))
>> +               mutex_lock_nested(&genpd->mlock, subclass);
>> +       else
>> +               mutex_lock(&genpd->mlock);
>> +
>> +       return 0;
>> +}
>> +
>> +static inline int genpd_lock_domain_interruptible_irq(
>> +                               struct generic_pm_domain *genpd)
>> +       __acquires(&genpd->mlock)
>> +{
>> +       return mutex_lock_interruptible(&genpd->mlock);
>> +}
>> +
>> +static inline int genpd_unlock_domain_irq(struct generic_pm_domain *genpd)
>> +       __releases(&genpd->mlock)
>> +{
>> +       mutex_unlock(&genpd->mlock);
>> +       return 0;
>> +}
>> +
>> +#define genpd_lock_domain(genpd)                               \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>> +                       : genpd_lock_domain_irq(genpd, 0))
>> +
>> +#define genpd_lock_domain_nested(genpd)                                \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, SINGLE_DEPTH_NESTING)\
>> +                       : genpd_lock_domain_irq(genpd, SINGLE_DEPTH_NESTING))
>> +
>> +#define genpd_unlock_domain(genpd)                             \
>> +       (genpd->irq_safe ? genpd_unlock_domain_noirq(genpd)     \
>> +                       : genpd_unlock_domain_irq(genpd))
>> +
>> +#define genpd_lock_domain_interruptible(genpd)                 \
>> +       (genpd->irq_safe ? genpd_lock_domain_noirq(genpd, 0)    \
>> +                       : genpd_lock_domain_interruptible_irq(genpd))
>
>In general I don't like macros (as Krzysztof).
>
>In this case I also don't see the benfit, could you consider to
>convert to in-line functions instead?
>
Done. Since I have no strong preference either way, I will change this
to inline functions.

>> +
>>  static struct generic_pm_domain *pm_genpd_lookup_name(const char *domain_name)
>>  {
>>         struct generic_pm_domain *genpd = NULL, *gpd;
>> @@ -262,9 +327,9 @@ int pm_genpd_poweron(struct generic_pm_domain *genpd)
>>  {
>>         int ret;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         ret = __pm_genpd_poweron(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>  }
>>
>> @@ -326,9 +391,9 @@ static int genpd_dev_pm_qos_notifier(struct notifier_block *nb,
>>                 spin_unlock_irq(&dev->power.lock);
>>
>>                 if (!IS_ERR(genpd)) {
>> -                       mutex_lock(&genpd->lock);
>> +                       genpd_lock_domain(genpd);
>>                         genpd->max_off_time_changed = true;
>> -                       mutex_unlock(&genpd->lock);
>> +                       genpd_unlock_domain(genpd);
>>                 }
>>
>>                 dev = dev->parent;
>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain *genpd)
>>                         return -EBUSY;
>>
>>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>> -                   || pdd->dev->power.irq_safe))
>> +                       || (pdd->dev->power.irq_safe && !genpd->irq_safe)))
>
>This deserves a comment in the code.
>
This check should simplify to
		if (!pm_runtime_suspended(pdd->dev))

The other connditions seem unnecessary just to determine if the devices are
suspended.

Do you see any problem with that?

>>                         not_suspended++;
>>         }
>>
>> @@ -453,9 +518,9 @@ static void genpd_power_off_work_fn(struct work_struct *work)
>>
>>         genpd = container_of(work, struct generic_pm_domain, power_off_work);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         pm_genpd_poweroff(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>  }
>>
>>  /**
>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return -EINVAL;
>>
>> -       /*
>> -        * We can't allow to power off the PM domain if it holds an irq_safe
>> -        * device. That's beacuse we use mutexes to protect data while power
>> -        * off and on the PM domain, thus we can't execute in atomic context.
>> -        */
>> -       if (dev->power.irq_safe)
>> +       /* We can't allow to power off a domain that is also not irq safe. */
>
>I wouldn't mind to keep some more pieces of the earlier comment, since
>it explains a bit more of the *why*. Could you try to rephrase this
>comment in that regard?
>
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>
>This looks correct...
>
Apparently not the best place to check this. We should still allow
runtime suspend of the device, irrespective of whether the domain is IRQ
safe or not and this check here is incorrect.

>>                 return -EBUSY;
>>
>>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device *dev)
>>                 return ret;
>>         }
>>
>> -       mutex_lock(&genpd->lock);
>> +       /*
>> +        * If power.irq_safe is set, this routine will be run with interrupts
>> +        * off, so suspend only if the power domain is irq_safe.
>> +        */
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>> +               return 0;
>
>... but this doesn't. You have already returned -EBUSY above for this case.
>
This is probably more appropriate, as we want to skip locking the
domain, because it may be in atomic context and the domain may use
mutexes to lock.

I feel like, we miss out an opportunity here to power off the domain, if
device is IRQ safe while the domain is not, but the function is called
from a process context. I am not sure how to check that correctly, yet.

>> +
>> +       genpd_lock_domain(genpd);
>> +
>>         genpd->in_progress++;
>>         pm_genpd_poweroff(genpd);
>>         genpd->in_progress--;
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return 0;
>>  }
>> @@ -528,13 +597,16 @@ static int pm_genpd_runtime_resume(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return -EINVAL;
>>
>> -       /* If power.irq_safe, the PM domain is never powered off. */
>> -       if (dev->power.irq_safe)
>> +       /*
>> +        * If power.irq_safe and domain is not, then
>
>Let's take the opportunity to rephrase this comment a bit.
>
>Perhaps something along the lines: "As we don't power off a non IRQ
>safe PM domain which holds an IRQ safe device, we don't need to
>restore the power to it."
>
Done. Makes sense.

>> +        * the PM domain is never powered off.
>> +        */
>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>                 return genpd_start_dev_no_timing(genpd, dev);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>         ret = __pm_genpd_poweron(genpd);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (ret)
>>                 return ret;
>> @@ -729,14 +801,14 @@ static int pm_genpd_prepare(struct device *dev)
>>         if (resume_needed(dev, genpd))
>>                 pm_runtime_resume(dev);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count++ == 0) {
>>                 genpd->suspended_count = 0;
>>                 genpd->suspend_power_off = genpd->status == GPD_STATE_POWER_OFF;
>>         }
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (genpd->suspend_power_off) {
>>                 pm_runtime_put_noidle(dev);
>> @@ -754,12 +826,12 @@ static int pm_genpd_prepare(struct device *dev)
>>
>>         ret = pm_generic_prepare(dev);
>>         if (ret) {
>> -               mutex_lock(&genpd->lock);
>> +               genpd_lock_domain(genpd);
>>
>>                 if (--genpd->prepared_count == 0)
>>                         genpd->suspend_power_off = false;
>>
>> -               mutex_unlock(&genpd->lock);
>> +               genpd_unlock_domain(genpd);
>>                 pm_runtime_enable(dev);
>>         }
>>
>> @@ -1116,13 +1188,13 @@ static void pm_genpd_complete(struct device *dev)
>>         if (IS_ERR(genpd))
>>                 return;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         run_complete = !genpd->suspend_power_off;
>>         if (--genpd->prepared_count == 0)
>>                 genpd->suspend_power_off = false;
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (run_complete) {
>>                 pm_generic_complete(dev);
>> @@ -1266,11 +1338,18 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev))
>>                 return -EINVAL;
>>
>> +       /* Devices in an IRQ safe PM Domain have to be irq safe too */
>
>Let's be consistent and stick with IRQ in upper case letters for the
>comments. I think there are other places in the patch which needs
>update as well.
>
Sorry. I will stick to IRQ and CPU (thanks Kevin) henceforth. 

>> +       if (genpd->irq_safe && !dev->power.irq_safe) {
>> +               dev_warn(dev,
>
>dev_err(), or perhaps even WARN() ?
>
OK, they are now WARN.

>> +                       "Devices in an irq-safe domain have to be irq safe.\n");
>> +               return -EINVAL;
>> +       }
>> +
>>         gpd_data = genpd_alloc_dev_data(dev, genpd, td);
>>         if (IS_ERR(gpd_data))
>>                 return PTR_ERR(gpd_data);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count > 0) {
>>                 ret = -EAGAIN;
>> @@ -1287,7 +1366,7 @@ int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev,
>>         list_add_tail(&gpd_data->base.list_node, &genpd->dev_list);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         if (ret)
>>                 genpd_free_dev_data(dev, gpd_data);
>> @@ -1331,7 +1410,7 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>         gpd_data = to_gpd_data(pdd);
>>         dev_pm_qos_remove_notifier(dev, &gpd_data->nb);
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->prepared_count > 0) {
>>                 ret = -EAGAIN;
>> @@ -1346,14 +1425,14 @@ int pm_genpd_remove_device(struct generic_pm_domain *genpd,
>>
>>         list_del_init(&pdd->list_node);
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         genpd_free_dev_data(dev, gpd_data);
>>
>>         return 0;
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         dev_pm_qos_add_notifier(dev, &gpd_data->nb);
>>
>>         return ret;
>> @@ -1374,12 +1453,23 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>             || genpd == subdomain)
>>                 return -EINVAL;
>>
>> +       /*
>> +        * If the domain can be powered on/off in an irq safe
>> +        * context, ensure that the subdomain can also be
>> +        * powered on/off in that context.
>> +        */
>
>Have you considered if/how to address this limitation?
>
>Just to be clear, for now I think this limitation is perfectly okay,
>but going forward we might see a need to mix non IRQ safe domains with
>IRQ safe domains, via sub-domains.
>
Yeah, if we could be cognizant of the context we are running in, we
can be better about operating on domain and devices. Then this
limitation would go away.

>> +       if (genpd->irq_safe && !subdomain->irq_safe) {
>> +               pr_warn("Incompatible sub-domain %s of irq-safe domain %s\n",
>> +                               subdomain->name, genpd->name);
>
>Similar comment as earlier. Perhaps pr_err() or WARN() instead.
>
Done.

>> +               return -EINVAL;
>> +       }
>> +
>>         link = kzalloc(sizeof(*link), GFP_KERNEL);
>>         if (!link)
>>                 return -ENOMEM;
>>
>> -       mutex_lock(&genpd->lock);
>> -       mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>> +       genpd_lock_domain(genpd);
>> +       genpd_lock_domain_nested(subdomain);
>>
>>         if (genpd->status == GPD_STATE_POWER_OFF
>>             &&  subdomain->status != GPD_STATE_POWER_OFF) {
>> @@ -1402,8 +1492,8 @@ int pm_genpd_add_subdomain(struct generic_pm_domain *genpd,
>>                 genpd_sd_counter_inc(genpd);
>>
>>   out:
>> -       mutex_unlock(&subdomain->lock);
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(subdomain);
>> +       genpd_unlock_domain(genpd);
>>
>>         return ret;
>>  }
>> @@ -1451,13 +1541,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>>         if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(subdomain))
>>                 return -EINVAL;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         list_for_each_entry(link, &genpd->master_links, master_node) {
>>                 if (link->slave != subdomain)
>>                         continue;
>>
>> -               mutex_lock_nested(&subdomain->lock, SINGLE_DEPTH_NESTING);
>> +               genpd_lock_domain_nested(subdomain);
>>
>>                 list_del(&link->master_node);
>>                 list_del(&link->slave_node);
>> @@ -1465,13 +1555,13 @@ int pm_genpd_remove_subdomain(struct generic_pm_domain *genpd,
>>                 if (subdomain->status != GPD_STATE_POWER_OFF)
>>                         genpd_sd_counter_dec(genpd);
>>
>> -               mutex_unlock(&subdomain->lock);
>> +               genpd_unlock_domain(subdomain);
>>
>>                 ret = 0;
>>                 break;
>>         }
>>
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return ret;
>>  }
>> @@ -1495,11 +1585,17 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>         if (IS_ERR_OR_NULL(genpd) || state < 0)
>>                 return -EINVAL;
>>
>> +       if (genpd->irq_safe) {
>> +               pr_err("Domain %s is irq safe, cannot attach to cpuidle\n",
>> +                               genpd->name);
>
>Here you have pr_err(), which is okay but not consistent. Also, I
>wonder if we should use a WARN() instead?
>
Done.

>> +               return -EINVAL;
>> +       }
>> +
>>         cpuidle_data = kzalloc(sizeof(*cpuidle_data), GFP_KERNEL);
>>         if (!cpuidle_data)
>>                 return -ENOMEM;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         if (genpd->cpuidle_data) {
>>                 ret = -EEXIST;
>> @@ -1526,7 +1622,7 @@ int pm_genpd_attach_cpuidle(struct generic_pm_domain *genpd, int state)
>>         genpd_recalc_cpu_exit_latency(genpd);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>
>>   err:
>> @@ -1563,7 +1659,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>>         if (IS_ERR_OR_NULL(genpd))
>>                 return -EINVAL;
>>
>> -       mutex_lock(&genpd->lock);
>> +       genpd_lock_domain(genpd);
>>
>>         cpuidle_data = genpd->cpuidle_data;
>>         if (!cpuidle_data) {
>> @@ -1581,7 +1677,7 @@ int pm_genpd_detach_cpuidle(struct generic_pm_domain *genpd)
>>         kfree(cpuidle_data);
>>
>>   out:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>         return ret;
>>  }
>>
>> @@ -1642,6 +1738,17 @@ static int pm_genpd_default_restore_state(struct device *dev)
>>         return cb ? cb(dev) : 0;
>>  }
>>
>> +static void genpd_lock_domain_init(struct generic_pm_domain *genpd)
>> +{
>> +       if (genpd->flags & GENPD_FLAG_IRQ_SAFE) {
>> +               spin_lock_init(&genpd->slock);
>> +               genpd->irq_safe = true;
>
>Copying the result from "genpd->flags & GENPD_FLAG_IRQ_SAFE" into the
>irq_safe bool, seems like a "micro" optimization. How about only using
>the genpd->flags instead?
>
Actually, I would prefer using the flag, there are a few places in the
code where I need to know if the domain is IRQ safe or not. Checking for
flags is cumbersome and not every aestheic.

>> +       } else {
>> +               mutex_init(&genpd->mlock);
>> +               genpd->irq_safe = false;
>> +       }
>> +}
>> +
>>  /**
>>   * pm_genpd_init - Initialize a generic I/O PM domain object.
>>   * @genpd: PM domain object to initialize.
>> @@ -1657,7 +1764,7 @@ void pm_genpd_init(struct generic_pm_domain *genpd,
>>         INIT_LIST_HEAD(&genpd->master_links);
>>         INIT_LIST_HEAD(&genpd->slave_links);
>>         INIT_LIST_HEAD(&genpd->dev_list);
>> -       mutex_init(&genpd->lock);
>> +       genpd_lock_domain_init(genpd);
>>         genpd->gov = gov;
>>         INIT_WORK(&genpd->power_off_work, genpd_power_off_work_fn);
>>         genpd->in_progress = 0;
>> @@ -2042,7 +2149,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>         struct gpd_link *link;
>>         int ret;
>>
>> -       ret = mutex_lock_interruptible(&genpd->lock);
>> +       ret = genpd_lock_domain_interruptible(genpd);
>>         if (ret)
>>                 return -ERESTARTSYS;
>>
>> @@ -2062,7 +2169,8 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>         }
>>
>>         list_for_each_entry(pm_data, &genpd->dev_list, list_node) {
>> -               kobj_path = kobject_get_path(&pm_data->dev->kobj, GFP_KERNEL);
>> +               kobj_path = kobject_get_path(&pm_data->dev->kobj,
>> +                               genpd->irq_safe ? GFP_ATOMIC : GFP_KERNEL);
>>                 if (kobj_path == NULL)
>>                         continue;
>>
>> @@ -2073,7 +2181,7 @@ static int pm_genpd_summary_one(struct seq_file *s,
>>
>>         seq_puts(s, "\n");
>>  exit:
>> -       mutex_unlock(&genpd->lock);
>> +       genpd_unlock_domain(genpd);
>>
>>         return 0;
>>  }
>> diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
>> index b2725e6..dc7cb53 100644
>> --- a/include/linux/pm_domain.h
>> +++ b/include/linux/pm_domain.h
>> @@ -16,9 +16,11 @@
>>  #include <linux/of.h>
>>  #include <linux/notifier.h>
>>  #include <linux/cpuidle.h>
>> +#include <linux/spinlock.h>
>>
>>  /* Defines used for the flags field in the struct generic_pm_domain */
>>  #define GENPD_FLAG_PM_CLK      (1U << 0) /* PM domain uses PM clk */
>> +#define GENPD_FLAG_IRQ_SAFE    (1U << 1) /* PM domain operates in atomic */
>>
>>  enum gpd_status {
>>         GPD_STATE_ACTIVE = 0,   /* PM domain is active */
>> @@ -49,7 +51,6 @@ struct generic_pm_domain {
>>         struct list_head master_links;  /* Links with PM domain as a master */
>>         struct list_head slave_links;   /* Links with PM domain as a slave */
>>         struct list_head dev_list;      /* List of devices */
>> -       struct mutex lock;
>>         struct dev_power_governor *gov;
>>         struct work_struct power_off_work;
>>         const char *name;
>> @@ -74,6 +75,14 @@ struct generic_pm_domain {
>>         void (*detach_dev)(struct generic_pm_domain *domain,
>>                            struct device *dev);
>>         unsigned int flags;             /* Bit field of configs for genpd */
>> +       bool irq_safe;
>> +       union {
>> +               struct mutex mlock;
>> +               struct {
>> +                       spinlock_t slock;
>> +                       unsigned long lock_flags;
>> +               };
>> +       };
>>  };
>>
>>  static inline struct generic_pm_domain *pd_to_genpd(struct dev_pm_domain *pd)
>> --
>> 2.1.4
>>

Thank you so much Ulf for the review.

--Lina

Re: [PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Ulf Hansson]

[...]

>>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain
>>> *genpd)
>>>                         return -EBUSY;
>>>
>>>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>>> -                   || pdd->dev->power.irq_safe))
>>> +                       || (pdd->dev->power.irq_safe &&
>>> !genpd->irq_safe)))
>>
>>
>> This deserves a comment in the code.
>>
> This check should simplify to
>                 if (!pm_runtime_suspended(pdd->dev))
>
> The other connditions seem unnecessary just to determine if the devices are
> suspended.
>
> Do you see any problem with that?

I am not sure, need to think a bit more about it. Anyway, I wouldn't
do the "simplification" in $subject patch, since it's a related to
different topic.

[...]

>>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device
>>> *dev)
>>>         if (IS_ERR(genpd))
>>>                 return -EINVAL;
>>>
>>> -       /*
>>> -        * We can't allow to power off the PM domain if it holds an
>>> irq_safe
>>> -        * device. That's beacuse we use mutexes to protect data while
>>> power
>>> -        * off and on the PM domain, thus we can't execute in atomic
>>> context.
>>> -        */
>>> -       if (dev->power.irq_safe)
>>> +       /* We can't allow to power off a domain that is also not irq
>>> safe. */
>>
>>
>> I wouldn't mind to keep some more pieces of the earlier comment, since
>> it explains a bit more of the *why*. Could you try to rephrase this
>> comment in that regard?
>>
>>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>
>>
>> This looks correct...
>>
> Apparently not the best place to check this. We should still allow
> runtime suspend of the device, irrespective of whether the domain is IRQ
> safe or not and this check here is incorrect.

You are absolutely correct!

That change is actually done as a part of the patch I posted [1],
which $subject patch is based upon.

Earlier we could only invoke the ->stop|start() callbacks for an IRQ
safe device. According to the changes made in [1], we are now able to
also invoke the ->runtime_suspend|resume() callbacks. Yet another
positive "side effect". I will cook a v4 and post a new version to fix
this.

>
>>>                 return -EBUSY;
>>>
>>>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device
>>> *dev)
>>>                 return ret;
>>>         }
>>>
>>> -       mutex_lock(&genpd->lock);
>>> +       /*
>>> +        * If power.irq_safe is set, this routine will be run with
>>> interrupts
>>> +        * off, so suspend only if the power domain is irq_safe.
>>> +        */
>>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>> +               return 0;
>>
>>
>> ... but this doesn't. You have already returned -EBUSY above for this
>> case.
>>
> This is probably more appropriate, as we want to skip locking the
> domain, because it may be in atomic context and the domain may use
> mutexes to lock.

Yep.

>
> I feel like, we miss out an opportunity here to power off the domain, if
> device is IRQ safe while the domain is not, but the function is called
> from a process context. I am not sure how to check that correctly, yet.

You may be able to power off the PM domain for some cases, yes. But...
if you do that, how would you then be able to power on the PM domain
from atomic context via pm_genpd_runtime_resume()?

[...]

[1]
http://www.spinics.net/lists/arm-kernel/msg424994.html

Kind regards
Uffe

[PATCH RFC 2/3] PM / Domains: Support atomic PM domains

[Ulf Hansson]

[...]

>>> @@ -387,7 +452,7 @@ static int pm_genpd_poweroff(struct generic_pm_domain
>>> *genpd)
>>>                         return -EBUSY;
>>>
>>>                 if (pdd->dev->driver && (!pm_runtime_suspended(pdd->dev)
>>> -                   || pdd->dev->power.irq_safe))
>>> +                       || (pdd->dev->power.irq_safe &&
>>> !genpd->irq_safe)))
>>
>>
>> This deserves a comment in the code.
>>
> This check should simplify to
>                 if (!pm_runtime_suspended(pdd->dev))
>
> The other connditions seem unnecessary just to determine if the devices are
> suspended.
>
> Do you see any problem with that?

I am not sure, need to think a bit more about it. Anyway, I wouldn't
do the "simplification" in $subject patch, since it's a related to
different topic.

[...]

>>> @@ -478,12 +543,8 @@ static int pm_genpd_runtime_suspend(struct device
>>> *dev)
>>>         if (IS_ERR(genpd))
>>>                 return -EINVAL;
>>>
>>> -       /*
>>> -        * We can't allow to power off the PM domain if it holds an
>>> irq_safe
>>> -        * device. That's beacuse we use mutexes to protect data while
>>> power
>>> -        * off and on the PM domain, thus we can't execute in atomic
>>> context.
>>> -        */
>>> -       if (dev->power.irq_safe)
>>> +       /* We can't allow to power off a domain that is also not irq
>>> safe. */
>>
>>
>> I wouldn't mind to keep some more pieces of the earlier comment, since
>> it explains a bit more of the *why*. Could you try to rephrase this
>> comment in that regard?
>>
>>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>
>>
>> This looks correct...
>>
> Apparently not the best place to check this. We should still allow
> runtime suspend of the device, irrespective of whether the domain is IRQ
> safe or not and this check here is incorrect.

You are absolutely correct!

That change is actually done as a part of the patch I posted [1],
which $subject patch is based upon.

Earlier we could only invoke the ->stop|start() callbacks for an IRQ
safe device. According to the changes made in [1], we are now able to
also invoke the ->runtime_suspend|resume() callbacks. Yet another
positive "side effect". I will cook a v4 and post a new version to fix
this.

>
>>>                 return -EBUSY;
>>>
>>>         stop_ok = genpd->gov ? genpd->gov->stop_ok : NULL;
>>> @@ -500,11 +561,19 @@ static int pm_genpd_runtime_suspend(struct device
>>> *dev)
>>>                 return ret;
>>>         }
>>>
>>> -       mutex_lock(&genpd->lock);
>>> +       /*
>>> +        * If power.irq_safe is set, this routine will be run with
>>> interrupts
>>> +        * off, so suspend only if the power domain is irq_safe.
>>> +        */
>>> +       if (dev->power.irq_safe && !genpd->irq_safe)
>>> +               return 0;
>>
>>
>> ... but this doesn't. You have already returned -EBUSY above for this
>> case.
>>
> This is probably more appropriate, as we want to skip locking the
> domain, because it may be in atomic context and the domain may use
> mutexes to lock.

Yep.

>
> I feel like, we miss out an opportunity here to power off the domain, if
> device is IRQ safe while the domain is not, but the function is called
> from a process context. I am not sure how to check that correctly, yet.

You may be able to power off the PM domain for some cases, yes. But...
if you do that, how would you then be able to power on the PM domain
from atomic context via pm_genpd_runtime_resume()?

[...]

[1]
http://www.spinics.net/lists/arm-kernel/msg424994.html

Kind regards
Uffe

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>Lina Iyer <lina.iyer@linaro.org> writes:
>>
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state, the cpu domain can also be
>>> powered off.
>>
>>How does this relate to a cluster, and why aren't you using that terminolgy?
>>
>>> Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>>
>>> Introduce support for defining and adding a generic power domain for the
>>> cpus based on the DT specification of power domain providers and
>>> consumers.  SoC's that have the cpu domain defined in their DT, can
>>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>>> this domain with the genpd.
>>>
>>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>>> last cpu going down would call the genpd->power_off(). Correspondingly,
>>> the first cpu up would call the genpd->power_on() callback before
>>> resuming from idle.
>>
>>Other patches also mention this genpd being useful to gate power to
>>non-CPU peripherals on the same power rail.  How are those devices to be
>>added?
>>
> I am not investigating DT nodes to figure out which node is a consumer
> for this domain provider. That could be a good way to do it, but
> practically speaking, there may be platform dependencies and specifics
> that may need to be met, before the device can be added to the CPU
> genpd.
>
> So that is not generalized here. I didn't see a better way to do that,
> generically. Do you have ideas on that?
>
> The platform is the owner of the genpd and therefore can add those
> non-cpu devices to the genpd as and when appropritate.

I'm pretty sure the generic code will already add devices to genpds if
the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
the extra stuff for CPUs is needed.

> In this regard, I also have a question, who initializes the genpd. My
> assumption is that CPUs will get probed before most other devices and
> therefore the domain provider could be initialized by this file. But, I
> could be wrong here.

Initializing it in this driver seems OK to me.

Kevin

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>Lina Iyer <lina.iyer@linaro.org> writes:
>>
>>> Generally cpus are grouped under a power domain in a SoC. When all cpus
>>> in the domain are in their power off state, the cpu domain can also be
>>> powered off.
>>
>>How does this relate to a cluster, and why aren't you using that terminolgy?
>>
>>> Genpd provides the framework for defining cpus as devices
>>> that are part of a cpu domain.
>>>
>>> Introduce support for defining and adding a generic power domain for the
>>> cpus based on the DT specification of power domain providers and
>>> consumers.  SoC's that have the cpu domain defined in their DT, can
>>> setup a genpd with a name and the power_on/power_off callbacks. Calling
>>> pm_cpu_domain_init() will register the genpd and attach the cpus for
>>> this domain with the genpd.
>>>
>>> CPU_PM notifications for are used to pm_runtime_get_sync() and
>>> pm_runtime_put_sync() for each cpu.  When all cpus are powered off, the
>>> last cpu going down would call the genpd->power_off(). Correspondingly,
>>> the first cpu up would call the genpd->power_on() callback before
>>> resuming from idle.
>>
>>Other patches also mention this genpd being useful to gate power to
>>non-CPU peripherals on the same power rail.  How are those devices to be
>>added?
>>
> I am not investigating DT nodes to figure out which node is a consumer
> for this domain provider. That could be a good way to do it, but
> practically speaking, there may be platform dependencies and specifics
> that may need to be met, before the device can be added to the CPU
> genpd.
>
> So that is not generalized here. I didn't see a better way to do that,
> generically. Do you have ideas on that?
>
> The platform is the owner of the genpd and therefore can add those
> non-cpu devices to the genpd as and when appropritate.

I'm pretty sure the generic code will already add devices to genpds if
the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
the extra stuff for CPUs is needed.

> In this regard, I also have a question, who initializes the genpd. My
> assumption is that CPUs will get probed before most other devices and
> therefore the domain provider could be initialized by this file. But, I
> could be wrong here.

Initializing it in this driver seems OK to me.

Kevin

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Mon, Jun 15 2015 at 12:43 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>>Lina Iyer <lina.iyer@linaro.org> writes:


>I'm pretty sure the generic code will already add devices to genpds if
>the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
>the extra stuff for CPUs is needed.
>
I dont see that automatically happening. When I attach a device, it
finds the corresponding genpd provider and attaches the device.  But I
dont see in any code that creates genpd and find the related device
nodes and adds them to the genpd.

May be I am missing something.

-- Lina

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Lina Iyer]

On Mon, Jun 15 2015 at 12:43 -0600, Kevin Hilman wrote:
>Lina Iyer <lina.iyer@linaro.org> writes:
>
>> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>>Lina Iyer <lina.iyer@linaro.org> writes:


>I'm pretty sure the generic code will already add devices to genpds if
>the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
>the extra stuff for CPUs is needed.
>
I dont see that automatically happening. When I attach a device, it
finds the corresponding genpd provider and attaches the device.  But I
dont see in any code that creates genpd and find the related device
nodes and adds them to the genpd.

May be I am missing something.

-- Lina

Re: [PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> On Mon, Jun 15 2015 at 12:43 -0600, Kevin Hilman wrote:
>>Lina Iyer <lina.iyer@linaro.org> writes:
>>
>>> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>>>Lina Iyer <lina.iyer@linaro.org> writes:
>
>
>>I'm pretty sure the generic code will already add devices to genpds if
>>the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
>>the extra stuff for CPUs is needed.
>>
> I dont see that automatically happening. When I attach a device, it
> finds the corresponding genpd provider and attaches the device.  But I
> dont see in any code that creates genpd and find the related device
> nodes and adds them to the genpd.

[summary from our IRC discussion]

You still need to create the genpd, but it's dev_pm_domain_attach()
called by the platform device probe path that will automaticaly try to
attach a device with a power-domains property to the correct PM domain.

Note that this assumes that the genpds are created before the devices
are probed.

Kevin

[PATCH RFC 3/3] PM / Domains: Introduce generic PM domain for cpu domain

[Kevin Hilman]

Lina Iyer <lina.iyer@linaro.org> writes:

> On Mon, Jun 15 2015 at 12:43 -0600, Kevin Hilman wrote:
>>Lina Iyer <lina.iyer@linaro.org> writes:
>>
>>> On Wed, Jun 10 2015 at 15:38 -0600, Kevin Hilman wrote:
>>>>Lina Iyer <lina.iyer@linaro.org> writes:
>
>
>>I'm pretty sure the generic code will already add devices to genpds if
>>the genpd is using the of_genpd_* stuff.  That is why I'm wondering why
>>the extra stuff for CPUs is needed.
>>
> I dont see that automatically happening. When I attach a device, it
> finds the corresponding genpd provider and attaches the device.  But I
> dont see in any code that creates genpd and find the related device
> nodes and adds them to the genpd.

[summary from our IRC discussion]

You still need to create the genpd, but it's dev_pm_domain_attach()
called by the platform device probe path that will automaticaly try to
attach a device with a power-domains property to the correct PM domain.

Note that this assumes that the genpds are created before the devices
are probed.

Kevin