[RFC PATCH 0/2] Better domain idle from device wakeup patterns

[RFC PATCH 0/2] Better domain idle from device wakeup patterns

[Lina Iyer]

Hello,

I was looking for an option to do better power management for some
domains where the devices enter runtime PM in a predictable fashion. For
example a display device that sends a vsync interrupt every 16 ms for a
60 Hz panel. These interrupts are not timer interrupts but tend to
interrupt periodically to service the workflow and the devices and
domains may go back to idle soon after. Two domains are affected by this
- the device's PM domain and the CPU PM domain.

As a first step, I am looking to solve for the device's PM domain idle
state (and hopefully solve for the CPU PM domains subsequently). The PM
domain could have multiple idle states and/or the enter/exit latencies
could be high. In either case, it may not always be beneficial to power
off the domain, only to turn it back on before satisfying the idle state
residency. When the wakeup is known for the device, we could use that to
determine the worthiness of entering a domain idle state. Only the
device can tell us when the future event would be and that could change
as the usecase changes. Like, when the panel refresh rate increases to
120 Hz. If this information was made available to runtime PM, we could
use that in the domain governor to determine a suitable idle state. This
is the idea behind these patches.

In the first patch, I am proposing an API for devices to specify their
wakeup as a time in the future and in the second patch, I am updating
the PM domain governor to use this information to determine the idle
state. I have not had a chance to test this out yet, but I wanted to
know if I am on the right track.

Would appreciate your thoughts on this.

Thanks,
Lina


Lina Iyer (2):
  PM / runtime: register device's next wakeup
  PM / Domains: use device's next wakeup to determine domain idle state

 drivers/base/power/domain_governor.c | 87 ++++++++++++++++++++++++++--
 drivers/base/power/runtime.c         | 31 ++++++++++
 include/linux/pm.h                   |  2 +
 include/linux/pm_domain.h            |  1 +
 include/linux/pm_runtime.h           |  1 +
 5 files changed, 116 insertions(+), 6 deletions(-)

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[RFC PATCH 1/2] PM / runtime: register device's next wakeup

[Lina Iyer]

Some devices have a predictable interrupt pattern while executing a
particular usecase. An example would be the VSYNC interrupt on devices
associated with displays. A 60 Hz display could cause a periodic
interrupt every 16 ms. A PM domain that holds such a device could power
off and on at similar intervals. But when the domain is a supplier for
multiple such devices, the idle pattern for the domain is bit muddled,
but still deducible.

If the next wakeup is known for all devices in a domain, it becomes easy
to find out the sleep duration for the domain. By allowing drivers to
notify runtime PM of their device's future wakeup, we can make better PM
domain governor idle state decisions when powering off the domain.

For now, let's allow updating the device's next wake up as long as
runtime PM is enabled. If the device is runtime suspended, the domain
could be as well and we don't want to wake up the domain to recompute
the domain idle state. As a first step, the onus is on the drivers to
update their device's wakeup correctly before the device and the domain
enter runtime idle.

Signed-off-by: Lina Iyer <ilina@codeaurora.org>
---
 drivers/base/power/runtime.c | 31 +++++++++++++++++++++++++++++++
 include/linux/pm.h           |  2 ++
 include/linux/pm_runtime.h   |  1 +
 3 files changed, 34 insertions(+)

diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
index 8143210a5c54..53c2b3d962bc 100644
--- a/drivers/base/power/runtime.c
+++ b/drivers/base/power/runtime.c
@@ -122,6 +122,33 @@ u64 pm_runtime_suspended_time(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(pm_runtime_suspended_time);
 
+/**
+ * pm_runtime_set_next_wakeup_event - Notify PM framework of an impending event.
+ * @dev: Device to handle
+ * @next: impending interrupt/wakeup for the device
+ */
+int pm_runtime_set_next_event(struct device *dev, ktime_t next)
+{
+	unsigned long flags;
+	int ret = -EINVAL;
+
+	/*
+	 * Note the next pending wakeup of a device,
+	 * if the device does not have runtime PM enabled.
+	 */
+	spin_lock_irqsave(&dev->power.lock, flags);
+	if (!dev->power.disable_depth) {
+		if (ktime_before(ktime_get(), next)) {
+			dev->power.next_event = next;
+			ret = 0;
+		}
+	}
+	spin_unlock_irqrestore(&dev->power.lock, flags);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(pm_runtime_set_next_event);
+
 /**
  * pm_runtime_deactivate_timer - Deactivate given device's suspend timer.
  * @dev: Device to handle.
@@ -1380,6 +1407,9 @@ void __pm_runtime_disable(struct device *dev, bool check_resume)
 	/* Update time accounting before disabling PM-runtime. */
 	update_pm_runtime_accounting(dev);
 
+	/* Reset the next wakeup for the device */
+	dev->power.next_event = KTIME_MAX;
+
 	if (!dev->power.disable_depth++)
 		__pm_runtime_barrier(dev);
 
@@ -1609,6 +1639,7 @@ void pm_runtime_init(struct device *dev)
 	dev->power.deferred_resume = false;
 	INIT_WORK(&dev->power.work, pm_runtime_work);
 
+	dev->power.next_event = KTIME_MAX;
 	dev->power.timer_expires = 0;
 	hrtimer_init(&dev->power.suspend_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
 	dev->power.suspend_timer.function = pm_suspend_timer_fn;
diff --git a/include/linux/pm.h b/include/linux/pm.h
index a30a4b54df52..9051658674a4 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -8,6 +8,7 @@
 #ifndef _LINUX_PM_H
 #define _LINUX_PM_H
 
+#include <linux/ktime.h>
 #include <linux/list.h>
 #include <linux/workqueue.h>
 #include <linux/spinlock.h>
@@ -616,6 +617,7 @@ struct dev_pm_info {
 	u64			active_time;
 	u64			suspended_time;
 	u64			accounting_timestamp;
+	ktime_t			next_event;
 #endif
 	struct pm_subsys_data	*subsys_data;  /* Owned by the subsystem. */
 	void (*set_latency_tolerance)(struct device *, s32);
diff --git a/include/linux/pm_runtime.h b/include/linux/pm_runtime.h
index 6245caa18034..af6d35178335 100644
--- a/include/linux/pm_runtime.h
+++ b/include/linux/pm_runtime.h
@@ -59,6 +59,7 @@ extern void pm_runtime_get_suppliers(struct device *dev);
 extern void pm_runtime_put_suppliers(struct device *dev);
 extern void pm_runtime_new_link(struct device *dev);
 extern void pm_runtime_drop_link(struct device *dev);
+extern int pm_runtime_set_next_event(struct device *dev, ktime_t next);
 
 /**
  * pm_runtime_get_if_in_use - Conditionally bump up runtime PM usage counter.
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[RFC PATCH 2/2] PM / Domains: use device's next wakeup to determine domain idle state

[Lina Iyer]

If the device's next event is known, determine if it is worthwhile
entering a domain idle state. To find the next wakeup, traverse a domain
for all child devices and find out the earliest wakeup value. A parent
domain's next wakeup is the earliest of all its child devices and
domains. The next wakeup is specified by devices in the domains before
they devices are suspended.

Update the domain governor logic to determine if it is worthwhile to
enter an idle state based on the next wakeup for the domain along with
other existing constraints.

Signed-off-by: Lina Iyer <ilina@codeaurora.org>
---
 drivers/base/power/domain_governor.c | 87 ++++++++++++++++++++++++++--
 include/linux/pm_domain.h            |  1 +
 2 files changed, 82 insertions(+), 6 deletions(-)

diff --git a/drivers/base/power/domain_governor.c b/drivers/base/power/domain_governor.c
index 490ed7deb99a..a71d7281e9c1 100644
--- a/drivers/base/power/domain_governor.c
+++ b/drivers/base/power/domain_governor.c
@@ -117,6 +117,56 @@ static bool default_suspend_ok(struct device *dev)
 	return td->cached_suspend_ok;
 }
 
+static void update_domain_next_wakeup(struct generic_pm_domain *genpd, ktime_t now)
+{
+	ktime_t domain_wakeup = KTIME_MAX;
+	ktime_t next_wakeup;
+	struct pm_domain_data *pdd;
+	struct gpd_link *link;
+
+	/* Find the earliest wakeup for all devices in the domain */
+	list_for_each_entry(pdd, &genpd->dev_list, list_node) {
+		if (pdd->dev->power.next_event != KTIME_MAX &&
+		    !ktime_before(pdd->dev->power.next_event, now)) {
+			next_wakeup = READ_ONCE(pdd->dev->power.next_event);
+			if (ktime_before(next_wakeup, domain_wakeup))
+				domain_wakeup = next_wakeup;
+		}
+	}
+
+	/* Then find the earliest wakeup of from all the child domains */
+	list_for_each_entry(link, &genpd->parent_links, parent_node) {
+		struct generic_pm_domain *sd = link->child;
+
+		next_wakeup = sd->next_wakeup;
+		if (next_wakeup != KTIME_MAX && !ktime_before(next_wakeup, now))
+			if (ktime_before(next_wakeup, domain_wakeup))
+				domain_wakeup = next_wakeup;
+	}
+
+	genpd->next_wakeup = domain_wakeup;
+}
+
+static bool next_wakeup_allows_state(struct generic_pm_domain *genpd,
+				     unsigned int state, ktime_t now)
+{
+	s64 idle_time_ns, min_sleep_ns;
+	ktime_t domain_wakeup = genpd->next_wakeup;
+
+	if (domain_wakeup == KTIME_MAX)
+		return true;
+
+	min_sleep_ns = genpd->states[state].power_off_latency_ns +
+		       genpd->states[state].power_on_latency_ns +
+		       genpd->states[state].residency_ns;
+
+	idle_time_ns = ktime_to_ns(ktime_sub(domain_wakeup, now));
+	if (idle_time_ns < min_sleep_ns)
+		return false;
+
+	return true;
+}
+
 static bool __default_power_down_ok(struct dev_pm_domain *pd,
 				     unsigned int state)
 {
@@ -210,6 +260,29 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 {
 	struct generic_pm_domain *genpd = pd_to_genpd(pd);
 	struct gpd_link *link;
+	unsigned int state_idx;
+	ktime_t now = ktime_get();
+
+	/*
+	 * Find the next wakeup from devices that can determine their own wakeup
+	 * to find when the domain would wakeup and do it for every device down
+	 * the hierarchy. It is not worth while to sleep if the state's residency
+	 * cannot be met.
+	 */
+	update_domain_next_wakeup(genpd, now);
+
+	/* Let's find out what domain idle state, the devices prefer */
+	genpd->state_idx = state_idx = genpd->state_count - 1;
+	while (!next_wakeup_allows_state(genpd, state_idx, now)) {
+		if (!state_idx) {
+			genpd->cached_power_down_ok = false;
+			return false;
+		}
+		state_idx--;
+	}
+
+	genpd->max_off_time_changed =
+		(state_idx == genpd->cached_power_down_state_idx);
 
 	if (!genpd->max_off_time_changed) {
 		genpd->state_idx = genpd->cached_power_down_state_idx;
@@ -228,18 +301,20 @@ static bool default_power_down_ok(struct dev_pm_domain *pd)
 	genpd->max_off_time_ns = -1;
 	genpd->max_off_time_changed = false;
 	genpd->cached_power_down_ok = true;
-	genpd->state_idx = genpd->state_count - 1;
 
-	/* Find a state to power down to, starting from the deepest. */
-	while (!__default_power_down_ok(pd, genpd->state_idx)) {
-		if (genpd->state_idx == 0) {
+	/* Find a state to power down to, starting from the state
+	 * determined by the next wakeup.
+	 */
+	while (!__default_power_down_ok(pd, state_idx)) {
+		if (state_idx == 0) {
 			genpd->cached_power_down_ok = false;
 			break;
 		}
-		genpd->state_idx--;
+		state_idx--;
 	}
 
-	genpd->cached_power_down_state_idx = genpd->state_idx;
+	genpd->state_idx = state_idx;
+	genpd->cached_power_down_state_idx = state_idx;
 	return genpd->cached_power_down_ok;
 }
 
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index ee11502a575b..9ea6f666967b 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -119,6 +119,7 @@ struct generic_pm_domain {
 				     unsigned int state);
 	struct gpd_dev_ops dev_ops;
 	s64 max_off_time_ns;	/* Maximum allowed "suspended" time. */
+	ktime_t next_wakeup;
 	bool max_off_time_changed;
 	bool cached_power_down_ok;
 	bool cached_power_down_state_idx;
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [RFC PATCH 0/2] Better domain idle from device wakeup patterns

[Ulf Hansson]

On Thu, 3 Sep 2020 at 01:26, Lina Iyer <ilina@codeaurora.org> wrote:
>
> Hello,
>
> I was looking for an option to do better power management for some
> domains where the devices enter runtime PM in a predictable fashion. For
> example a display device that sends a vsync interrupt every 16 ms for a
> 60 Hz panel. These interrupts are not timer interrupts but tend to
> interrupt periodically to service the workflow and the devices and
> domains may go back to idle soon after. Two domains are affected by this
> - the device's PM domain and the CPU PM domain.
>
> As a first step, I am looking to solve for the device's PM domain idle
> state (and hopefully solve for the CPU PM domains subsequently). The PM
> domain could have multiple idle states and/or the enter/exit latencies
> could be high. In either case, it may not always be beneficial to power
> off the domain, only to turn it back on before satisfying the idle state
> residency. When the wakeup is known for the device, we could use that to
> determine the worthiness of entering a domain idle state. Only the
> device can tell us when the future event would be and that could change
> as the usecase changes. Like, when the panel refresh rate increases to
> 120 Hz. If this information was made available to runtime PM, we could
> use that in the domain governor to determine a suitable idle state. This
> is the idea behind these patches.

While striving towards entering the most optimal (energy and
performance wise) idle state, I think it's an interesting approach.

>
> In the first patch, I am proposing an API for devices to specify their
> wakeup as a time in the future and in the second patch, I am updating
> the PM domain governor to use this information to determine the idle
> state. I have not had a chance to test this out yet, but I wanted to
> know if I am on the right track.

I don't have any immediate objections - I think the approach seems
reasonable. Still, my first thought that this could be an extension to
the dev_pm_qos interface, but perhaps that isn't a good fit.

I also have a couple of comments to the code, but I will reply to each
patch separately about that.

>
> Would appreciate your thoughts on this.

When it comes to showing real results, other than in theory, I think
we need to make the genpd's cpu governor to cope with the next wakeup
event as well.

>
> Thanks,
> Lina
>
>
> Lina Iyer (2):
>   PM / runtime: register device's next wakeup
>   PM / Domains: use device's next wakeup to determine domain idle state
>
>  drivers/base/power/domain_governor.c | 87 ++++++++++++++++++++++++++--
>  drivers/base/power/runtime.c         | 31 ++++++++++
>  include/linux/pm.h                   |  2 +
>  include/linux/pm_domain.h            |  1 +
>  include/linux/pm_runtime.h           |  1 +
>  5 files changed, 116 insertions(+), 6 deletions(-)
>
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>

Kind regards
Uffe

Re: [RFC PATCH 0/2] Better domain idle from device wakeup patterns

[Lina Iyer]

On Tue, Sep 08 2020 at 06:15 -0600, Ulf Hansson wrote:
>On Thu, 3 Sep 2020 at 01:26, Lina Iyer <ilina@codeaurora.org> wrote:
>>
>> Hello,
>>
>> I was looking for an option to do better power management for some
>> domains where the devices enter runtime PM in a predictable fashion. For
>> example a display device that sends a vsync interrupt every 16 ms for a
>> 60 Hz panel. These interrupts are not timer interrupts but tend to
>> interrupt periodically to service the workflow and the devices and
>> domains may go back to idle soon after. Two domains are affected by this
>> - the device's PM domain and the CPU PM domain.
>>
>> As a first step, I am looking to solve for the device's PM domain idle
>> state (and hopefully solve for the CPU PM domains subsequently). The PM
>> domain could have multiple idle states and/or the enter/exit latencies
>> could be high. In either case, it may not always be beneficial to power
>> off the domain, only to turn it back on before satisfying the idle state
>> residency. When the wakeup is known for the device, we could use that to
>> determine the worthiness of entering a domain idle state. Only the
>> device can tell us when the future event would be and that could change
>> as the usecase changes. Like, when the panel refresh rate increases to
>> 120 Hz. If this information was made available to runtime PM, we could
>> use that in the domain governor to determine a suitable idle state. This
>> is the idea behind these patches.
>
>While striving towards entering the most optimal (energy and
>performance wise) idle state, I think it's an interesting approach.
>
Thanks for taking time to review this.

>>
>> In the first patch, I am proposing an API for devices to specify their
>> wakeup as a time in the future and in the second patch, I am updating
>> the PM domain governor to use this information to determine the idle
>> state. I have not had a chance to test this out yet, but I wanted to
>> know if I am on the right track.
>
>I don't have any immediate objections - I think the approach seems
>reasonable. Still, my first thought that this could be an extension to
>the dev_pm_qos interface, but perhaps that isn't a good fit.
>
Yeah, I did look into that, but it did not the bill exactly either
starting with the locking requirements and how the next event could be
placed from an interrupt context.

>I also have a couple of comments to the code, but I will reply to each
>patch separately about that.
>
>>
>> Would appreciate your thoughts on this.
>
>When it comes to showing real results, other than in theory, I think

Well, the device's PM domain power saving is largely dependent on the
board design and the power saving or performance benefit may not be
easily quantifiable.

>we need to make the genpd's cpu governor to cope with the next wakeup
>event as well.
>
I was hoping to tie this with the genirq/timing framework as well for
the CPUs, but it is something I was thinking as a next step. The API and
the governor changes here lay the foundation for that as well. All we
need to do is hook up the predicted event to the CPU device and the CPU
domain governor already calls into the default_power_down_ok() which
takes care of the next event to effect a better governor decision.

--Lina

>>
>> Thanks,
>> Lina
>>
>>
>> Lina Iyer (2):
>>   PM / runtime: register device's next wakeup
>>   PM / Domains: use device's next wakeup to determine domain idle state
>>
>>  drivers/base/power/domain_governor.c | 87 ++++++++++++++++++++++++++--
>>  drivers/base/power/runtime.c         | 31 ++++++++++
>>  include/linux/pm.h                   |  2 +
>>  include/linux/pm_domain.h            |  1 +
>>  include/linux/pm_runtime.h           |  1 +
>>  5 files changed, 116 insertions(+), 6 deletions(-)
>>
>> --
>> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>
>
>Kind regards
>Uffe