[PATCH 0/2] RFC: CPU frequency max as PM QoS param

[PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

This is a continuation to "RFC: CPU frequency min as PM QoS param"
patchset. This patchset adds CPU frequency maximum as a PM QoS
parameter and modifies CPU frequncy core to enforce the limit. CPU
frequency ceiling can be used to improve the energy efficiency of
workloads that would cause the cpufreq governors to enforce an
unnecessarily high operating point. In other words, CPU frequency
maximum can act as an energy efficiency level request.

Tested on Dell E6420 with the ACPI cpufreq driver against Ubuntu
3.2. Patches are against linux-next, compile tested.

Antti P Miettinen (2):
  PM QoS: Add CPU frequency maximum as PM QoS param
  cpufreq: Enforce PM QoS maximum frequency

 drivers/cpufreq/cpufreq.c |   16 +++++++++++++---
 include/linux/pm_qos.h    |    2 ++
 kernel/power/qos.c        |   15 +++++++++++++++
 3 files changed, 30 insertions(+), 3 deletions(-)

-- 
1.7.4.1

[PATCH 1/2] PM QoS: Add CPU frequency maximum as PM QoS param

[Antti P Miettinen]

Add maximum CPU frequency as PM QoS parameter.

Signed-off-by: Antti P Miettinen <amiettinen@nvidia.com>
---
 include/linux/pm_qos.h |    2 ++
 kernel/power/qos.c     |   15 +++++++++++++++
 2 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/include/linux/pm_qos.h b/include/linux/pm_qos.h
index 54a0d00..7b8d08b 100644
--- a/include/linux/pm_qos.h
+++ b/include/linux/pm_qos.h
@@ -15,6 +15,7 @@ enum {
 	PM_QOS_NETWORK_LATENCY,
 	PM_QOS_NETWORK_THROUGHPUT,
 	PM_QOS_CPU_FREQ_MIN,
+	PM_QOS_CPU_FREQ_MAX,
 
 	/* insert new class ID */
 
@@ -28,6 +29,7 @@ enum {
 #define PM_QOS_NETWORK_THROUGHPUT_DEFAULT_VALUE	0
 #define PM_QOS_DEV_LAT_DEFAULT_VALUE		0
 #define PM_QOS_CPU_FREQ_MIN_DEFAULT_VALUE	0
+#define PM_QOS_CPU_FREQ_MAX_DEFAULT_VALUE	LONG_MAX
 
 struct pm_qos_request {
 	struct plist_node node;
diff --git a/kernel/power/qos.c b/kernel/power/qos.c
index 07d761a..04b744b 100644
--- a/kernel/power/qos.c
+++ b/kernel/power/qos.c
@@ -115,12 +115,27 @@ static struct pm_qos_object cpu_freq_min_pm_qos = {
 };
 
 
+static BLOCKING_NOTIFIER_HEAD(cpu_freq_max_notifier);
+static struct pm_qos_constraints cpu_freq_max_constraints = {
+	.list = PLIST_HEAD_INIT(cpu_freq_max_constraints.list),
+	.target_value = PM_QOS_CPU_FREQ_MAX_DEFAULT_VALUE,
+	.default_value = PM_QOS_CPU_FREQ_MAX_DEFAULT_VALUE,
+	.type = PM_QOS_MIN,
+	.notifiers = &cpu_freq_max_notifier,
+};
+static struct pm_qos_object cpu_freq_max_pm_qos = {
+	.constraints = &cpu_freq_max_constraints,
+	.name = "cpu_freq_max",
+};
+
+
 static struct pm_qos_object *pm_qos_array[] = {
 	&null_pm_qos,
 	&cpu_dma_pm_qos,
 	&network_lat_pm_qos,
 	&network_throughput_pm_qos,
 	&cpu_freq_min_pm_qos,
+	&cpu_freq_max_pm_qos,
 };
 
 static ssize_t pm_qos_power_write(struct file *filp, const char __user *buf,
-- 
1.7.4.1

[PATCH 2/2] cpufreq: Enforce PM QoS maximum frequency

[Antti P Miettinen]

Observe PM QoS CPU frequency maximum in addition to
policy settings.

Signed-off-by: Antti P Miettinen <amiettinen@nvidia.com>
---
 drivers/cpufreq/cpufreq.c |   16 +++++++++++++---
 1 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index c2c1c62..d233a8b 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -1638,12 +1638,15 @@ static int __cpufreq_set_policy(struct cpufreq_policy *data,
 	unsigned int pmax = policy->max;
 	unsigned int qmin = min(pm_qos_request(PM_QOS_CPU_FREQ_MIN),
 				data->max);
+	unsigned int qmax = max(pm_qos_request(PM_QOS_CPU_FREQ_MAX),
+				data->min);
 
-	pr_debug("setting new policy for CPU %u: %u/%u - %u kHz\n",
-		 policy->cpu, pmin, qmin, pmax);
+	pr_debug("setting new policy for CPU %u: %u/%u - %u/%u kHz\n",
+		 policy->cpu, pmin, qmin, pmax, qmax);
 
 	/* clamp the new policy to PM QoS limits */
 	policy->min = max(pmin, qmin);
+	policy->max = min(pmax, qmax);
 
 	memcpy(&policy->cpuinfo, &data->cpuinfo,
 				sizeof(struct cpufreq_cpuinfo));
@@ -1920,12 +1923,16 @@ static int cpu_freq_notify(struct notifier_block *b,
 static struct notifier_block min_freq_notifier = {
 	.notifier_call = cpu_freq_notify,
 };
+static struct notifier_block max_freq_notifier = {
+	.notifier_call = cpu_freq_notify,
+};
 
 static int cpu_freq_notify(struct notifier_block *b,
 			   unsigned long l, void *v)
 {
 	int cpu;
-	pr_debug("PM QoS min %lu\n", l);
+	pr_debug("PM QoS %s %lu\n",
+		 b == &min_freq_notifier ? "min" : "max", l);
 	for_each_online_cpu(cpu) {
 		struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
 		if (policy) {
@@ -1952,6 +1959,9 @@ static int __init cpufreq_core_init(void)
 	rc = pm_qos_add_notifier(PM_QOS_CPU_FREQ_MIN,
 				 &min_freq_notifier);
 	BUG_ON(rc);
+	rc = pm_qos_add_notifier(PM_QOS_CPU_FREQ_MAX,
+				 &max_freq_notifier);
+	BUG_ON(rc);
 
 	return 0;
 }
-- 
1.7.4.1

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Kevin Hilman]

Hello,

Antti P Miettinen <amiettinen@nvidia.com> writes:

> This is a continuation to "RFC: CPU frequency min as PM QoS param"
> patchset. This patchset adds CPU frequency maximum as a PM QoS
> parameter and modifies CPU frequncy core to enforce the limit. CPU
> frequency ceiling can be used to improve the energy efficiency of
> workloads that would cause the cpufreq governors to enforce an
> unnecessarily high operating point. In other words, CPU frequency
> maximum can act as an energy efficiency level request.
>
> Tested on Dell E6420 with the ACPI cpufreq driver against Ubuntu
> 3.2. Patches are against linux-next, compile tested.

I know there were some earlier discussions about the usefulness of a max
frequency QoS parameter, so I wanted to throw in a reason to include max
as well as min frequency parameters.

IMO, having a max frequency QoS parameter would be very useful from a
thermal perspective.  

There are some ongoing projects in the PM working group at Linaro that
are exploring plugins to the thermal framework that implment a "cooling
device" by capping CPU frequency.  Having a QoS parameter do do this
would be the logical interface.

I also agree with some earlier requests that these should probably be
per-CPU instead of global.  That would make it simple to cap frequency
of one cluster while leaving another alone.

Kevin

Re: [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[mark gross]

On Wed, Feb 15, 2012 at 05:06:40PM -0800, Kevin Hilman wrote:
> Hello,
> 
> Antti P Miettinen <amiettinen@nvidia.com> writes:
> 
> > This is a continuation to "RFC: CPU frequency min as PM QoS param"
> > patchset. This patchset adds CPU frequency maximum as a PM QoS
> > parameter and modifies CPU frequncy core to enforce the limit. CPU
> > frequency ceiling can be used to improve the energy efficiency of
> > workloads that would cause the cpufreq governors to enforce an
> > unnecessarily high operating point. In other words, CPU frequency
> > maximum can act as an energy efficiency level request.
> >
> > Tested on Dell E6420 with the ACPI cpufreq driver against Ubuntu
> > 3.2. Patches are against linux-next, compile tested.
> 
> I know there were some earlier discussions about the usefulness of a max
> frequency QoS parameter, so I wanted to throw in a reason to include max
> as well as min frequency parameters.
> 
> IMO, having a max frequency QoS parameter would be very useful from a
> thermal perspective.  
> 
> There are some ongoing projects in the PM working group at Linaro that
> are exploring plugins to the thermal framework that implment a "cooling
> device" by capping CPU frequency.  Having a QoS parameter do do this
> would be the logical interface.
> 
> I also agree with some earlier requests that these should probably be
> per-CPU instead of global.  That would make it simple to cap frequency
> of one cluster while leaving another alone.
>
Hi,

My problem is that overloading pm_qos with a performance limits is a
hack that will likely have problems coexisting with the core notion of
throttling limiting that is pm_qos as thermal limiting enabling expands
its scope.

Sure limiting performance governors seem to need the same notifications
and infrastructure as pm_qos but I worry that as its really doing the
opposite to qos I really don't see the logic of having it in pmqos.

Lets consider thermal throttling a bit maybe I can convince you to that
maybe we need some new mechanism for energy and thermal limiting.  (BTW
I agree something is needed for this).

For thermals the culprit's are:
cpu -- cpufreq governor currently not-thermal aware
Display -- brightness not thermal aware
GFX -- graphics driver not thermal aware
battery charging -- somewhat thermal aware but needs more than just bat
temp to count.
Flash -- led flash light  not thermal aware.

Other devices may have other heat sources of course.

Wouldn't it be better to have thermal awareness as part of the driver
model and then have a pm_qos like thing to do out of band signaling of
devices and user mode when a temperature is hit?

Putting it a qos class for limiting the high end of the performance that
a governor really doesn't go far enough IMO.  It looks like a hack just
for cpufreq that already has sysfs interfaces for limiting the top
frequency.

ok, maybe my argument is not so strong. but as long as its named pm QOS
and not pm-constraints I don't like the idea of having the max
parameter in the pm_qos code.  

It feature creeps the infrastructure and violates the spirit of what
pm_qos was to solve.  i.e. constrain the lower power states not upper
states.

But, for thermal / energy management I think we need more notification
of thermal or energy events than what is needed for the pm_qos problem.

For pm_qos its mostly a static problem that is tuned in the lab.  Know
the workload needs and block too much throttling.  For thermal / energy
management you isn't any tuning you can do.  You don't apply the
constraint until it gets too hot.  You don't know if or when you'll get
too hot.

Note: its more than just temperature that we need to worry about.  Peak
current is also a related issue (that is much more transient)

Shouldn't we do something more forward looking than hack pm_qos with
max-freq so that cpufreq can check a redundant parameter to what it
already has exported through sysfs?


--mark

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Valentin, Eduardo]

Hello,

On Fri, Feb 17, 2012 at 5:04 AM, mark gross <markgross@thegnar.org> wrote:
> On Wed, Feb 15, 2012 at 05:06:40PM -0800, Kevin Hilman wrote:
>> Hello,
>>
>> Antti P Miettinen <amiettinen@nvidia.com> writes:
>>
>> > This is a continuation to "RFC: CPU frequency min as PM QoS param"
>> > patchset. This patchset adds CPU frequency maximum as a PM QoS
>> > parameter and modifies CPU frequncy core to enforce the limit. CPU
>> > frequency ceiling can be used to improve the energy efficiency of
>> > workloads that would cause the cpufreq governors to enforce an
>> > unnecessarily high operating point. In other words, CPU frequency
>> > maximum can act as an energy efficiency level request.
>> >
>> > Tested on Dell E6420 with the ACPI cpufreq driver against Ubuntu
>> > 3.2. Patches are against linux-next, compile tested.
>>
>> I know there were some earlier discussions about the usefulness of a max
>> frequency QoS parameter, so I wanted to throw in a reason to include max
>> as well as min frequency parameters.
>>
>> IMO, having a max frequency QoS parameter would be very useful from a
>> thermal perspective.
>>
>> There are some ongoing projects in the PM working group at Linaro that
>> are exploring plugins to the thermal framework that implment a "cooling
>> device" by capping CPU frequency.  Having a QoS parameter do do this
>> would be the logical interface.

I saw the work started by Amit (Cced), but I believe we may need to expand it.

>>
>> I also agree with some earlier requests that these should probably be
>> per-CPU instead of global.  That would make it simple to cap frequency
>> of one cluster while leaving another alone.

I believe we need to consider a broader set of cases, not only cpu.

>>
> Hi,
>
> My problem is that overloading pm_qos with a performance limits is a
> hack that will likely have problems coexisting with the core notion of
> throttling limiting that is pm_qos as thermal limiting enabling expands
> its scope.
>
> Sure limiting performance governors seem to need the same notifications
> and infrastructure as pm_qos but I worry that as its really doing the
> opposite to qos I really don't see the logic of having it in pmqos.
>
> Lets consider thermal throttling a bit maybe I can convince you to that
> maybe we need some new mechanism for energy and thermal limiting.  (BTW
> I agree something is needed for this).
>
> For thermals the culprit's are:
> cpu -- cpufreq governor currently not-thermal aware
> Display -- brightness not thermal aware
> GFX -- graphics driver not thermal aware
> battery charging -- somewhat thermal aware but needs more than just bat
> temp to count.
> Flash -- led flash light  not thermal aware.


This is probably what we want to consider, at least to start with. As
Mark mentions above, the cpu domain is just part of the equation.


The work started by Amit is going towards the right direction, but I
believe we need to review the thermal problem for those systems that
don't have ACPI. The generic part of current thermal framework
(drivers/thermal/) is still somewhat bound to ACPI.

>
> Other devices may have other heat sources of course.

Indeed.

>
> Wouldn't it be better to have thermal awareness as part of the driver
> model and then have a pm_qos like thing to do out of band signaling of
> devices and user mode when a temperature is hit?
>


Having things exposed to drivers is maybe one we to go, but we need to
be careful and not leave the decision making to drivers, IMO. Might be
worth considering thermal governors as well, instead of having the
decision making spread all over driver code. The point I am trying to
make here is that sometimes you may want to consider correlation
between these thermal domains.


> Putting it a qos class for limiting the high end of the performance that
> a governor really doesn't go far enough IMO.  It looks like a hack just
> for cpufreq that already has sysfs interfaces for limiting the top
> frequency.
>
> ok, maybe my argument is not so strong. but as long as its named pm QOS
> and not pm-constraints I don't like the idea of having the max
> parameter in the pm_qos code.

I also agree that this type of problem goes more like a set of
constraint instead of trying to keep QoS.

>
> It feature creeps the infrastructure and violates the spirit of what
> pm_qos was to solve.  i.e. constrain the lower power states not upper
> states.
>
> But, for thermal / energy management I think we need more notification
> of thermal or energy events than what is needed for the pm_qos problem.
>
> For pm_qos its mostly a static problem that is tuned in the lab.  Know
> the workload needs and block too much throttling.  For thermal / energy
> management you isn't any tuning you can do.  You don't apply the
> constraint until it gets too hot.  You don't know if or when you'll get
> too hot.
>
> Note: its more than just temperature that we need to worry about.  Peak
> current is also a related issue (that is much more transient)
>
> Shouldn't we do something more forward looking than hack pm_qos with
> max-freq so that cpufreq can check a redundant parameter to what it
> already has exported through sysfs?


Agreed here, we need something to represent the problem in better way.
But, I believe what is currently at the sysfs for cpufreq is a
constraint set by user, and for thermal aspect, sometimes you can not
confuse what the user has set to what the system really needs. The
system may not want to wait for user to say you have to reduce
performance in order to try contain thermal, specially if the
situation is going critical, or as you mentioned, to consider peak
current management as well. So, I believe we need to consider both as
different constraints. Besides having these constraints coming from
different places and not having a good way to show what is going on on
the system (and also good ways to debug it) is something that may need
to be thought as well.


>
>
> --mark
>
> _______________________________________________
> linux-pm mailing list
> linux-pm@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/linux-pm


All best,

-- 

Eduardo Valentin

Re: [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

Thanks for the comments. I'd like to comment on maximum CPU frequency,
sysfs files and per device contraints..

Maximum CPU frequency could be useful for thermal. However, it is not a
complete solution for thermal. Just like minimum CPU frequency is not a
complete solution for computing throughput (e.g. memory and accelerator
control are not directly addressed by a CPU frequency
constraint). Maximum CPU frequency can be also useful for energy
efficiency even though the constraint is not a complete solution here
either. I guess latency constraints do not completely solve end-to-end
latency requirements but the mechanism is useful so it is good to have
it. I'd argue minimum and maximum frequency are simular in this respect.

There are sysfs files for constraining CPU frequency. However, there is
no arbitration for several applications trying to place constraints. PM
QoS provides a way to consolidate requests from several applications and
cleanup upon application crash. I think the existing sysfs files are not
an appropriate inferface for user space applications.

Currently CPU sleep states are blocked globally for latency
contraints. Finer granularity control would be possible with per CPU
contraints. However - are there clients that know or want to contrain a
specific CPU? Same question is applicable also to CPU frequency. Even
though per CPU control is more flexible, what are the clients that want
to constrain a specific CPU?

Another complication for the per device constraints is the user space
interface. Dynamic minors run out pretty fast if we have per CPU
parameters and the system has huge number of CPUs. Does anyone have any
opinions about the user space interface for device PM QoS?

	--Antti

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Pihet-XID, Jean]

Re-adding cpufreq and linux-pm MLs in Cc.

Regards,
Jean

On Sat, Feb 25, 2012 at 6:44 PM, mark gross <markgross@thegnar.org> wrote:
> On Tue, Feb 21, 2012 at 10:00:18AM -0800, Kevin Hilman wrote:
>> mark gross <markgross@thegnar.org> writes:
>>
>> > On Mon, Feb 20, 2012 at 05:44:23PM +0200, Valentin, Eduardo wrote:
>> >> Hello Antti,
>> >>
>> >> On Mon, Feb 20, 2012 at 12:00 PM, Antti P Miettinen
>> >> <amiettinen@nvidia.com> wrote:
>> >> > The following message is a courtesy copy of an article
>> >> > that has been posted to gmane.linux.kernel.cpufreq,gmane.linux.power-management.general as well.
>> >> >
>> >> > Thanks for the comments. I'd like to comment on maximum CPU frequency,
>> >> > sysfs files and per device contraints..
>> >> >
>> >> > Maximum CPU frequency could be useful for thermal. However, it is not a
>> >> > complete solution for thermal. Just like minimum CPU frequency is not a
>> >> > complete solution for computing throughput (e.g. memory and accelerator
>> >> > control are not directly addressed by a CPU frequency
>> >> > constraint). Maximum CPU frequency can be also useful for energy
>> >> > efficiency even though the constraint is not a complete solution here
>> >> > either. I guess latency constraints do not completely solve end-to-end
>> >> > latency requirements but the mechanism is useful so it is good to have
>> >> > it. I'd argue minimum and maximum frequency are simular in this respect.
>> >> >
>> >>
>> >> I believe we are aligned that this is not complete solution, but the
>> >> constraints does help on both cases.
>> >>
>> >> > There are sysfs files for constraining CPU frequency. However, there is
>> >> > no arbitration for several applications trying to place constraints. PM
>> >> > QoS provides a way to consolidate requests from several applications and
>> >> > cleanup upon application crash. I think the existing sysfs files are not
>> >> > an appropriate inferface for user space applications.
>> >>
>> >> Agreed, the current CPU frequency interfaces were not design to
>> >> achieve lists of constraints at all, but they had very simplistic
>> >> design which you would consider the last constraint applied.
>> >>
>> >> But the point Mark was trying to bring was that PM QoS was not really
>> >> meant for max- like constraints. As he said, that is not for QoS but
>> >> for constraints. We may be abusing a bit the FW by addressing the
>> >> problem with it. That would look like more as a pm-constraint FW.
>> >>
>> >> It just happens to be so that the PM QoS has already the needed infrastructure.
>> >>
>> >
>> > Aside from the naming and design intents of pm_qos (qos vrs perforce
>> > constraints) which I acknowledge is arguably not a good reason on its
>> > own to block this, the other issue I have is that thermal constraints
>> > are much more dynamic than pm-qos constraints and I don't think the
>> > existing framework is up to it.
>> >
>> > almost all applications of pm-qos are static.
>>
>> Not true.
>>
>> With the new per-device PM QoS, we are adding/removing constraints
>> dynamically depending on usage.  Drivers/subsystems add/remove
>> constraints depending on what is happening and their needs.  This can
>> happen dynamically and very often.
>
> I have to admit I no experience and limited  insight into the per-device
> pm-qos or how its getting used.  However; I think there is a difference
> between dynamic use cases and ambient environmental effects on the
> system.
>
>> Also, as you know based on changes submitted to PM QoS infrastructure
>> after it's initial addition, lots of work was done so that it can be
>> used in the fast path, so many people care about the ability to change
>> constraints dynamically.
>
> true.
>
>> > Some device has a lower bound of performance needed by some other
>> > platform component and uses pm-qos to request tat the lower bound not
>> > be violated.
>>
>> And with per-device constraints (I prefer the term constraints too),
>> this can be very dynamic, and very targetted.  e.g. the constraint might
>> apply to a single power domain/island and can can be added/removed
>> dynamically and often.
>>
>> > Performance constraints depend on dynamic values (temperature, peak
>> > current, perhaps someday even noise) that are effected by ambient
>> > conditions.
>>
>> And current QoS settings also depend on dynamic values: instantaneous
>> throughput, frequency dependencies between IP blocks, etc. etc.
>> Depending on which devices/subsystems are in use by a given usecase,
>> these are all dynamic conditions.
>
> They are static per use case.  Just because the use case can change
> dynamically does not make it dynamic in the same sense as thermal or
> battery power envelopes.
>
> Further the effects of truly dynamic conditions is significant to the
> user.
>
>> > I think adding perf-constraints to the pm-qos without taking the time to
>> > think through these things to be a quick and dirty hack.
>>
>> I don't see it that way, and to me it's a very logical extention.
>>
>
> I see it as a convenient extension that is very close to being a quick
> hack.  That only solves a small part of a more general problem.
>
>> Current QoS settings could be thought of as performance constraints
>> too.  It's just that they determine minimum performance.  Adding
>> constraints for maxium performance is not a big stretch in my mind.
>
> Its not a big stretch to me either.  I just think its a bit of a hack
> and there is a bigger more interesting issue getting overlooked.
>
> Lastly why not simply make cpufreq thermal aware and talk directly to
> it if you even need too?
>
> --mark
>
>> Kevin
>>
>> > I don't know
>> > if we should be quick and dirty with main line changes like this.  maybe
>> > pm-qos only needs some dynamic constrain aggregation (for ambiant
>> > conditions) and a few new constraint classes.  Maybe there needs to be
>> > more done.
>> >
>> > maybe I'm over thinking it and we should simply add the perf-constraint
>> > class.
>>
>> >>
>> >>
>> >> >
>> >> > Currently CPU sleep states are blocked globally for latency
>> >> > contraints. Finer granularity control would be possible with per CPU
>> >> > contraints. However - are there clients that know or want to contrain a
>> >> > specific CPU? Same question is applicable also to CPU frequency. Even
>> >> > though per CPU control is more flexible, what are the clients that want
>> >> > to constrain a specific CPU?
>> >>
>> >> If I got your point right, I agree with you, we definitely need a way
>> >> to expose who is setting the constraint.
>> >>
>> >> >
>> >> > Another complication for the per device constraints is the user space
>> >> > interface. Dynamic minors run out pretty fast if we have per CPU
>> >> > parameters and the system has huge number of CPUs. Does anyone have any
>> >> > opinions about the user space interface for device PM QoS?
>> >> >
>> >> >        --Antti
>> >>
>> >>
>> >> All best,
>> >>
>> >> --
>> >>
>> >> Eduardo Valentin

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

"Pihet-XID, Jean" <j-pihet@ti.com> writes:
> Re-adding cpufreq and linux-pm MLs in Cc.
>
> Regards,
> Jean

Thanks Jean. And sorry for causing the lists to get dropped.

Does anyone have comments about the user space interface for per device
constraints:

>>>>>> Another complication for the per device constraints is the user space
>>>>>> interface. Dynamic minors run out pretty fast if we have per CPU
>>>>>> parameters and the system has huge number of CPUs. Does anyone have any
>>>>>> opinions about the user space interface for device PM QoS?

My current view is that CPU frequency constraints could be global just
like the CPU latency constraints are.

	--Antti

Re: [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti Miettinen]

To the lists too..

On 02/27/2012 04:49 PM, Kevin Hilman wrote:
> mark gross <markgross@thegnar.org> writes:
>
>  >> Current QoS settings could be thought of as performance constraints
>  >> too. It's just that they determine minimum performance. Adding
>  >> constraints for maxium performance is not a big stretch in my mind.
>  >
>  > Its not a big stretch to me either. I just think its a bit of a hack
>  > and there is a bigger more interesting issue getting overlooked.
>  >
>  > Lastly why not simply make cpufreq thermal aware and talk directly to
>  > it if you even need too?
>
> In fact, making a thermal framework "cooling device" that talks directly
> to CPUfreq is already what's being done by the Linaro PMWG folks.
>
> The problem is that CPUfreq only controls the CPU frequency.
>
> There are other devices that could be scaled back to reduce heat as well
> (DSP, and especially GPU), so having a more generic per-device
> constraint interface that can cap the frequency for *any* scalable
> device is a better framework IMO.
>
> It just so happens that pm_qos is already a good per-device constraint
> framework and can easily modified to cap performance as well as request
> a minimum performance.
>
> Kevin
>

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[mark gross]

On Mon, Feb 27, 2012 at 04:04:00PM +0100, Antti Miettinen wrote:
> To the lists too..
> 
> On 02/27/2012 04:49 PM, Kevin Hilman wrote:
> > mark gross <markgross@thegnar.org> writes:
> >
> >  >> Current QoS settings could be thought of as performance constraints
> >  >> too. It's just that they determine minimum performance. Adding
> >  >> constraints for maxium performance is not a big stretch in my mind.
> >  >
> >  > Its not a big stretch to me either. I just think its a bit of a hack
> >  > and there is a bigger more interesting issue getting overlooked.
> >  >
> >  > Lastly why not simply make cpufreq thermal aware and talk directly to
> >  > it if you even need too?
> >
> > In fact, making a thermal framework "cooling device" that talks directly
> > to CPUfreq is already what's being done by the Linaro PMWG folks.
> >
> > The problem is that CPUfreq only controls the CPU frequency.
> >
> > There are other devices that could be scaled back to reduce heat as well
> > (DSP, and especially GPU), so having a more generic per-device
> > constraint interface that can cap the frequency for *any* scalable
> > device is a better framework IMO.
> >
> > It just so happens that pm_qos is already a good per-device constraint
> > framework and can easily modified to cap performance as well as request
> > a minimum performance.
> >
> > Kevin

ok I'll stop trying to block it.

I want to re-do the whole works anyway.  If this helps in the mean time
then go for it.

--mark

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

Adding people that were part of the thread in the beginning..

mark gross <markgross@thegnar.org> writes:
> On Mon, Feb 27, 2012 at 04:04:00PM +0100, Antti Miettinen wrote:
>> To the lists too..
>> 
>> On 02/27/2012 04:49 PM, Kevin Hilman wrote:
>> > mark gross <markgross@thegnar.org> writes:
>> >
>> >  >> Current QoS settings could be thought of as performance constraints
>> >  >> too. It's just that they determine minimum performance. Adding
>> >  >> constraints for maxium performance is not a big stretch in my mind.
>> >  >
>> >  > Its not a big stretch to me either. I just think its a bit of a hack
>> >  > and there is a bigger more interesting issue getting overlooked.
>> >  >
>> >  > Lastly why not simply make cpufreq thermal aware and talk directly to
>> >  > it if you even need too?
>> >
>> > In fact, making a thermal framework "cooling device" that talks directly
>> > to CPUfreq is already what's being done by the Linaro PMWG folks.
>> >
>> > The problem is that CPUfreq only controls the CPU frequency.
>> >
>> > There are other devices that could be scaled back to reduce heat as well
>> > (DSP, and especially GPU), so having a more generic per-device
>> > constraint interface that can cap the frequency for *any* scalable
>> > device is a better framework IMO.
>> >
>> > It just so happens that pm_qos is already a good per-device constraint
>> > framework and can easily modified to cap performance as well as request
>> > a minimum performance.
>> >
>> > Kevin
>
> ok I'll stop trying to block it.
>
> I want to re-do the whole works anyway.  If this helps in the mean time
> then go for it.

Great :-)

So what do other people think? Could we merge global CPU frequency
constraints for now?

I agree that more work is needed for e.g. per CPU constraints, user
space interface and more complete thermal management. Actually for
future I think the constraints could also become more general than just
min/max "reduction operators". For e.g. core online status you might
want union/intersection of bitmaps. Also, the more complete thermal
management is related to load management in general (power budgeting for
other reasons than just thermal).

	--Antti

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Rafael J. Wysocki]

On Tuesday, February 28, 2012, Antti P Miettinen wrote:
> Adding people that were part of the thread in the beginning..
> 
> mark gross <markgross@thegnar.org> writes:
> > On Mon, Feb 27, 2012 at 04:04:00PM +0100, Antti Miettinen wrote:
> >> To the lists too..
> >> 
> >> On 02/27/2012 04:49 PM, Kevin Hilman wrote:
> >> > mark gross <markgross@thegnar.org> writes:
> >> >
> >> >  >> Current QoS settings could be thought of as performance constraints
> >> >  >> too. It's just that they determine minimum performance. Adding
> >> >  >> constraints for maxium performance is not a big stretch in my mind.
> >> >  >
> >> >  > Its not a big stretch to me either. I just think its a bit of a hack
> >> >  > and there is a bigger more interesting issue getting overlooked.
> >> >  >
> >> >  > Lastly why not simply make cpufreq thermal aware and talk directly to
> >> >  > it if you even need too?
> >> >
> >> > In fact, making a thermal framework "cooling device" that talks directly
> >> > to CPUfreq is already what's being done by the Linaro PMWG folks.
> >> >
> >> > The problem is that CPUfreq only controls the CPU frequency.
> >> >
> >> > There are other devices that could be scaled back to reduce heat as well
> >> > (DSP, and especially GPU), so having a more generic per-device
> >> > constraint interface that can cap the frequency for *any* scalable
> >> > device is a better framework IMO.
> >> >
> >> > It just so happens that pm_qos is already a good per-device constraint
> >> > framework and can easily modified to cap performance as well as request
> >> > a minimum performance.
> >> >
> >> > Kevin
> >
> > ok I'll stop trying to block it.
> >
> > I want to re-do the whole works anyway.  If this helps in the mean time
> > then go for it.
> 
> Great :-)
> 
> So what do other people think? Could we merge global CPU frequency
> constraints for now?

Not without an ACK from Dave (the cpufreq maintainer), that's for sure.

> I agree that more work is needed for e.g. per CPU constraints, user
> space interface and more complete thermal management. Actually for
> future I think the constraints could also become more general than just
> min/max "reduction operators". For e.g. core online status you might
> want union/intersection of bitmaps. Also, the more complete thermal
> management is related to load management in general (power budgeting for
> other reasons than just thermal).

Then perhaps let's not merge "temporary" stuff and figure out how to
implement what we _really_ want.

Thanks,
Rafael

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

"Rafael J. Wysocki" <rjw@sisk.pl> writes:
> On Tuesday, February 28, 2012, Antti P Miettinen wrote:
[..]
>> So what do other people think? Could we merge global CPU frequency
>> constraints for now?
>
> Not without an ACK from Dave (the cpufreq maintainer), that's for sure.

Dave - any comments about these?

http://thread.gmane.org/gmane.linux.kernel.cpufreq/7794
http://thread.gmane.org/gmane.linux.kernel.cpufreq/7797
http://thread.gmane.org/gmane.linux.kernel.cpufreq/7800

>> I agree that more work is needed for e.g. per CPU constraints, user
>> space interface and more complete thermal management. Actually for
>> future I think the constraints could also become more general than just
>> min/max "reduction operators". For e.g. core online status you might
>> want union/intersection of bitmaps. Also, the more complete thermal
>> management is related to load management in general (power budgeting for
>> other reasons than just thermal).
>
> Then perhaps let's not merge "temporary" stuff and figure out how to
> implement what we _really_ want.

Well, I'd say "partial" instead of "temporary". I think frequency min
and max are really needed but as discussed, they are hardly a complete
solution for power management.

	--Antti

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Dave Jones]

On Tue, Mar 06, 2012 at 02:23:52PM +0200, Antti P Miettinen wrote:
 > "Rafael J. Wysocki" <rjw@sisk.pl> writes:
 > > On Tuesday, February 28, 2012, Antti P Miettinen wrote:
 > [..]
 > >> So what do other people think? Could we merge global CPU frequency
 > >> constraints for now?
 > >
 > > Not without an ACK from Dave (the cpufreq maintainer), that's for sure.
 > 
 > Dave - any comments about these?
 > 
 > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7794
 > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7797
 > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7800

I really dislike how this is exposed to userspace.
How is a user to know whether scaling_max_freq or cpu_freq_max takes priority ?
Given the confusion we already have from users when the bios_limit enforces limits,
giving them two knobs to do the same thing seems like a bad idea to me.

I don't see what problem this is solving that you couldn't solve just by
setting scaling_max_freq.

	Dave

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

Dave Jones <davej@redhat.com> writes:
> On Tue, Mar 06, 2012 at 02:23:52PM +0200, Antti P Miettinen wrote:
[..]
>  > Dave - any comments about these?
>  > 
>  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7794
>  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7797
>  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7800
>
> I really dislike how this is exposed to userspace.
> How is a user to know whether scaling_max_freq or cpu_freq_max takes
> priority ? Given the confusion we already have from users when the
> bios_limit enforces limits, giving them two knobs to do the same thing
> seems like a bad idea to me.
>
> I don't see what problem this is solving that you couldn't solve just by
> setting scaling_max_freq.
>
> 	Dave

PM QoS handles multiple clients - the sysfs files are like global
variables: there is no arbitration/consolidation for multiple
clients. The sysfs files are a sort of override for system administrator
whereas the PM QoS is the interface applications should use.

	--Antti

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Dave Jones]

On Wed, Mar 07, 2012 at 08:38:40AM +0200, Antti P Miettinen wrote:
 > Dave Jones <davej@redhat.com> writes:
 > > On Tue, Mar 06, 2012 at 02:23:52PM +0200, Antti P Miettinen wrote:
 > [..]
 > >  > Dave - any comments about these?
 > >  > 
 > >  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7794
 > >  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7797
 > >  > http://thread.gmane.org/gmane.linux.kernel.cpufreq/7800
 > >
 > > I really dislike how this is exposed to userspace.
 > > How is a user to know whether scaling_max_freq or cpu_freq_max takes
 > > priority ? Given the confusion we already have from users when the
 > > bios_limit enforces limits, giving them two knobs to do the same thing
 > > seems like a bad idea to me.
 > >
 > > I don't see what problem this is solving that you couldn't solve just by
 > > setting scaling_max_freq.
 > 
 > PM QoS handles multiple clients - the sysfs files are like global
 > variables: there is no arbitration/consolidation for multiple
 > clients. The sysfs files are a sort of override for system administrator
 > whereas the PM QoS is the interface applications should use.

I think exposing absolute frequencies to applications is a mistake.
(And one that the core cpufreq made a long time ago). How is an application
to decide what to set it to without knowledge of the hardware it's running on ?

I much prefer the idea that was mentioned a few weeks ago during the
discussion with Peter Zijlstra about cpufreq being more connected to
the scheduler, and essentially having per-process governors.

Each process gets a /proc/self/power-policy
This can be 'performance' 'power-save' or 'ondemand'
 - A global sysfs knob sets the default new processes get.
 - Processes can adjust it themselves if desired.
 - There's no need for a system-wide governor any more.

There are some open questions about how this could work.

- A list of rules for desired behaviour when performing state changes
  when switching between tasks with different policies is needed.

- We don't want to be doing power transitions every context switch,
  or switching overhead will be brutal.
  So some kind of lazy state changing may be necessary.

- For 'ondemand', when would the scheduler decide to ramp up/down
  the speed ?

	Dave

Re: [linux-pm] [PATCH 0/2] RFC: CPU frequency max as PM QoS param

[Antti P Miettinen]

Dave Jones <davej@redhat.com> writes:
> I think exposing absolute frequencies to applications is a mistake.
> (And one that the core cpufreq made a long time ago). How is an application
> to decide what to set it to without knowledge of the hardware it's
> running on ?

Abstracting the CPU frequency was briefly discussed
previously. Computing performance is affected by many factors, not just
CPU frequency. My view is that the actual frequency values for a given
use case are likely to be very system specific (CPU, memory etc). The
values would probably come from a tuning excercise and would be
configurable parameters of applications.

> I much prefer the idea that was mentioned a few weeks ago during the
> discussion with Peter Zijlstra about cpufreq being more connected to
> the scheduler, and essentially having per-process governors.
>
> Each process gets a /proc/self/power-policy
> This can be 'performance' 'power-save' or 'ondemand'
>  - A global sysfs knob sets the default new processes get.
>  - Processes can adjust it themselves if desired.
>  - There's no need for a system-wide governor any more.
>
> There are some open questions about how this could work.
>
> - A list of rules for desired behaviour when performing state changes
>   when switching between tasks with different policies is needed.
>
> - We don't want to be doing power transitions every context switch,
>   or switching overhead will be brutal.
>   So some kind of lazy state changing may be necessary.
>
> - For 'ondemand', when would the scheduler decide to ramp up/down
>   the speed ?
>
> 	Dave

I think better power/perf requests from applications to kernel are
indeed sorely needed. However, if we are going to construct application
oriented power/performance requests, I think something else than
performance/powersave/ondemand might be more appropriate. For a fixed
hardware platform, an application might well be tuned to know very well
the minimum and maximum levels of CPU frequency that it needs in certain
situation in order to meet the required responsiveness and/or consume
minimum amount of energy. I think performance vs power-save is
unnecessarily coarse setting.

The big problem I see with application oriented metrics is that there
are so many of them. I'd rather keep this kind of complexity outside the
kernel. To me it feels easier to start from the hardware knobs that we
can control, since - in the end - those are the knobs that the kernel
needs to control. And we need consolidation of requests from multiple
applications, but to me exposing hardware oriented parameters towards
the user space is not a problem as such.

	--Antti