[PATCH 0/4] Add sustainable OPP concept

[PATCH 0/4] Add sustainable OPP concept

[Lukasz Luba]

Hi all,

This patch set introduces a concept of sustainable OPP, which then can be used
by kernel frameworks or governors for estimating system sustainable system
state. This kind of estimation is done e.g. in thermal governor Intelligent
Power Allocation (IPA), which calculates sustainable power of the whole system
and then derives some coefficients for internal algorithm.

The patch set introduces a new DT bindings 'opp-sustainable', with parsing
code. It also adds a function (in patch 3/4) which allows device drivers to set
directly the sustainable OPP. This is helpful when the device drivers populate
the OPP table by themself (example in patch 4/4).

The patch set should apply on top of next-20201028.

Regards,
Lukasz Luba

Lukasz Luba (4):
  dt-bindings: opp: Introduce opp-sustainable bindings
  OPP: Add support for parsing the 'opp-sustainable' property
  OPP: Add dev_pm_opp_set_sustainable_opp_freq()
  firmware: arm_scmi/perf: Mark sustainable OPP

 Documentation/devicetree/bindings/opp/opp.txt |  4 +
 drivers/firmware/arm_scmi/perf.c              | 10 +++
 drivers/opp/core.c                            | 75 +++++++++++++++++++
 drivers/opp/of.c                              | 14 ++++
 drivers/opp/opp.h                             |  3 +
 include/linux/pm_opp.h                        | 14 ++++
 6 files changed, 120 insertions(+)

-- 
2.17.1

[PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Lukasz Luba]

Add opp-sustainable as an additional property in the OPP node to describe
the sustainable performance level of the device. This will help to
estimate the sustainable performance of the whole system.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
index 9847dfeeffcb..cd01028de305 100644
--- a/Documentation/devicetree/bindings/opp/opp.txt
+++ b/Documentation/devicetree/bindings/opp/opp.txt
@@ -154,6 +154,10 @@ Optional properties:
 - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
   in the table have this, the OPP with highest opp-hz will be used.
 
+- opp-sustainable: Marks the OPP as sustainable. This property can be used for
+  estimating sustainable performance of the whole system. If multiple OPPs in
+  the table have this, the OPP with highest opp-hz will be used.
+
 - opp-supported-hw: This property allows a platform to enable only a subset of
   the OPPs from the larger set present in the OPP table, based on the current
   version of the hardware (already known to the operating system).
-- 
2.17.1

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Nishanth Menon]

On 14:08-20201028, Lukasz Luba wrote:
> Add opp-sustainable as an additional property in the OPP node to describe
> the sustainable performance level of the device. This will help to
> estimate the sustainable performance of the whole system.
> 
> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> ---
>  Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
> index 9847dfeeffcb..cd01028de305 100644
> --- a/Documentation/devicetree/bindings/opp/opp.txt
> +++ b/Documentation/devicetree/bindings/opp/opp.txt
> @@ -154,6 +154,10 @@ Optional properties:
>  - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>    in the table have this, the OPP with highest opp-hz will be used.
>  
> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
> +  estimating sustainable performance of the whole system. If multiple OPPs in
> +  the table have this, the OPP with highest opp-hz will be used.


By "sustainable", do you mean sustainable across Process, Voltage and
Temperature corners upto the max rated operational Power-ON hours
without IDLE state being achieved on the processor?

OR do you mean to leave it up to interpretation?
> +
>  - opp-supported-hw: This property allows a platform to enable only a subset of
>    the OPPs from the larger set present in the OPP table, based on the current
>    version of the hardware (already known to the operating system).
> -- 
> 2.17.1
> 

-- 
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3  1A34 DDB5 849D 1736 249D

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Lukasz Luba]

On 10/28/20 9:47 PM, Nishanth Menon wrote:
> On 14:08-20201028, Lukasz Luba wrote:
>> Add opp-sustainable as an additional property in the OPP node to describe
>> the sustainable performance level of the device. This will help to
>> estimate the sustainable performance of the whole system.
>>
>> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
>> ---
>>   Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
>> index 9847dfeeffcb..cd01028de305 100644
>> --- a/Documentation/devicetree/bindings/opp/opp.txt
>> +++ b/Documentation/devicetree/bindings/opp/opp.txt
>> @@ -154,6 +154,10 @@ Optional properties:
>>   - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>>     in the table have this, the OPP with highest opp-hz will be used.
>>   
>> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
>> +  estimating sustainable performance of the whole system. If multiple OPPs in
>> +  the table have this, the OPP with highest opp-hz will be used.
> 
> 
> By "sustainable", do you mean sustainable across Process, Voltage and
> Temperature corners upto the max rated operational Power-ON hours
> without IDLE state being achieved on the processor?

Yes, in case of CPU: running 100% without idle at that particular OPP.
Running above that OPP would lead to cross control temperature.

> 
> OR do you mean to leave it up to interpretation?

I can tell how I would use them. There is thermal governor IPA, which
needs sustainable power either form DT or uses internal algorithm to
estimate it based on lowest allowed freq OPPs. Then it estimated
internal coefficients based on that value, which is not optimal
for lowest OPPs. When some higher OPP could be marked as sustainable,
it would lead to better estimation and better power budget split.

Regards,
Lukasz

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Nishanth Menon]

On 10:04-20201029, Lukasz Luba wrote:
> 
> 
> On 10/28/20 9:47 PM, Nishanth Menon wrote:
> > On 14:08-20201028, Lukasz Luba wrote:
> > > Add opp-sustainable as an additional property in the OPP node to describe
> > > the sustainable performance level of the device. This will help to
> > > estimate the sustainable performance of the whole system.
> > > 
> > > Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> > > ---
> > >   Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
> > >   1 file changed, 4 insertions(+)
> > > 
> > > diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
> > > index 9847dfeeffcb..cd01028de305 100644
> > > --- a/Documentation/devicetree/bindings/opp/opp.txt
> > > +++ b/Documentation/devicetree/bindings/opp/opp.txt
> > > @@ -154,6 +154,10 @@ Optional properties:
> > >   - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
> > >     in the table have this, the OPP with highest opp-hz will be used.
> > > +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
> > > +  estimating sustainable performance of the whole system. If multiple OPPs in
> > > +  the table have this, the OPP with highest opp-hz will be used.
> > 
> > 
> > By "sustainable", do you mean sustainable across Process, Voltage and
> > Temperature corners upto the max rated operational Power-ON hours
> > without IDLE state being achieved on the processor?
> 
> Yes, in case of CPU: running 100% without idle at that particular OPP.
> Running above that OPP would lead to cross control temperature.

We need to tighten the definitions a lot more here and add that to the
binding. What we are stating, if I am not misunderstanding is an OPP
that is guaranteed by SoC vendor that across Process Voltage and
Temperature corners - aka across the entire production spectrum
for the part number, *all* devices will operate at this OPP for the
mandated power-on-hours rating without hitting IDLE.

Example: So -40C to 125C, across the process (hot/cold/nominal), 100s of
thousands/millions of units can operate upto 125,0000 power-on-hours
while running a tight deadloop OR maybe high processing function or even
cpuburn[1]?


Can you give me one SoC vendor and part that guarantees this? I am
wondering if this is all theoretical... There are tons of parameters
that come into play for "reliability" "sustainability" etc. Those are
tricky terminology that typically makes legal folks pretty happy to
debate for decades..

just my 2 cents.
> 
> > 
> > OR do you mean to leave it up to interpretation?
> 
> I can tell how I would use them. There is thermal governor IPA, which
> needs sustainable power either form DT or uses internal algorithm to
> estimate it based on lowest allowed freq OPPs. Then it estimated
> internal coefficients based on that value, which is not optimal
> for lowest OPPs. When some higher OPP could be marked as sustainable,
> it would lead to better estimation and better power budget split.

Seeing your series, I got an idea about how you plan on using it, I
just think we need to be more precise in our definition..

[1] https://patrickmn.com/projects/cpuburn/
-- 
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3  1A34 DDB5 849D 1736 249D

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Lukasz Luba]

On 10/29/20 12:59 PM, Nishanth Menon wrote:
> On 10:04-20201029, Lukasz Luba wrote:
>>
>>
>> On 10/28/20 9:47 PM, Nishanth Menon wrote:
>>> On 14:08-20201028, Lukasz Luba wrote:
>>>> Add opp-sustainable as an additional property in the OPP node to describe
>>>> the sustainable performance level of the device. This will help to
>>>> estimate the sustainable performance of the whole system.
>>>>
>>>> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
>>>> ---
>>>>    Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>>>>    1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
>>>> index 9847dfeeffcb..cd01028de305 100644
>>>> --- a/Documentation/devicetree/bindings/opp/opp.txt
>>>> +++ b/Documentation/devicetree/bindings/opp/opp.txt
>>>> @@ -154,6 +154,10 @@ Optional properties:
>>>>    - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>>>>      in the table have this, the OPP with highest opp-hz will be used.
>>>> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
>>>> +  estimating sustainable performance of the whole system. If multiple OPPs in
>>>> +  the table have this, the OPP with highest opp-hz will be used.
>>>
>>>
>>> By "sustainable", do you mean sustainable across Process, Voltage and
>>> Temperature corners upto the max rated operational Power-ON hours
>>> without IDLE state being achieved on the processor?
>>
>> Yes, in case of CPU: running 100% without idle at that particular OPP.
>> Running above that OPP would lead to cross control temperature.
> 
> We need to tighten the definitions a lot more here and add that to the
> binding. What we are stating, if I am not misunderstanding is an OPP
> that is guaranteed by SoC vendor that across Process Voltage and
> Temperature corners - aka across the entire production spectrum
> for the part number, *all* devices will operate at this OPP for the
> mandated power-on-hours rating without hitting IDLE.
> 
> Example: So -40C to 125C, across the process (hot/cold/nominal), 100s of
> thousands/millions of units can operate upto 125,0000 power-on-hours
> while running a tight deadloop OR maybe high processing function or even
> cpuburn[1]?

I think I know what you mean. But this would lead to redefining a lot
more that just this optional field. This wide range -40C to 125C is for
automotive chips, then what about opp-suspend, when the device cannot
even reach that OPP under some stress test e.g. outside temp
~100-110C...
Or opp-turbo, shell all the OPPs have multidimensional table to reflect
the temperature dependency for all affected optional fields?

> 
> 
> Can you give me one SoC vendor and part that guarantees this? I am
> wondering if this is all theoretical... There are tons of parameters
> that come into play for "reliability" "sustainability" etc. Those are
> tricky terminology that typically makes legal folks pretty happy to
> debate for decades..

Yes, but the outside temperature is probably most important for this use
case.

> 
> just my 2 cents.
>>
>>>
>>> OR do you mean to leave it up to interpretation?
>>
>> I can tell how I would use them. There is thermal governor IPA, which
>> needs sustainable power either form DT or uses internal algorithm to
>> estimate it based on lowest allowed freq OPPs. Then it estimated
>> internal coefficients based on that value, which is not optimal
>> for lowest OPPs. When some higher OPP could be marked as sustainable,
>> it would lead to better estimation and better power budget split.
> 
> Seeing your series, I got an idea about how you plan on using it, I
> just think we need to be more precise in our definition..

Thank you for having a look on that and understanding the motivation
behind this series.

How about adding a description that this sustainable OPP is considered
for normal room temp (20-25C)?

BTW, in the Arm SCMI spec definition of that value (used in patch 4/4),
there is no specific temperature for it, just:
'This is the maximum performance level that the platform can
sustain under normal conditions. In exceptional circumstances,
such as thermal runaway, the platform might not be be able to
guarantee this level.'

I can put this whole description into the DT binding, if you like.

> 
> [1] https://patrickmn.com/projects/cpuburn/
> 

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Nishanth Menon]

On 13:33-20201029, Lukasz Luba wrote:
> 
> 
> On 10/29/20 12:59 PM, Nishanth Menon wrote:
> > On 10:04-20201029, Lukasz Luba wrote:
> > > 
> > > 
> > > On 10/28/20 9:47 PM, Nishanth Menon wrote:
> > > > On 14:08-20201028, Lukasz Luba wrote:
> > > > > Add opp-sustainable as an additional property in the OPP node to describe
> > > > > the sustainable performance level of the device. This will help to
> > > > > estimate the sustainable performance of the whole system.
> > > > > 
> > > > > Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> > > > > ---
> > > > >    Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
> > > > >    1 file changed, 4 insertions(+)
> > > > > 
> > > > > diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
> > > > > index 9847dfeeffcb..cd01028de305 100644
> > > > > --- a/Documentation/devicetree/bindings/opp/opp.txt
> > > > > +++ b/Documentation/devicetree/bindings/opp/opp.txt
> > > > > @@ -154,6 +154,10 @@ Optional properties:
> > > > >    - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
> > > > >      in the table have this, the OPP with highest opp-hz will be used.
> > > > > +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
> > > > > +  estimating sustainable performance of the whole system. If multiple OPPs in
> > > > > +  the table have this, the OPP with highest opp-hz will be used.
> > > > 
> > > > 
> > > > By "sustainable", do you mean sustainable across Process, Voltage and
> > > > Temperature corners upto the max rated operational Power-ON hours
> > > > without IDLE state being achieved on the processor?
> > > 
> > > Yes, in case of CPU: running 100% without idle at that particular OPP.
> > > Running above that OPP would lead to cross control temperature.
> > 
> > We need to tighten the definitions a lot more here and add that to the
> > binding. What we are stating, if I am not misunderstanding is an OPP
> > that is guaranteed by SoC vendor that across Process Voltage and
> > Temperature corners - aka across the entire production spectrum
> > for the part number, *all* devices will operate at this OPP for the
> > mandated power-on-hours rating without hitting IDLE.
> > 
> > Example: So -40C to 125C, across the process (hot/cold/nominal), 100s of
> > thousands/millions of units can operate upto 125,0000 power-on-hours
> > while running a tight deadloop OR maybe high processing function or even
> > cpuburn[1]?
> 
> I think I know what you mean. But this would lead to redefining a lot
> more that just this optional field. This wide range -40C to 125C is for
> automotive chips, then what about opp-suspend, when the device cannot
> even reach that OPP under some stress test e.g. outside temp
> ~100-110C...
> Or opp-turbo, shell all the OPPs have multidimensional table to reflect
> the temperature dependency for all affected optional fields?

yes, and down the rabbit hole we will go :)

> 
> > 
> > 
> > Can you give me one SoC vendor and part that guarantees this? I am
> > wondering if this is all theoretical... There are tons of parameters
> > that come into play for "reliability" "sustainability" etc. Those are
> > tricky terminology that typically makes legal folks pretty happy to
> > debate for decades..
> 
> Yes, but the outside temperature is probably most important for this use
> case.
> 
> > 
> > just my 2 cents.
> > > 
> > > > 
> > > > OR do you mean to leave it up to interpretation?
> > > 
> > > I can tell how I would use them. There is thermal governor IPA, which
> > > needs sustainable power either form DT or uses internal algorithm to
> > > estimate it based on lowest allowed freq OPPs. Then it estimated
> > > internal coefficients based on that value, which is not optimal
> > > for lowest OPPs. When some higher OPP could be marked as sustainable,
> > > it would lead to better estimation and better power budget split.
> > 
> > Seeing your series, I got an idea about how you plan on using it, I
> > just think we need to be more precise in our definition..
> 
> Thank you for having a look on that and understanding the motivation
> behind this series.
> 
> How about adding a description that this sustainable OPP is considered
> for normal room temp (20-25C)?

You could.. but then, practically as we go into smaller process nodes,
the 20-25C reliability is just theoretical. I mean, we Texans in summer
or Finns in winter would probably define "normal room temperature" as
something different in practise (ISO not withstanding ;) ).. Challenge
of reliability has always been on the edge of the PVT ranges. To ignore
that OR to have a scheme that does not scale to describe that, IMHO is a
lacking definition.

My entire point is, if we can avoid getting into rabbit hole
definitions, we probably should.. IMHO.. keep things as simple as
possible.

> 
> BTW, in the Arm SCMI spec definition of that value (used in patch 4/4),

You mean [1] Table 11 Performance Domain Levels with Special
	Significance
> there is no specific temperature for it, just:
> 'This is the maximum performance level that the platform can
> sustain under normal conditions. In exceptional circumstances,
> such as thermal runaway, the platform might not be be able to
> guarantee this level.'
> 

Hehe.. Vincent and SCMI teams have been having fun there :)... But, I
think the definition has little practical significance for the very
reasons I made above IMHO, and with full respect to SCMI team(defining
SCMI is not an easy task, I admit) - it is at best a theoretical,
"works at the engineer's cube definition", as typical "nominal
operation conditions" escape clause tend to be, OR at the worst
ignoring to define the parameters that constitute what would bound
things in a closed box precisely (example: does'nt mention process, so
just nominal OR considers all process corners - what does omission of
that factor really mean?).


> I can put this whole description into the DT binding, if you like.

Will leave it to Viresh and others to comment and guide, the terminology
got my attention, since I almost got bit by a similar usage.. my 2 cents:
I dont think that suffices unfortunately. what it lacks are the
parameters of what that terminology really means,

One actual production part that demonstrates this will probably help
guide the discussion, I guess..

/me goes back to OPP hibernation

> 

[1] https://developer.arm.com/documentation/den0056/b

-- 
Regards,
Nishanth Menon
Key (0xDDB5849D1736249D) / Fingerprint: F8A2 8693 54EB 8232 17A3  1A34 DDB5 849D 1736 249D

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Lukasz Luba]

On 10/29/20 1:49 PM, Nishanth Menon wrote:
> On 13:33-20201029, Lukasz Luba wrote:
>>
>>
>> On 10/29/20 12:59 PM, Nishanth Menon wrote:
>>> On 10:04-20201029, Lukasz Luba wrote:
>>>>
>>>>
>>>> On 10/28/20 9:47 PM, Nishanth Menon wrote:
>>>>> On 14:08-20201028, Lukasz Luba wrote:
>>>>>> Add opp-sustainable as an additional property in the OPP node to describe
>>>>>> the sustainable performance level of the device. This will help to
>>>>>> estimate the sustainable performance of the whole system.
>>>>>>
>>>>>> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
>>>>>> ---
>>>>>>     Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>>>>>>     1 file changed, 4 insertions(+)
>>>>>>
>>>>>> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
>>>>>> index 9847dfeeffcb..cd01028de305 100644
>>>>>> --- a/Documentation/devicetree/bindings/opp/opp.txt
>>>>>> +++ b/Documentation/devicetree/bindings/opp/opp.txt
>>>>>> @@ -154,6 +154,10 @@ Optional properties:
>>>>>>     - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>>>>>>       in the table have this, the OPP with highest opp-hz will be used.
>>>>>> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
>>>>>> +  estimating sustainable performance of the whole system. If multiple OPPs in
>>>>>> +  the table have this, the OPP with highest opp-hz will be used.
>>>>>
>>>>>
>>>>> By "sustainable", do you mean sustainable across Process, Voltage and
>>>>> Temperature corners upto the max rated operational Power-ON hours
>>>>> without IDLE state being achieved on the processor?
>>>>
>>>> Yes, in case of CPU: running 100% without idle at that particular OPP.
>>>> Running above that OPP would lead to cross control temperature.
>>>
>>> We need to tighten the definitions a lot more here and add that to the
>>> binding. What we are stating, if I am not misunderstanding is an OPP
>>> that is guaranteed by SoC vendor that across Process Voltage and
>>> Temperature corners - aka across the entire production spectrum
>>> for the part number, *all* devices will operate at this OPP for the
>>> mandated power-on-hours rating without hitting IDLE.
>>>
>>> Example: So -40C to 125C, across the process (hot/cold/nominal), 100s of
>>> thousands/millions of units can operate upto 125,0000 power-on-hours
>>> while running a tight deadloop OR maybe high processing function or even
>>> cpuburn[1]?
>>
>> I think I know what you mean. But this would lead to redefining a lot
>> more that just this optional field. This wide range -40C to 125C is for
>> automotive chips, then what about opp-suspend, when the device cannot
>> even reach that OPP under some stress test e.g. outside temp
>> ~100-110C...
>> Or opp-turbo, shell all the OPPs have multidimensional table to reflect
>> the temperature dependency for all affected optional fields?
> 
> yes, and down the rabbit hole we will go :)
> 
>>
>>>
>>>
>>> Can you give me one SoC vendor and part that guarantees this? I am
>>> wondering if this is all theoretical... There are tons of parameters
>>> that come into play for "reliability" "sustainability" etc. Those are
>>> tricky terminology that typically makes legal folks pretty happy to
>>> debate for decades..
>>
>> Yes, but the outside temperature is probably most important for this use
>> case.
>>
>>>
>>> just my 2 cents.
>>>>
>>>>>
>>>>> OR do you mean to leave it up to interpretation?
>>>>
>>>> I can tell how I would use them. There is thermal governor IPA, which
>>>> needs sustainable power either form DT or uses internal algorithm to
>>>> estimate it based on lowest allowed freq OPPs. Then it estimated
>>>> internal coefficients based on that value, which is not optimal
>>>> for lowest OPPs. When some higher OPP could be marked as sustainable,
>>>> it would lead to better estimation and better power budget split.
>>>
>>> Seeing your series, I got an idea about how you plan on using it, I
>>> just think we need to be more precise in our definition..
>>
>> Thank you for having a look on that and understanding the motivation
>> behind this series.
>>
>> How about adding a description that this sustainable OPP is considered
>> for normal room temp (20-25C)?
> 
> You could.. but then, practically as we go into smaller process nodes,
> the 20-25C reliability is just theoretical. I mean, we Texans in summer
> or Finns in winter would probably define "normal room temperature" as
> something different in practise (ISO not withstanding ;) ).. Challenge
> of reliability has always been on the edge of the PVT ranges. To ignore
> that OR to have a scheme that does not scale to describe that, IMHO is a
> lacking definition.
> 
> My entire point is, if we can avoid getting into rabbit hole
> definitions, we probably should.. IMHO.. keep things as simple as
> possible.
> 
>>
>> BTW, in the Arm SCMI spec definition of that value (used in patch 4/4),
> 
> You mean [1] Table 11 Performance Domain Levels with Special
> 	Significance

Yes, the table 11 from that SCMI doc (under link you provided).

>> there is no specific temperature for it, just:
>> 'This is the maximum performance level that the platform can
>> sustain under normal conditions. In exceptional circumstances,
>> such as thermal runaway, the platform might not be be able to
>> guarantee this level.'
>>
> 
> Hehe.. Vincent and SCMI teams have been having fun there :)... But, I
> think the definition has little practical significance for the very
> reasons I made above IMHO, and with full respect to SCMI team(defining
> SCMI is not an easy task, I admit) - it is at best a theoretical,
> "works at the engineer's cube definition", as typical "nominal
> operation conditions" escape clause tend to be, OR at the worst
> ignoring to define the parameters that constitute what would bound
> things in a closed box precisely (example: does'nt mention process, so
> just nominal OR considers all process corners - what does omission of
> that factor really mean?).
> 
> 
>> I can put this whole description into the DT binding, if you like.
> 
> Will leave it to Viresh and others to comment and guide, the terminology
> got my attention, since I almost got bit by a similar usage.. my 2 cents:
> I dont think that suffices unfortunately. what it lacks are the
> parameters of what that terminology really means,
> 
> One actual production part that demonstrates this will probably help
> guide the discussion, I guess..
> 
> /me goes back to OPP hibernation
> 
>>
> 
> [1] https://developer.arm.com/documentation/den0056/b
> 

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Rob Herring]

On Wed, Oct 28, 2020 at 02:08:44PM +0000, Lukasz Luba wrote:
> Add opp-sustainable as an additional property in the OPP node to describe
> the sustainable performance level of the device. This will help to
> estimate the sustainable performance of the whole system.
> 
> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> ---
>  Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
> index 9847dfeeffcb..cd01028de305 100644
> --- a/Documentation/devicetree/bindings/opp/opp.txt
> +++ b/Documentation/devicetree/bindings/opp/opp.txt
> @@ -154,6 +154,10 @@ Optional properties:
>  - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>    in the table have this, the OPP with highest opp-hz will be used.
>  
> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
> +  estimating sustainable performance of the whole system. If multiple OPPs in
> +  the table have this, the OPP with highest opp-hz will be used.
> +

Isn't this just the inverse of the turbo? or boost? flag we already 
have? 

Couldn't this be learned? I ran at this frequency and then overheated. 
That could be dependent on ambient temperatures or dust build up on 
fans/heatsink.

Rob

Re: [PATCH 1/4] dt-bindings: opp: Introduce opp-sustainable bindings

[Lukasz Luba]

Hi Rob,

On 10/30/20 7:34 PM, Rob Herring wrote:
> On Wed, Oct 28, 2020 at 02:08:44PM +0000, Lukasz Luba wrote:
>> Add opp-sustainable as an additional property in the OPP node to describe
>> the sustainable performance level of the device. This will help to
>> estimate the sustainable performance of the whole system.
>>
>> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
>> ---
>>   Documentation/devicetree/bindings/opp/opp.txt | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt
>> index 9847dfeeffcb..cd01028de305 100644
>> --- a/Documentation/devicetree/bindings/opp/opp.txt
>> +++ b/Documentation/devicetree/bindings/opp/opp.txt
>> @@ -154,6 +154,10 @@ Optional properties:
>>   - opp-suspend: Marks the OPP to be used during device suspend. If multiple OPPs
>>     in the table have this, the OPP with highest opp-hz will be used.
>>   
>> +- opp-sustainable: Marks the OPP as sustainable. This property can be used for
>> +  estimating sustainable performance of the whole system. If multiple OPPs in
>> +  the table have this, the OPP with highest opp-hz will be used.
>> +
> 
> Isn't this just the inverse of the turbo? or boost? flag we already
> have?

True, it could be possible to use those flags. Then a function which
returns one OPP below, would be 'sustainable'. I've already
suggested to skip binding and only have get/set functions for OPP
framework to mark 'sustainable' level, but Viresh is against.
We have discussed this new opp sustainable with Viresh and since it
would be only used by IPA but the definition is a bit blurry,
we decided to drop this series. Maybe in future when there will be
another user, we will come back to this.
I am going to use Energy Model and mark the em_perf_state as sustained.

> 
> Couldn't this be learned? I ran at this frequency and then overheated.
> That could be dependent on ambient temperatures or dust build up on
> fans/heatsink.

Yes it could be learned. IPA tries to do that for different workloads,
but it needs some starting conditions (coefficients and sustainable
power) to converge in short time. Sustainable power is currently
estimated based on lowest OPP. With that new 'sustainable' flag,
it would be much easier for IPA to converge.

Regards,
Lukasz

> 
> Rob
> 

[PATCH 2/4] OPP: Add support for parsing the 'opp-sustainable' property

[Lukasz Luba]

Now that the OPP bindings are updated to include an optional
'opp-sustainable' property, add support to parse it from device tree and
store it as part of dev_pm_opp structure.
Also add and export an helper 'dev_pm_opp_get_sustainable()' that can be
used to get the sustainable OPP when present.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 drivers/opp/core.c     | 26 ++++++++++++++++++++++++++
 drivers/opp/of.c       | 14 ++++++++++++++
 drivers/opp/opp.h      |  3 +++
 include/linux/pm_opp.h |  6 ++++++
 4 files changed, 49 insertions(+)

diff --git a/drivers/opp/core.c b/drivers/opp/core.c
index 9915e8487f0b..bb1e68b96d0e 100644
--- a/drivers/opp/core.c
+++ b/drivers/opp/core.c
@@ -299,6 +299,32 @@ unsigned long dev_pm_opp_get_suspend_opp_freq(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(dev_pm_opp_get_suspend_opp_freq);
 
+/**
+ * dev_pm_opp_get_sustainable_opp_freq() - Get frequency of sustainable opp
+ *					   in Hz
+ * @dev:	device for which we do this operation
+ *
+ * Return: This function returns the frequency of the OPP marked as
+ * sustainable_opp if one is available, else returns 0;
+ */
+unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev)
+{
+	struct opp_table *opp_table;
+	unsigned long freq = 0;
+
+	opp_table = _find_opp_table(dev);
+	if (IS_ERR(opp_table))
+		return 0;
+
+	if (opp_table->sustainable_opp && opp_table->sustainable_opp->available)
+		freq = dev_pm_opp_get_freq(opp_table->sustainable_opp);
+
+	dev_pm_opp_put_opp_table(opp_table);
+
+	return freq;
+}
+EXPORT_SYMBOL_GPL(dev_pm_opp_get_sustainable_opp_freq);
+
 int _get_opp_count(struct opp_table *opp_table)
 {
 	struct dev_pm_opp *opp;
diff --git a/drivers/opp/of.c b/drivers/opp/of.c
index 9faeb83e4b32..1f6b19bb1a95 100644
--- a/drivers/opp/of.c
+++ b/drivers/opp/of.c
@@ -815,6 +815,20 @@ static struct dev_pm_opp *_opp_add_static_v2(struct opp_table *opp_table,
 		}
 	}
 
+	if (of_property_read_bool(np, "opp-sustainable")) {
+		if (opp_table->sustainable_opp) {
+			/* Pick the OPP with higher rate as sustainable OPP */
+			if (new_opp->rate > opp_table->sustainable_opp->rate) {
+				opp_table->sustainable_opp->sustainable = false;
+				new_opp->sustainable = true;
+				opp_table->sustainable_opp = new_opp;
+			}
+		} else {
+			new_opp->sustainable = true;
+			opp_table->sustainable_opp = new_opp;
+		}
+	}
+
 	if (new_opp->clock_latency_ns > opp_table->clock_latency_ns_max)
 		opp_table->clock_latency_ns_max = new_opp->clock_latency_ns;
 
diff --git a/drivers/opp/opp.h b/drivers/opp/opp.h
index ebd930e0b3ca..45288ccbb295 100644
--- a/drivers/opp/opp.h
+++ b/drivers/opp/opp.h
@@ -56,6 +56,7 @@ extern struct list_head opp_tables;
  * @dynamic:	not-created from static DT entries.
  * @turbo:	true if turbo (boost) OPP
  * @suspend:	true if suspend OPP
+ * @sustainable: true if sustainable OPP
  * @pstate: Device's power domain's performance state.
  * @rate:	Frequency in hertz
  * @level:	Performance level
@@ -78,6 +79,7 @@ struct dev_pm_opp {
 	bool dynamic;
 	bool turbo;
 	bool suspend;
+	bool sustainable;
 	unsigned int pstate;
 	unsigned long rate;
 	unsigned int level;
@@ -183,6 +185,7 @@ struct opp_table {
 	unsigned int parsed_static_opps;
 	enum opp_table_access shared_opp;
 	struct dev_pm_opp *suspend_opp;
+	struct dev_pm_opp *sustainable_opp;
 
 	struct mutex genpd_virt_dev_lock;
 	struct device **genpd_virt_devs;
diff --git a/include/linux/pm_opp.h b/include/linux/pm_opp.h
index dbb484524f82..363d5a2c1ef3 100644
--- a/include/linux/pm_opp.h
+++ b/include/linux/pm_opp.h
@@ -106,6 +106,7 @@ unsigned long dev_pm_opp_get_max_clock_latency(struct device *dev);
 unsigned long dev_pm_opp_get_max_volt_latency(struct device *dev);
 unsigned long dev_pm_opp_get_max_transition_latency(struct device *dev);
 unsigned long dev_pm_opp_get_suspend_opp_freq(struct device *dev);
+unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev);
 
 struct dev_pm_opp *dev_pm_opp_find_freq_exact(struct device *dev,
 					      unsigned long freq,
@@ -215,6 +216,11 @@ static inline unsigned long dev_pm_opp_get_suspend_opp_freq(struct device *dev)
 	return 0;
 }
 
+static inline unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev)
+{
+	return 0;
+}
+
 static inline struct dev_pm_opp *dev_pm_opp_find_freq_exact(struct device *dev,
 					unsigned long freq, bool available)
 {
-- 
2.17.1

Re: [PATCH 2/4] OPP: Add support for parsing the 'opp-sustainable' property

[Quentin Perret]

Hi Lukasz,

On Wednesday 28 Oct 2020 at 14:08:45 (+0000), Lukasz Luba wrote:
> +unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev)
> +{
> +	struct opp_table *opp_table;
> +	unsigned long freq = 0;
> +
> +	opp_table = _find_opp_table(dev);
> +	if (IS_ERR(opp_table))
> +		return 0;
> +
> +	if (opp_table->sustainable_opp && opp_table->sustainable_opp->available)
> +		freq = dev_pm_opp_get_freq(opp_table->sustainable_opp);
> +
> +	dev_pm_opp_put_opp_table(opp_table);
> +
> +	return freq;
> +}
> +EXPORT_SYMBOL_GPL(dev_pm_opp_get_sustainable_opp_freq);

I'm guessing this is what IPA will use to find out what the sustainable
frequency is right?

Is PM_OPP the right place for that? It feels odd IPA will get the EM
from one place, which includes the performance state, and the sustained
OPP from another. Should we move that to PM_EM instead?

Thanks,
Quentin

Re: [PATCH 2/4] OPP: Add support for parsing the 'opp-sustainable' property

[Lukasz Luba]

Hi Quentin,

On 10/30/20 11:47 AM, Quentin Perret wrote:
> Hi Lukasz,
> 
> On Wednesday 28 Oct 2020 at 14:08:45 (+0000), Lukasz Luba wrote:
>> +unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev)
>> +{
>> +	struct opp_table *opp_table;
>> +	unsigned long freq = 0;
>> +
>> +	opp_table = _find_opp_table(dev);
>> +	if (IS_ERR(opp_table))
>> +		return 0;
>> +
>> +	if (opp_table->sustainable_opp && opp_table->sustainable_opp->available)
>> +		freq = dev_pm_opp_get_freq(opp_table->sustainable_opp);
>> +
>> +	dev_pm_opp_put_opp_table(opp_table);
>> +
>> +	return freq;
>> +}
>> +EXPORT_SYMBOL_GPL(dev_pm_opp_get_sustainable_opp_freq);
> 
> I'm guessing this is what IPA will use to find out what the sustainable
> frequency is right?

Yes, you are right.

> 
> Is PM_OPP the right place for that? It feels odd IPA will get the EM
> from one place, which includes the performance state, and the sustained
> OPP from another. Should we move that to PM_EM instead?

True, it might looks strange, but the OPP framework is available when we
are adding the OPPs in scmi perf layer. The EM is available after we
register the device, so at the end of scmi-cpufreq init.
It would require a new scmi perf api function e.g. get_sustained_freq(),
and a set/get function for EM, which is doable.

I've discussed this approach to Viresh and he likes it better.
I am happy that you are also suggesting the EM approach.

I will send different patches for EM and SCMI to make that happen.
Should I re-based them on top of the patch adding this milliwatts filed
in EM [1]? Or do the opposite, changing the dependency order?

Regards,
Lukasz

[1] https://lkml.org/lkml/2020/10/19/392

> 
> Thanks,
> Quentin
> 

[PATCH 3/4] OPP: Add dev_pm_opp_set_sustainable_opp_freq()

[Lukasz Luba]

Add a new function allowing to set sustainable OPP from the drivers code.
This is helpful when the device drivers populate the OPP table by
themself.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 drivers/opp/core.c     | 49 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/pm_opp.h |  8 +++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/opp/core.c b/drivers/opp/core.c
index bb1e68b96d0e..753443f17327 100644
--- a/drivers/opp/core.c
+++ b/drivers/opp/core.c
@@ -325,6 +325,55 @@ unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(dev_pm_opp_get_sustainable_opp_freq);
 
+/**
+ * dev_pm_opp_set_sustainable_opp_freq() - Find and set the OPP sustainable
+ *					   based on given frequency
+ * @dev:	device for which we do this operation
+ * @freq:	frequency which OPP is going to be marked
+ *
+ * Mark the OPP of a given frequency @freq as sustainable_opp. If there is
+ * already sustainable OPP, choose the highest frequency one and set.
+ *
+ * Return: This function return 0 when the new OPP is marked as sustainable
+ * or error return value, which can be:
+ * EINVAL:	for bad pointer or no match found for frequency search
+ * EEXIST:	if existing sustainable OPP has higher rate
+ */
+int dev_pm_opp_set_sustainable_opp_freq(struct device *dev, unsigned long freq)
+{
+	struct opp_table *opp_table;
+	struct dev_pm_opp *opp;
+	int ret = 0;
+
+	opp = dev_pm_opp_find_freq_exact(dev, freq, true);
+	if (IS_ERR_OR_NULL(opp))
+		return -EINVAL;
+
+	opp_table = _find_opp_table(dev);
+	if (IS_ERR(opp_table))
+		return -EINVAL;
+
+	if (opp_table->sustainable_opp) {
+		/* Pick the OPP with higher rate as sustainable OPP */
+		if (opp->rate > opp_table->sustainable_opp->rate) {
+			opp_table->sustainable_opp->sustainable = false;
+			opp->sustainable = true;
+			opp_table->sustainable_opp = opp;
+		} else {
+			ret = -EEXIST;
+		}
+	} else {
+		opp->sustainable = true;
+		opp_table->sustainable_opp = opp;
+	}
+
+	dev_pm_opp_put(opp);
+	dev_pm_opp_put_opp_table(opp_table);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(dev_pm_opp_set_sustainable_opp_freq);
+
 int _get_opp_count(struct opp_table *opp_table)
 {
 	struct dev_pm_opp *opp;
diff --git a/include/linux/pm_opp.h b/include/linux/pm_opp.h
index 363d5a2c1ef3..e634a6b00e65 100644
--- a/include/linux/pm_opp.h
+++ b/include/linux/pm_opp.h
@@ -107,6 +107,8 @@ unsigned long dev_pm_opp_get_max_volt_latency(struct device *dev);
 unsigned long dev_pm_opp_get_max_transition_latency(struct device *dev);
 unsigned long dev_pm_opp_get_suspend_opp_freq(struct device *dev);
 unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *dev);
+int dev_pm_opp_set_sustainable_opp_freq(struct device *dev,
+					unsigned long freq);
 
 struct dev_pm_opp *dev_pm_opp_find_freq_exact(struct device *dev,
 					      unsigned long freq,
@@ -221,6 +223,12 @@ static inline unsigned long dev_pm_opp_get_sustainable_opp_freq(struct device *d
 	return 0;
 }
 
+static inline int dev_pm_opp_set_sustainable_freq(struct device *dev,
+					unsigned long freq)
+{
+	return -EOPNOTSUPP;
+}
+
 static inline struct dev_pm_opp *dev_pm_opp_find_freq_exact(struct device *dev,
 					unsigned long freq, bool available)
 {
-- 
2.17.1

[PATCH 4/4] firmware: arm_scmi/perf: Mark sustainable OPP

[Lukasz Luba]

Mark sustainable OPP frequency based on previously received value from
the firmware.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 drivers/firmware/arm_scmi/perf.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/firmware/arm_scmi/perf.c b/drivers/firmware/arm_scmi/perf.c
index 82fb3babff72..a8f3fa1d7f1e 100644
--- a/drivers/firmware/arm_scmi/perf.c
+++ b/drivers/firmware/arm_scmi/perf.c
@@ -669,6 +669,16 @@ static int scmi_dvfs_device_opps_add(const struct scmi_handle *handle,
 			return ret;
 		}
 	}
+
+	if (dom->sustained_freq_khz) {
+		/* The sustained frequency is always in kHz */
+		freq = dom->sustained_freq_khz * 1000;
+		ret = dev_pm_opp_set_sustainable_opp_freq(dev, freq);
+		if (ret)
+			dev_warn(dev, "failed to add sustainable freq=%lu %d\n",
+				 freq, ret);
+	}
+
 	return 0;
 }
 
-- 
2.17.1

Re: [PATCH 0/4] Add sustainable OPP concept

[Viresh Kumar]

On 28-10-20, 14:08, Lukasz Luba wrote:
> Hi all,
> 
> This patch set introduces a concept of sustainable OPP, which then can be used
> by kernel frameworks or governors for estimating system sustainable system
> state. This kind of estimation is done e.g. in thermal governor Intelligent
> Power Allocation (IPA), which calculates sustainable power of the whole system
> and then derives some coefficients for internal algorithm.
> 
> The patch set introduces a new DT bindings 'opp-sustainable', with parsing
> code. It also adds a function (in patch 3/4) which allows device drivers to set
> directly the sustainable OPP. This is helpful when the device drivers populate
> the OPP table by themself (example in patch 4/4).
> 

Can we please have some more information about this ? What does the
sustainable OPP mean ? How will platform guys know or learn about this
? How we are going to use it finally ? What does it have to do with
temperature of the SoC or the thermal affects, etc.

-- 
viresh

Re: [PATCH 0/4] Add sustainable OPP concept

[Viresh Kumar]

On 29-10-20, 13:10, Viresh Kumar wrote:
> On 28-10-20, 14:08, Lukasz Luba wrote:
> > Hi all,
> > 
> > This patch set introduces a concept of sustainable OPP, which then can be used
> > by kernel frameworks or governors for estimating system sustainable system
> > state. This kind of estimation is done e.g. in thermal governor Intelligent
> > Power Allocation (IPA), which calculates sustainable power of the whole system
> > and then derives some coefficients for internal algorithm.
> > 
> > The patch set introduces a new DT bindings 'opp-sustainable', with parsing
> > code. It also adds a function (in patch 3/4) which allows device drivers to set
> > directly the sustainable OPP. This is helpful when the device drivers populate
> > the OPP table by themself (example in patch 4/4).
> > 
> 
> Can we please have some more information about this ? What does the
> sustainable OPP mean ? How will platform guys know or learn about this
> ? How we are going to use it finally ? What does it have to do with
> temperature of the SoC or the thermal affects, etc.

And that we need a real user of this first if it is ever going to be
merged.

-- 
viresh

Re: [PATCH 0/4] Add sustainable OPP concept

[Lukasz Luba]

On 10/29/20 7:53 AM, Viresh Kumar wrote:
> On 29-10-20, 13:10, Viresh Kumar wrote:
>> On 28-10-20, 14:08, Lukasz Luba wrote:
>>> Hi all,
>>>
>>> This patch set introduces a concept of sustainable OPP, which then can be used
>>> by kernel frameworks or governors for estimating system sustainable system
>>> state. This kind of estimation is done e.g. in thermal governor Intelligent
>>> Power Allocation (IPA), which calculates sustainable power of the whole system
>>> and then derives some coefficients for internal algorithm.
>>>
>>> The patch set introduces a new DT bindings 'opp-sustainable', with parsing
>>> code. It also adds a function (in patch 3/4) which allows device drivers to set
>>> directly the sustainable OPP. This is helpful when the device drivers populate
>>> the OPP table by themself (example in patch 4/4).
>>>
>>
>> Can we please have some more information about this ? What does the
>> sustainable OPP mean ? How will platform guys know or learn about this
>> ? How we are going to use it finally ? What does it have to do with
>> temperature of the SoC or the thermal affects, etc.

There were discussions about Energy Model (EM), scale of values (mW or
abstract scale) and relation to EAS and IPA. You can find quite long
discussion below v2 [1] (there is also v3 send after agreement [2]).
We have in thermal DT binding: 'sustainable-power' expressed in mW,
which is used by IPA, but it would not support bogoWatts.
The sustainable power is used for estimation of internal coefficients
(also for power budget), which I am trying to change to work with
'abstract scale' [3][4].

This would allow to estimate sustainable power of the system based on
CPUs, GPU opp-sustainable points, where we don't have
'sustainable-power' or devices using bogoWatts.

> 
> And that we need a real user of this first if it is ever going to be
> merged.
> 

IPA would be the first user of this in combination with scmi-cpufreq.c,
which can feed 'abstract scale' in to EM.
Currently IPA takes lowest allowed OPPs into account for this estimation
which is not optimal. This marked OPPs would make estimation a lot
better.

Regards,
Lukasz


[1] https://lore.kernel.org/lkml/20201002114426.31277-1-lukasz.luba@arm.com/
[2] https://lore.kernel.org/lkml/20201019140601.3047-1-lukasz.luba@arm.com/
[3] 
https://lore.kernel.org/linux-pm/5f682bbb-b250-49e6-dbb7-aea522a58595@arm.com/
[4] https://lore.kernel.org/lkml/20201009135850.14727-1-lukasz.luba@arm.com/

Re: [PATCH 0/4] Add sustainable OPP concept

[Viresh Kumar]

On 29-10-20, 09:56, Lukasz Luba wrote:
> There were discussions about Energy Model (EM), scale of values (mW or
> abstract scale) and relation to EAS and IPA. You can find quite long
> discussion below v2 [1] (there is also v3 send after agreement [2]).
> We have in thermal DT binding: 'sustainable-power' expressed in mW,
> which is used by IPA, but it would not support bogoWatts.

Why so ? (I am sorry, can't dig into such long threads without knowing
which message I am looking for :( ). Lets assume if that same property
can be used for bogoWatts, will that be sufficient for you ? Or you
will still need this patch set ?

> The sustainable power is used for estimation of internal coefficients
> (also for power budget), which I am trying to change to work with
> 'abstract scale' [3][4].
> 
> This would allow to estimate sustainable power of the system based on
> CPUs, GPU opp-sustainable points, where we don't have
> 'sustainable-power' or devices using bogoWatts.

Then maybe we should ahve sustainable-power in those cases too instead
of adding a meaningless (IMHO) binding.

Honestly speaking, as Nishanth said, there is nothing like a
sustainable OPP in reality. Moreover, the DT needs to describe the
hardware as it is (and in some cases the behavior of the firmware).
And what you are trying to add here is none of them and so it should
not go in DT as such. There are too many factors which play a part
here, ambient temperature is one of the biggest ones, and the software
needs to find the sustainable OPP by itself based on the current
situation.

So I don't really see a good reason why such a property should be
added here.

Coming to properties like suspend-opp, it made sense for some of the
platforms as the last configured frequency of the CPU plays a part in
deciding the power consumed by the SoC even when the system is
suspended. And finding an optimal OPP (normally the lowest) there
would make sense and so was that property added.

-- 
viresh

Re: [PATCH 0/4] Add sustainable OPP concept

[Lukasz Luba]

On 10/30/20 8:29 AM, Viresh Kumar wrote:
> On 29-10-20, 09:56, Lukasz Luba wrote:
>> There were discussions about Energy Model (EM), scale of values (mW or
>> abstract scale) and relation to EAS and IPA. You can find quite long
>> discussion below v2 [1] (there is also v3 send after agreement [2]).
>> We have in thermal DT binding: 'sustainable-power' expressed in mW,
>> which is used by IPA, but it would not support bogoWatts.
> 
> Why so ? (I am sorry, can't dig into such long threads without knowing
> which message I am looking for :( ). Lets assume if that same property
> can be used for bogoWatts, will that be sufficient for you ? Or you
> will still need this patch set ?

I had a patch for that, but I know Rob's opinion on this one [1] (which
is below in that thread).

> 
>> The sustainable power is used for estimation of internal coefficients
>> (also for power budget), which I am trying to change to work with
>> 'abstract scale' [3][4].
>>
>> This would allow to estimate sustainable power of the system based on
>> CPUs, GPU opp-sustainable points, where we don't have
>> 'sustainable-power' or devices using bogoWatts.
> 
> Then maybe we should ahve sustainable-power in those cases too instead
> of adding a meaningless (IMHO) binding.

How about dropping the DT binding, but just adding this new field into
dev_pm_opp? There will be no DT parsing code, just the get/set
functions, which will be used in SCMI patch 4/4 and in IPA?
That would not require to change any DT bindings.

> 
> Honestly speaking, as Nishanth said, there is nothing like a
> sustainable OPP in reality. Moreover, the DT needs to describe the
> hardware as it is (and in some cases the behavior of the firmware).
> And what you are trying to add here is none of them and so it should
> not go in DT as such. There are too many factors which play a part
> here, ambient temperature is one of the biggest ones, and the software
> needs to find the sustainable OPP by itself based on the current
> situation.
> 
> So I don't really see a good reason why such a property should be
> added here.

I see. Just for your information SCMI supports 'Sustained Performance'
  expressed in kHz.

> 
> Coming to properties like suspend-opp, it made sense for some of the
> platforms as the last configured frequency of the CPU plays a part in
> deciding the power consumed by the SoC even when the system is
> suspended. And finding an optimal OPP (normally the lowest) there
> would make sense and so was that property added.
> 

I also found that suspend-opp (83f8ca45afbf041e312909).
I hope you wouldn't mind if I add this new field into dev_pm_opp (no DT
support, just FW).


[1] https://lore.kernel.org/lkml/20201002114426.31277-4-lukasz.luba@arm.com/

Re: [PATCH 0/4] Add sustainable OPP concept

[Viresh Kumar]

On 30-10-20, 09:19, Lukasz Luba wrote:
> How about dropping the DT binding, but just adding this new field into
> dev_pm_opp? There will be no DT parsing code, just the get/set
> functions, which will be used in SCMI patch 4/4 and in IPA?
> That would not require to change any DT bindings.
 
> I see. Just for your information SCMI supports 'Sustained Performance'
>  expressed in kHz.

Even that doesn't sound great (but then I don't have any background of
why that was added there). The problem is not about how do we get this
data into the kernel (from DT or firmware), but why is it even
required. I really feel that software can find the sustainable OPP by
itself (which can keep changing).

About moving it into the OPP core, I am open to getting something
added there if it is really useful and if the OPP core is the best
suited place to keep such data. Though I am not sure of that for this
field right now.

Is it ever going to be used by anyone else apart from IPA ? If not,
what about adding a helper in IPA to set sustainable-freq for a device
?

So only SCMI based platforms will be able to use this stuff ? That's
very limited, isn't it ? I think we should still try to make it better
for everyone by making the software smarter. It has so much data, the
OPPs, the power it will consume (based on microvolt property?), the
heat we produce from that (from thermal framework), etc. Perhaps
building this information continuously at runtime based on when and
how we hit the trip points ? So we know which is the right frequency
where we can refrain from hitting the trip points.

But may be I am asking too much :(

-- 
viresh

Re: [PATCH 0/4] Add sustainable OPP concept

[Lukasz Luba]

On 10/30/20 9:52 AM, Viresh Kumar wrote:
> On 30-10-20, 09:19, Lukasz Luba wrote:
>> How about dropping the DT binding, but just adding this new field into
>> dev_pm_opp? There will be no DT parsing code, just the get/set
>> functions, which will be used in SCMI patch 4/4 and in IPA?
>> That would not require to change any DT bindings.
>   
>> I see. Just for your information SCMI supports 'Sustained Performance'
>>   expressed in kHz.
> 
> Even that doesn't sound great (but then I don't have any background of
> why that was added there). The problem is not about how do we get this
> data into the kernel (from DT or firmware), but why is it even
> required. I really feel that software can find the sustainable OPP by
> itself (which can keep changing).

IPA tries to do that, even dynamically when e.g. GPU is supper busy
in 3D games (~2000W) or almost idle showing 2D home screen.
It tries to find highest 'sustainable' frequencies for the devices,
at that various workloads and temp. But it needs some coefficients to
start, which have big impact on the algorithm. It could slow down IPA a
lot, when those coefficients are calculated based on lowest OPPs.


> 
> About moving it into the OPP core, I am open to getting something
> added there if it is really useful and if the OPP core is the best
> suited place to keep such data. Though I am not sure of that for this
> field right now.
> 
> Is it ever going to be used by anyone else apart from IPA ? If not,
> what about adding a helper in IPA to set sustainable-freq for a device
> ?

My backup plan was to add a flag into EM em_perf_state, extend SCMI perf
exposing the 'sustained_freq_khz' to scmi-cpufreq, which would set that
field after registering EM. IPA depends on EM, so should be OK.

> 
> So only SCMI based platforms will be able to use this stuff ? That's

I don't know who would also use it in future. I just presented you
current user of this, as you asked.

> very limited, isn't it ? I think we should still try to make it better
> for everyone by making the software smarter. It has so much data, the
> OPPs, the power it will consume (based on microvolt property?), the
> heat we produce from that (from thermal framework), etc. Perhaps
> building this information continuously at runtime based on when and
> how we hit the trip points ? So we know which is the right frequency
> where we can refrain from hitting the trip points.

IPA works in this way.

> 
> But may be I am asking too much :(
> 

When you asked for user of this, I gave you instantly. This is one is
more difficult. I am still not there with IPA tests in LISA. I have some
out-of-tree kernel driver for testing, which also need polishing before
can be used with LISA. Then proper workloads with results processing.
EM for devfreq cooling devices. Then decent 'hot' board running
preferably mainline kernel.
What you requested is on my list, but it needs more work, which
won't be ready over night.

Regards,
Lukasz

Re: [PATCH 0/4] Add sustainable OPP concept

[Viresh Kumar]

On 30-10-20, 10:56, Lukasz Luba wrote:
> IPA tries to do that, even dynamically when e.g. GPU is supper busy
> in 3D games (~2000W) or almost idle showing 2D home screen.
> It tries to find highest 'sustainable' frequencies for the devices,
> at that various workloads and temp. But it needs some coefficients to
> start, which have big impact on the algorithm. It could slow down IPA a
> lot, when those coefficients are calculated based on lowest OPPs.

I see. So when you say it slows down IPA, what does that really mean ?
IPA isn't performing that accurately during the initial period of
booting (any time estimate here) ? Does it work fine after a time
duration? Or will it suffer for ever ?

And maybe you shouldn't start with the lowest OPPs while you calculate
these coefficients dynamically ? Maybe start from the middle ? As the
sustainable OPP would be something there only or maybe a bit higher
only. But yeah, I don't have any idea about how those coefficients are
calculated so this idea can be simply ignored as well :)

> My backup plan was to add a flag into EM em_perf_state, extend SCMI perf
> exposing the 'sustained_freq_khz' to scmi-cpufreq, which would set that
> field after registering EM. IPA depends on EM, so should be OK.

I think at this point (considering the limited number of users (only
IPA) and providers (only SCMI)), it would be better that way only
instead of updating the OPP framework. Of course we can revisit that
if we ever feel that we need a better placeholder for it.

> > So only SCMI based platforms will be able to use this stuff ? That's
> > very limited, isn't it ? I think we should still try to make it better
> > for everyone by making the software smarter. It has so much data, the
> > OPPs, the power it will consume (based on microvolt property?), the
> > heat we produce from that (from thermal framework), etc. Perhaps
> > building this information continuously at runtime based on when and
> > how we hit the trip points ? So we know which is the right frequency
> > where we can refrain from hitting the trip points.
> 
> IPA works in this way.

Nice, that's what I thought as well but then got a bit confused with
your patchset.

> > But may be I am asking too much :(
> > 
> 
> When you asked for user of this, I gave you instantly. This is one is
> more difficult. I am still not there with IPA tests in LISA. I have some
> out-of-tree kernel driver for testing, which also need polishing before
> can be used with LISA. Then proper workloads with results processing.
> EM for devfreq cooling devices. Then decent 'hot' board running
> preferably mainline kernel.
> What you requested is on my list, but it needs more work, which
> won't be ready over night.

I can understand what you are trying to do here. And this surely
requires a lot of effort.

-- 
viresh

Re: [PATCH 0/4] Add sustainable OPP concept

[Lukasz Luba]

On 10/30/20 11:17 AM, Viresh Kumar wrote:
> On 30-10-20, 10:56, Lukasz Luba wrote:
>> IPA tries to do that, even dynamically when e.g. GPU is supper busy
>> in 3D games (~2000W) or almost idle showing 2D home screen.
>> It tries to find highest 'sustainable' frequencies for the devices,
>> at that various workloads and temp. But it needs some coefficients to
>> start, which have big impact on the algorithm. It could slow down IPA a
>> lot, when those coefficients are calculated based on lowest OPPs.
> 
> I see. So when you say it slows down IPA, what does that really mean ?
> IPA isn't performing that accurately during the initial period of
> booting (any time estimate here) ? Does it work fine after a time
> duration? Or will it suffer for ever ?

The coefficients would stay 'forever', which determine the temp rising
slope, until someone change them via sysfs (the: k_po, k_pu, k_i,
sustainable_power).

> 
> And maybe you shouldn't start with the lowest OPPs while you calculate
> these coefficients dynamically ? Maybe start from the middle ? As the
> sustainable OPP would be something there only or maybe a bit higher
> only. But yeah, I don't have any idea about how those coefficients are
> calculated so this idea can be simply ignored as well :)
> 
>> My backup plan was to add a flag into EM em_perf_state, extend SCMI perf
>> exposing the 'sustained_freq_khz' to scmi-cpufreq, which would set that
>> field after registering EM. IPA depends on EM, so should be OK.
> 
> I think at this point (considering the limited number of users (only
> IPA) and providers (only SCMI)), it would be better that way only
> instead of updating the OPP framework. Of course we can revisit that
> if we ever feel that we need a better placeholder for it.

OK, sounds good.

> 
>>> So only SCMI based platforms will be able to use this stuff ? That's
>>> very limited, isn't it ? I think we should still try to make it better
>>> for everyone by making the software smarter. It has so much data, the
>>> OPPs, the power it will consume (based on microvolt property?), the
>>> heat we produce from that (from thermal framework), etc. Perhaps
>>> building this information continuously at runtime based on when and
>>> how we hit the trip points ? So we know which is the right frequency
>>> where we can refrain from hitting the trip points.
>>
>> IPA works in this way.
> 
> Nice, that's what I thought as well but then got a bit confused with
> your patchset.
> 
>>> But may be I am asking too much :(
>>>
>>
>> When you asked for user of this, I gave you instantly. This is one is
>> more difficult. I am still not there with IPA tests in LISA. I have some
>> out-of-tree kernel driver for testing, which also need polishing before
>> can be used with LISA. Then proper workloads with results processing.
>> EM for devfreq cooling devices. Then decent 'hot' board running
>> preferably mainline kernel.
>> What you requested is on my list, but it needs more work, which
>> won't be ready over night.
> 
> I can understand what you are trying to do here. And this surely
> requires a lot of effort.
> 

Thank you Viresh for your opinion.
I will take the EM approach, please ignore this patch set.

Regards,
Lukasz