[PATCH v2 0/3] Add basic tracing for uclamp and schedutil

[PATCH v2 0/3] Add basic tracing for uclamp and schedutil

[Lukasz Luba]

Hi all,

The task scheduler feature: Uclamp, begins to take off. To better understand
the dynamics in the task scheduler and CPU frequency requests we need some
better tracing.
In schedutil (cpufreq governor) we allow to enter the scheduler
and make the frequency change. Although, there is some limit in regards to how
often this can happen. That min period is provided by the cpufreq driver.
Thus, some of the cpufreq requests might be filter out and the frequency won't
be changed (hopefuly will be set a bit later). We would like to know about
those situations, especially in context of the user-space hints made via
Uclamp for particular tasks.
This patch set aims to add base for our toolkits and post-processing trace
analyzes.

Changelog:
v2:
- solved the issue from CI build warning, dropped schedutil.h and re-used
  the sched.h which is available in build_utility.c where cpufreq_schedutil.c
  is included
- added tag for the last patch 3/3 for the CI robot helping hend 
- re-based on top of v6.4-rc1
v1:
- implementation can be found here [1]


Regards,
Lukasz Luba

[1] https://lore.kernel.org/lkml/20230322151843.14390-1-lukasz.luba@arm.com/

Lukasz Luba (3):
  sched/tp: Add new tracepoint to track uclamp set from user-space
  cpufreq: schedutil: Refactor sugov_update_shared() internals
  schedutil: trace: Add tracing to capture filter out requests

 include/trace/events/sched.h     |  8 ++++++++
 kernel/sched/core.c              |  5 +++++
 kernel/sched/cpufreq_schedutil.c | 28 ++++++++++++++++++----------
 3 files changed, 31 insertions(+), 10 deletions(-)

-- 
2.25.1

[PATCH v2 1/3] sched/tp: Add new tracepoint to track uclamp set from user-space

[Lukasz Luba]

The user-space can set uclamp value for a given task. It impacts task
placement decisions made by the scheduler. This is very useful information
and helps to understand the system behavior or track improvements in
middleware and applications which start using uclamp mechanisms and report
better performance in tests.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 include/trace/events/sched.h | 4 ++++
 kernel/sched/core.c          | 5 +++++
 2 files changed, 9 insertions(+)

diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index fbb99a61f714..dbfb30809f15 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -735,6 +735,10 @@ DECLARE_TRACE(sched_update_nr_running_tp,
 	TP_PROTO(struct rq *rq, int change),
 	TP_ARGS(rq, change));
 
+DECLARE_TRACE(uclamp_update_tsk_tp,
+	TP_PROTO(struct task_struct *tsk, int uclamp_id,  unsigned int value),
+	TP_ARGS(tsk, uclamp_id, value));
+
 #endif /* _TRACE_SCHED_H */
 
 /* This part must be outside protection */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 944c3ae39861..7b9b800ebb6c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -114,6 +114,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_overutilized_tp);
 EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_cfs_tp);
 EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp);
 EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp);
+EXPORT_TRACEPOINT_SYMBOL_GPL(uclamp_update_tsk_tp);
 
 DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
 
@@ -1956,12 +1957,16 @@ static void __setscheduler_uclamp(struct task_struct *p,
 	    attr->sched_util_min != -1) {
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],
 			      attr->sched_util_min, true);
+		trace_uclamp_update_tsk_tp(p, UCLAMP_MIN,
+					   attr->sched_util_min);
 	}
 
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX &&
 	    attr->sched_util_max != -1) {
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
 			      attr->sched_util_max, true);
+		trace_uclamp_update_tsk_tp(p, UCLAMP_MAX,
+					   attr->sched_util_max);
 	}
 }
 
-- 
2.25.1

[PATCH v2 2/3] cpufreq: schedutil: Refactor sugov_update_shared() internals

[Lukasz Luba]

Remove the if section block. Use the simple check to bail out
and jump to the unlock at the end. That makes the code more readable
and ready for some future tracing.

Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 kernel/sched/cpufreq_schedutil.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index e3211455b203..f462496e5c07 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -446,17 +446,19 @@ sugov_update_shared(struct update_util_data *hook, u64 time, unsigned int flags)
 
 	ignore_dl_rate_limit(sg_cpu);
 
-	if (sugov_should_update_freq(sg_policy, time)) {
-		next_f = sugov_next_freq_shared(sg_cpu, time);
+	if (!sugov_should_update_freq(sg_policy, time))
+		goto unlock;
 
-		if (!sugov_update_next_freq(sg_policy, time, next_f))
-			goto unlock;
+	next_f = sugov_next_freq_shared(sg_cpu, time);
+
+	if (!sugov_update_next_freq(sg_policy, time, next_f))
+		goto unlock;
+
+	if (sg_policy->policy->fast_switch_enabled)
+		cpufreq_driver_fast_switch(sg_policy->policy, next_f);
+	else
+		sugov_deferred_update(sg_policy);
 
-		if (sg_policy->policy->fast_switch_enabled)
-			cpufreq_driver_fast_switch(sg_policy->policy, next_f);
-		else
-			sugov_deferred_update(sg_policy);
-	}
 unlock:
 	raw_spin_unlock(&sg_policy->update_lock);
 }
-- 
2.25.1

[PATCH v2 3/3] schedutil: trace: Add tracing to capture filter out requests

[Lukasz Luba]

Some of the frequency update requests coming form the task scheduler
might be filter out. It can happen when the previous request was served
not that long ago (in a period smaller than provided by the cpufreq driver
as minimum for frequency update). In such case, we want to know if some of
the frequency updates cannot make through.
Export the new tracepoint as well. That would allow to handle it by a
toolkit for trace analyzes.

Reported-by: kernel test robot <lkp@intel.com> # solved tricky build
Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
---
 include/trace/events/sched.h     |  4 ++++
 kernel/sched/cpufreq_schedutil.c | 10 ++++++++--
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index dbfb30809f15..e34b7cd5de73 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -739,6 +739,10 @@ DECLARE_TRACE(uclamp_update_tsk_tp,
 	TP_PROTO(struct task_struct *tsk, int uclamp_id,  unsigned int value),
 	TP_ARGS(tsk, uclamp_id, value));
 
+DECLARE_TRACE(schedutil_update_filtered_tp,
+	TP_PROTO(int cpu),
+	TP_ARGS(cpu));
+
 #endif /* _TRACE_SCHED_H */
 
 /* This part must be outside protection */
diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index f462496e5c07..4f9daf258a65 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -6,6 +6,8 @@
  * Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
  */
 
+EXPORT_TRACEPOINT_SYMBOL_GPL(schedutil_update_filtered_tp);
+
 #define IOWAIT_BOOST_MIN	(SCHED_CAPACITY_SCALE / 8)
 
 struct sugov_tunables {
@@ -318,8 +320,10 @@ static inline bool sugov_update_single_common(struct sugov_cpu *sg_cpu,
 
 	ignore_dl_rate_limit(sg_cpu);
 
-	if (!sugov_should_update_freq(sg_cpu->sg_policy, time))
+	if (!sugov_should_update_freq(sg_cpu->sg_policy, time)) {
+		trace_schedutil_update_filtered_tp(sg_cpu->cpu);
 		return false;
+	}
 
 	sugov_get_util(sg_cpu);
 	sugov_iowait_apply(sg_cpu, time, max_cap);
@@ -446,8 +450,10 @@ sugov_update_shared(struct update_util_data *hook, u64 time, unsigned int flags)
 
 	ignore_dl_rate_limit(sg_cpu);
 
-	if (!sugov_should_update_freq(sg_policy, time))
+	if (!sugov_should_update_freq(sg_policy, time)) {
+		trace_schedutil_update_filtered_tp(sg_cpu->cpu);
 		goto unlock;
+	}
 
 	next_f = sugov_next_freq_shared(sg_cpu, time);
 
-- 
2.25.1

Re: [PATCH v2 0/3] Add basic tracing for uclamp and schedutil

[Lukasz Luba]

Hi Rafael,



On 5/9/23 13:22, Lukasz Luba wrote:
> Hi all,
> 
> The task scheduler feature: Uclamp, begins to take off. To better understand
> the dynamics in the task scheduler and CPU frequency requests we need some
> better tracing.
> In schedutil (cpufreq governor) we allow to enter the scheduler
> and make the frequency change. Although, there is some limit in regards to how
> often this can happen. That min period is provided by the cpufreq driver.
> Thus, some of the cpufreq requests might be filter out and the frequency won't
> be changed (hopefuly will be set a bit later). We would like to know about
> those situations, especially in context of the user-space hints made via
> Uclamp for particular tasks.
> This patch set aims to add base for our toolkits and post-processing trace
> analyzes.
> 
> Changelog:
> v2:
> - solved the issue from CI build warning, dropped schedutil.h and re-used
>    the sched.h which is available in build_utility.c where cpufreq_schedutil.c
>    is included
> - added tag for the last patch 3/3 for the CI robot helping hend
> - re-based on top of v6.4-rc1
> v1:
> - implementation can be found here [1]
> 

I was going to gently ping you, while I've realized that you
are not on CC list :( I don't know what happened, my apologies.

Shell I resend this patch set so you can have it in a proper way
in your mailbox?

Could you have a look at this, please?

This is getting more attention, since in Android we have a
daemon which can now communicate with the kernel and send
those Uclamp values on behalf of an unprivileged app.

Regards,
Lukasz

Re: [PATCH v2 0/3] Add basic tracing for uclamp and schedutil

[Rafael J. Wysocki]

Hi Lukasz,

On Mon, May 22, 2023 at 3:38 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
>
> Hi Rafael,
>
>
>
> On 5/9/23 13:22, Lukasz Luba wrote:
> > Hi all,
> >
> > The task scheduler feature: Uclamp, begins to take off. To better understand
> > the dynamics in the task scheduler and CPU frequency requests we need some
> > better tracing.
> > In schedutil (cpufreq governor) we allow to enter the scheduler
> > and make the frequency change. Although, there is some limit in regards to how
> > often this can happen. That min period is provided by the cpufreq driver.
> > Thus, some of the cpufreq requests might be filter out and the frequency won't
> > be changed (hopefuly will be set a bit later). We would like to know about
> > those situations, especially in context of the user-space hints made via
> > Uclamp for particular tasks.
> > This patch set aims to add base for our toolkits and post-processing trace
> > analyzes.
>
> > Changelog:
> > v2:
> > - solved the issue from CI build warning, dropped schedutil.h and re-used
> >    the sched.h which is available in build_utility.c where cpufreq_schedutil.c
> >    is included
> > - added tag for the last patch 3/3 for the CI robot helping hend
> > - re-based on top of v6.4-rc1
> > v1:
> > - implementation can be found here [1]
> >
>
> I was going to gently ping you, while I've realized that you
> are not on CC list :( I don't know what happened, my apologies.

No worries.

> Shell I resend this patch set so you can have it in a proper way
> in your mailbox?

Well, for schedutil you should also CC linux-pm (done now), so please resend it.

> Could you have a look at this, please?

I could, but if I'm to reply, it will be much more convenient for me
if it is there in my inbox.

> This is getting more attention, since in Android we have a
> daemon which can now communicate with the kernel and send
> those Uclamp values on behalf of an unprivileged app.

Re: [PATCH v2 0/3] Add basic tracing for uclamp and schedutil

[Lukasz Luba]

On 5/22/23 15:40, Rafael J. Wysocki wrote:
> Hi Lukasz,
> 
> On Mon, May 22, 2023 at 3:38 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
>>
>> Hi Rafael,
>>
>>
>>
>> On 5/9/23 13:22, Lukasz Luba wrote:
>>> Hi all,
>>>
>>> The task scheduler feature: Uclamp, begins to take off. To better understand
>>> the dynamics in the task scheduler and CPU frequency requests we need some
>>> better tracing.
>>> In schedutil (cpufreq governor) we allow to enter the scheduler
>>> and make the frequency change. Although, there is some limit in regards to how
>>> often this can happen. That min period is provided by the cpufreq driver.
>>> Thus, some of the cpufreq requests might be filter out and the frequency won't
>>> be changed (hopefuly will be set a bit later). We would like to know about
>>> those situations, especially in context of the user-space hints made via
>>> Uclamp for particular tasks.
>>> This patch set aims to add base for our toolkits and post-processing trace
>>> analyzes.
>>
>>> Changelog:
>>> v2:
>>> - solved the issue from CI build warning, dropped schedutil.h and re-used
>>>     the sched.h which is available in build_utility.c where cpufreq_schedutil.c
>>>     is included
>>> - added tag for the last patch 3/3 for the CI robot helping hend
>>> - re-based on top of v6.4-rc1
>>> v1:
>>> - implementation can be found here [1]
>>>
>>
>> I was going to gently ping you, while I've realized that you
>> are not on CC list :( I don't know what happened, my apologies.
> 
> No worries.
> 
>> Shell I resend this patch set so you can have it in a proper way
>> in your mailbox?
> 
> Well, for schedutil you should also CC linux-pm (done now), so please resend it.
> 
>> Could you have a look at this, please?
> 
> I could, but if I'm to reply, it will be much more convenient for me
> if it is there in my inbox.
> 

Thanks Rafael for instant response. I'll resend it with the proper CC
list this time.