From: Vincent Guittot <vincent.guittot@linaro.org>
To: Lukasz Luba <lukasz.luba@arm.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>,
Sam Wu <wusamuel@google.com>,
Saravana Kannan <saravanak@google.com>,
Viresh Kumar <viresh.kumar@linaro.org>,
Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
"Isaac J . Manjarres" <isaacmanjarres@google.com>,
kernel-team@android.com,
"Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1] Revert "cpufreq: schedutil: Move max CPU capacity to sugov_policy"
Date: Wed, 30 Nov 2022 15:29:59 +0100 [thread overview]
Message-ID: <CAKfTPtA=7DkjADnNijLPDm_6hh9XkFjC9ZUVQ_5_NSU2Fn5pHQ@mail.gmail.com> (raw)
In-Reply-To: <75bba88a-0516-a6a2-d4e6-8cedabadf413@arm.com>
On Wed, 30 Nov 2022 at 15:04, Lukasz Luba <lukasz.luba@arm.com> wrote:
>
> Hi Vincent,
>
> On 11/30/22 10:42, Vincent Guittot wrote:
> > Hi All
> >
> > Just for the log and because it took me a while to figure out the root
> > cause of the problem: This patch also creates a regression for
> > snapdragon845 based systems and probably on any QC chipsets that use a
> > LUT to update the OPP table at boot. The behavior is the same as
> > described by Sam with a staled value in sugov_policy.max field.
>
> Thanks for sharing this info and apologies that you spent cycles
> on it.
>
> I have checked that whole setup code (capacity + cpufreq policy and
> governor). It looks like to have a proper capacity of CPUs, we need
> to wait till the last policy is created. It's due to the arch_topology.c
> mechanism which is only triggered after all CPUs' got the policy.
> Unfortunately, this leads to a chicken & egg situation for this
> schedutil setup of max capacity.
>
> I have experimented with this code, which triggers an update in
> the schedutil, when all CPUs got the policy and sugov gov
> (with trace_printk() to mach the output below)
Your proposal below looks similar to what is done in arch_topology.c.
arch_topology.c triggers a rebuild of sched_domain and removes its
cpufreq notifier cb once it has visited all CPUs, could it also
trigger an update of CPU's policy with cpufreq_update_policy() ?
At this point you will be sure that the normalization has happened and
the max capacity will not change.
I don't know if it's a global problem or only for systems using arch_topology
>
> -------------------------8<-----------------------------------------
> diff --git a/kernel/sched/cpufreq_schedutil.c
> b/kernel/sched/cpufreq_schedutil.c
> index 9161d1136d01..f1913a857218 100644
> --- a/kernel/sched/cpufreq_schedutil.c
> +++ b/kernel/sched/cpufreq_schedutil.c
> @@ -59,6 +59,7 @@ struct sugov_cpu {
> };
>
> static DEFINE_PER_CPU(struct sugov_cpu, sugov_cpu);
> +static cpumask_var_t cpus_to_visit;
>
> /************************ Governor internals ***********************/
>
> @@ -783,6 +784,22 @@ static int sugov_start(struct cpufreq_policy *policy)
>
> cpufreq_add_update_util_hook(cpu, &sg_cpu->update_util,
> uu);
> }
> +
> + cpumask_andnot(cpus_to_visit, cpus_to_visit, policy->related_cpus);
> +
> + if (cpumask_empty(cpus_to_visit)) {
> + trace_printk("schedutil the visit cpu mask is empty now\n");
> + for_each_possible_cpu(cpu) {
> + struct sugov_cpu *sg_cpu = &per_cpu(sugov_cpu, cpu);
> + struct sugov_policy *sg_policy = sg_cpu->sg_policy;
> +
> + sg_policy->max = arch_scale_cpu_capacity(cpu);
> +
> + trace_printk("SCHEDUTIL: NEW CPU%u
> cpu_capacity=%lu\n",
> + cpu, sg_policy->max);
> + }
> + }
> +
> return 0;
> }
>
> @@ -800,6 +817,8 @@ static void sugov_stop(struct cpufreq_policy *policy)
> irq_work_sync(&sg_policy->irq_work);
> kthread_cancel_work_sync(&sg_policy->work);
> }
> +
> + cpumask_or(cpus_to_visit, cpus_to_visit, policy->related_cpus);
> }
>
> static void sugov_limits(struct cpufreq_policy *policy)
> @@ -829,6 +848,11 @@ struct cpufreq_governor schedutil_gov = {
> #ifdef CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL
> struct cpufreq_governor *cpufreq_default_governor(void)
> {
> + if (!alloc_cpumask_var(&cpus_to_visit, GFP_KERNEL))
> + return NULL;
> +
> + cpumask_copy(cpus_to_visit, cpu_possible_mask);
> +
> return &schedutil_gov;
> }
> #endif
>
> ---------------------------------->8---------------------------------
>
>
> That simple approach fixes the issue. I have also tested it with
> governor change a few times and setting back the schedutil.
>
> -------------------------------------------
> kworker/u12:1-48 [004] ..... 2.208847: sugov_start:
> schedutil the visit cpu mask is empty now
> kworker/u12:1-48 [004] ..... 2.208854: sugov_start:
> SCHEDUTIL: NEW CPU0 cpu_capacity=381
> kworker/u12:1-48 [004] ..... 2.208857: sugov_start:
> SCHEDUTIL: NEW CPU1 cpu_capacity=381
> kworker/u12:1-48 [004] ..... 2.208860: sugov_start:
> SCHEDUTIL: NEW CPU2 cpu_capacity=381
> kworker/u12:1-48 [004] ..... 2.208862: sugov_start:
> SCHEDUTIL: NEW CPU3 cpu_capacity=381
> kworker/u12:1-48 [004] ..... 2.208864: sugov_start:
> SCHEDUTIL: NEW CPU4 cpu_capacity=1024
> kworker/u12:1-48 [004] ..... 2.208866: sugov_start:
> SCHEDUTIL: NEW CPU5 cpu_capacity=1024
> bash-615 [005] ..... 35.317113: sugov_start:
> schedutil the visit cpu mask is empty now
> bash-615 [005] ..... 35.317120: sugov_start:
> SCHEDUTIL: NEW CPU0 cpu_capacity=381
> bash-615 [005] ..... 35.317123: sugov_start:
> SCHEDUTIL: NEW CPU1 cpu_capacity=381
> bash-615 [005] ..... 35.317125: sugov_start:
> SCHEDUTIL: NEW CPU2 cpu_capacity=381
> bash-615 [005] ..... 35.317127: sugov_start:
> SCHEDUTIL: NEW CPU3 cpu_capacity=381
> bash-615 [005] ..... 35.317129: sugov_start:
> SCHEDUTIL: NEW CPU4 cpu_capacity=1024
> bash-615 [005] ..... 35.317131: sugov_start:
> SCHEDUTIL: NEW CPU5 cpu_capacity=1024
> bash-623 [003] ..... 57.633328: sugov_start:
> schedutil the visit cpu mask is empty now
> bash-623 [003] ..... 57.633336: sugov_start:
> SCHEDUTIL: NEW CPU0 cpu_capacity=381
> bash-623 [003] ..... 57.633339: sugov_start:
> SCHEDUTIL: NEW CPU1 cpu_capacity=381
> bash-623 [003] ..... 57.633340: sugov_start:
> SCHEDUTIL: NEW CPU2 cpu_capacity=381
> bash-623 [003] ..... 57.633342: sugov_start:
> SCHEDUTIL: NEW CPU3 cpu_capacity=381
> bash-623 [003] ..... 57.633343: sugov_start:
> SCHEDUTIL: NEW CPU4 cpu_capacity=1024
> bash-623 [003] ..... 57.633344: sugov_start:
> SCHEDUTIL: NEW CPU5 cpu_capacity=1024
> ----------------------------------------------------
>
> It should work.
>
> Regards,
> Lukasz
next prev parent reply other threads:[~2022-11-30 14:30 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-10 19:57 [PATCH v1] Revert "cpufreq: schedutil: Move max CPU capacity to sugov_policy" Sam Wu
2022-11-15 22:35 ` Saravana Kannan
2022-11-16 11:43 ` Lukasz Luba
2022-11-16 12:17 ` Rafael J. Wysocki
2022-11-18 1:00 ` Sam Wu
2022-11-21 19:18 ` Rafael J. Wysocki
2022-11-22 8:58 ` Lukasz Luba
2022-11-30 10:42 ` Vincent Guittot
2022-11-30 14:04 ` Lukasz Luba
2022-11-30 14:29 ` Vincent Guittot [this message]
2022-11-30 15:00 ` Lukasz Luba
2022-11-16 11:35 ` Lukasz Luba
2022-11-20 18:07 ` [PATCH v1] Revert "cpufreq: schedutil: Move max CPU capacity to sugov_policy" #forregzbot Thorsten Leemhuis
2022-11-27 12:06 ` Thorsten Leemhuis
2022-12-05 9:18 ` [PATCH v1] Revert "cpufreq: schedutil: Move max CPU capacity to sugov_policy" Viresh Kumar
2022-12-06 8:17 ` Lukasz Luba
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAKfTPtA=7DkjADnNijLPDm_6hh9XkFjC9ZUVQ_5_NSU2Fn5pHQ@mail.gmail.com' \
--to=vincent.guittot@linaro.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=isaacmanjarres@google.com \
--cc=juri.lelli@redhat.com \
--cc=kernel-team@android.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=lukasz.luba@arm.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rafael.j.wysocki@intel.com \
--cc=rafael@kernel.org \
--cc=rostedt@goodmis.org \
--cc=saravanak@google.com \
--cc=viresh.kumar@linaro.org \
--cc=vschneid@redhat.com \
--cc=wusamuel@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).