From: "Rafael J. Wysocki" <rjw@rjwysocki.net>
To: Linux PM list <linux-pm@vger.kernel.org>
Cc: Juri Lelli <juri.lelli@arm.com>,
Steve Muckle <steve.muckle@linaro.org>,
ACPI Devel Maling List <linux-acpi@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
Viresh Kumar <viresh.kumar@linaro.org>,
Vincent Guittot <vincent.guittot@linaro.org>,
Michael Turquette <mturquette@baylibre.com>,
Ingo Molnar <mingo@kernel.org>
Subject: [PATCH v2 1/10] cpufreq: Reduce cpufreq_update_util() overhead a bit
Date: Fri, 04 Mar 2016 03:58:22 +0100 [thread overview]
Message-ID: <1988425.XTpZIAJINa@vostro.rjw.lan> (raw)
In-Reply-To: <2409306.qzzMXcm4dm@vostro.rjw.lan>
From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Use the observation that cpufreq_update_util() is only called
by the scheduler with rq->lock held, so the callers of
cpufreq_set_update_util_data() can use synchronize_sched()
instead of synchronize_rcu() to wait for cpufreq_update_util()
to complete. Moreover, if they are updated to do that,
rcu_read_(un)lock() calls in cpufreq_update_util() might be
replaced with rcu_read_(un)lock_sched(), respectively, but
those aren't really necessary, because the scheduler calls
that function from RCU-sched read-side critical sections
already.
In addition to that, if cpufreq_set_update_util_data() checks
the func field in the struct update_util_data before setting
the per-CPU pointer to it, the data->func check may be dropped
from cpufreq_update_util() as well.
Make the above changes to reduce the overhead from
cpufreq_update_util() in the scheduler paths invoking it
and to make the cleanup after removing its callbacks less
heavy-weight somewhat.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
---
Changes from the previous version:
- Use rcu_dereference_sched() in cpufreq_update_util().
---
drivers/cpufreq/cpufreq.c | 25 +++++++++++++++++--------
drivers/cpufreq/cpufreq_governor.c | 2 +-
drivers/cpufreq/intel_pstate.c | 4 ++--
3 files changed, 20 insertions(+), 11 deletions(-)
Index: linux-pm/drivers/cpufreq/cpufreq.c
===================================================================
--- linux-pm.orig/drivers/cpufreq/cpufreq.c
+++ linux-pm/drivers/cpufreq/cpufreq.c
@@ -77,12 +77,15 @@ static DEFINE_PER_CPU(struct update_util
* to call from cpufreq_update_util(). That function will be called from an RCU
* read-side critical section, so it must not sleep.
*
- * Callers must use RCU callbacks to free any memory that might be accessed
- * via the old update_util_data pointer or invoke synchronize_rcu() right after
- * this function to avoid use-after-free.
+ * Callers must use RCU-sched callbacks to free any memory that might be
+ * accessed via the old update_util_data pointer or invoke synchronize_sched()
+ * right after this function to avoid use-after-free.
*/
void cpufreq_set_update_util_data(int cpu, struct update_util_data *data)
{
+ if (WARN_ON(data && !data->func))
+ return;
+
rcu_assign_pointer(per_cpu(cpufreq_update_util_data, cpu), data);
}
EXPORT_SYMBOL_GPL(cpufreq_set_update_util_data);
@@ -95,18 +98,24 @@ EXPORT_SYMBOL_GPL(cpufreq_set_update_uti
*
* This function is called by the scheduler on every invocation of
* update_load_avg() on the CPU whose utilization is being updated.
+ *
+ * It can only be called from RCU-sched read-side critical sections.
*/
void cpufreq_update_util(u64 time, unsigned long util, unsigned long max)
{
struct update_util_data *data;
- rcu_read_lock();
+#ifdef CONFIG_LOCKDEP
+ WARN_ON(debug_locks && !rcu_read_lock_sched_held());
+#endif
- data = rcu_dereference(*this_cpu_ptr(&cpufreq_update_util_data));
- if (data && data->func)
+ data = rcu_dereference_sched(*this_cpu_ptr(&cpufreq_update_util_data));
+ /*
+ * If this isn't inside of an RCU-sched read-side critical section, data
+ * may become NULL after the check below.
+ */
+ if (data)
data->func(data, time, util, max);
-
- rcu_read_unlock();
}
/* Flag to suspend/resume CPUFreq governors */
Index: linux-pm/drivers/cpufreq/cpufreq_governor.c
===================================================================
--- linux-pm.orig/drivers/cpufreq/cpufreq_governor.c
+++ linux-pm/drivers/cpufreq/cpufreq_governor.c
@@ -280,7 +280,7 @@ static inline void gov_clear_update_util
for_each_cpu(i, policy->cpus)
cpufreq_set_update_util_data(i, NULL);
- synchronize_rcu();
+ synchronize_sched();
}
static void gov_cancel_work(struct cpufreq_policy *policy)
Index: linux-pm/drivers/cpufreq/intel_pstate.c
===================================================================
--- linux-pm.orig/drivers/cpufreq/intel_pstate.c
+++ linux-pm/drivers/cpufreq/intel_pstate.c
@@ -1174,7 +1174,7 @@ static void intel_pstate_stop_cpu(struct
pr_debug("intel_pstate: CPU %d exiting\n", cpu_num);
cpufreq_set_update_util_data(cpu_num, NULL);
- synchronize_rcu();
+ synchronize_sched();
if (hwp_active)
return;
@@ -1442,7 +1442,7 @@ out:
for_each_online_cpu(cpu) {
if (all_cpu_data[cpu]) {
cpufreq_set_update_util_data(cpu, NULL);
- synchronize_rcu();
+ synchronize_sched();
kfree(all_cpu_data[cpu]);
}
}
next prev parent reply other threads:[~2016-03-04 3:36 UTC|newest]
Thread overview: 158+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-02 1:56 [PATCH 0/6] cpufreq: schedutil governor Rafael J. Wysocki
2016-03-02 2:04 ` [PATCH 1/6] cpufreq: Reduce cpufreq_update_util() overhead a bit Rafael J. Wysocki
2016-03-03 5:48 ` Viresh Kumar
2016-03-03 11:47 ` Juri Lelli
2016-03-03 13:04 ` Peter Zijlstra
2016-03-02 2:05 ` [PATCH 2/6][Resend] cpufreq: acpi-cpufreq: Make read and write operations more efficient Rafael J. Wysocki
2016-03-02 2:08 ` [PATCH 3/6] cpufreq: governor: New data type for management part of dbs_data Rafael J. Wysocki
2016-03-03 5:53 ` Viresh Kumar
2016-03-03 19:26 ` Rafael J. Wysocki
2016-03-04 5:49 ` Viresh Kumar
2016-03-02 2:10 ` [PATCH 4/6] cpufreq: governor: Move abstract gov_tunables code to a seperate file Rafael J. Wysocki
2016-03-03 6:03 ` Viresh Kumar
2016-03-02 2:12 ` [PATCH 5/6] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-03 6:00 ` Viresh Kumar
2016-03-04 2:15 ` Rafael J. Wysocki
2016-03-03 11:16 ` Peter Zijlstra
2016-03-03 20:56 ` Rafael J. Wysocki
2016-03-03 21:12 ` Peter Zijlstra
2016-03-03 11:18 ` Peter Zijlstra
2016-03-03 19:39 ` Rafael J. Wysocki
2016-03-02 2:27 ` [PATCH 6/6] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-02 17:10 ` Vincent Guittot
2016-03-02 17:58 ` Rafael J. Wysocki
2016-03-02 22:49 ` Rafael J. Wysocki
2016-03-03 12:20 ` Peter Zijlstra
2016-03-03 12:32 ` Juri Lelli
2016-03-03 16:24 ` Rafael J. Wysocki
2016-03-03 16:37 ` Peter Zijlstra
2016-03-03 16:47 ` Peter Zijlstra
2016-03-04 1:14 ` Rafael J. Wysocki
2016-03-03 16:55 ` Juri Lelli
2016-03-03 16:56 ` Peter Zijlstra
2016-03-03 17:14 ` Juri Lelli
2016-03-03 14:01 ` Vincent Guittot
2016-03-03 15:38 ` Peter Zijlstra
2016-03-03 16:28 ` Peter Zijlstra
2016-03-03 16:42 ` Peter Zijlstra
2016-03-03 17:28 ` Dietmar Eggemann
2016-03-03 18:26 ` Peter Zijlstra
2016-03-03 19:14 ` Dietmar Eggemann
2016-03-08 13:09 ` Peter Zijlstra
2016-03-03 18:58 ` Rafael J. Wysocki
2016-03-03 13:07 ` Vincent Guittot
2016-03-03 20:06 ` Steve Muckle
2016-03-03 20:20 ` Rafael J. Wysocki
2016-03-03 21:37 ` Steve Muckle
2016-03-07 2:41 ` Rafael J. Wysocki
2016-03-08 11:27 ` Peter Zijlstra
2016-03-08 18:00 ` Rafael J. Wysocki
2016-03-08 19:26 ` Peter Zijlstra
2016-03-08 20:05 ` Rafael J. Wysocki
2016-03-09 10:15 ` Juri Lelli
2016-03-09 23:41 ` Rafael J. Wysocki
2016-03-10 4:30 ` Juri Lelli
2016-03-10 21:01 ` Rafael J. Wysocki
2016-03-10 23:19 ` Michael Turquette
2016-03-09 16:39 ` Peter Zijlstra
2016-03-09 23:28 ` Rafael J. Wysocki
2016-03-10 3:44 ` Vincent Guittot
2016-03-10 10:07 ` Peter Zijlstra
2016-03-10 10:26 ` Vincent Guittot
[not found] ` <CAKfTPtCbjgbJn+68NJPCnmPFtcHD0wGmZRYaw37zSqPXNpo_Uw@mail.gmail.com>
2016-03-10 10:30 ` Peter Zijlstra
2016-03-10 10:56 ` Peter Zijlstra
2016-03-10 22:28 ` Rafael J. Wysocki
2016-03-10 8:43 ` Peter Zijlstra
2016-03-04 2:56 ` [PATCH v2 0/10] cpufreq: schedutil governor Rafael J. Wysocki
2016-03-04 2:58 ` Rafael J. Wysocki [this message]
2016-03-09 12:39 ` [PATCH v2 1/10] cpufreq: Reduce cpufreq_update_util() overhead a bit Peter Zijlstra
2016-03-09 14:17 ` Rafael J. Wysocki
2016-03-09 15:29 ` Peter Zijlstra
2016-03-09 21:35 ` Rafael J. Wysocki
2016-03-10 9:19 ` Peter Zijlstra
2016-03-04 2:59 ` [PATCH v2 2/10][Resend] cpufreq: acpi-cpufreq: Make read and write operations more efficient Rafael J. Wysocki
2016-03-04 3:01 ` [PATCH v2 3/10] cpufreq: governor: New data type for management part of dbs_data Rafael J. Wysocki
2016-03-04 5:52 ` Viresh Kumar
2016-03-04 3:03 ` [PATCH v2 4/10] cpufreq: governor: Move abstract gov_attr_set code to seperate file Rafael J. Wysocki
2016-03-04 5:52 ` Viresh Kumar
2016-03-04 3:05 ` [PATCH v2 5/10] cpufreq: Move governor attribute set headers to cpufreq.h Rafael J. Wysocki
2016-03-04 5:53 ` Viresh Kumar
2016-03-04 3:07 ` [PATCH v2 6/10] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-04 22:18 ` Steve Muckle
2016-03-04 22:32 ` Rafael J. Wysocki
2016-03-04 22:40 ` Rafael J. Wysocki
2016-03-04 23:18 ` Rafael J. Wysocki
2016-03-04 23:56 ` Steve Muckle
2016-03-05 0:18 ` Rafael J. Wysocki
2016-03-05 11:58 ` Ingo Molnar
2016-03-05 16:49 ` Peter Zijlstra
2016-03-06 2:17 ` Rafael J. Wysocki
2016-03-07 8:00 ` Peter Zijlstra
2016-03-07 13:15 ` Rafael J. Wysocki
2016-03-07 13:32 ` Peter Zijlstra
2016-03-07 13:42 ` Rafael J. Wysocki
2016-03-04 22:58 ` Rafael J. Wysocki
2016-03-04 23:59 ` Steve Muckle
2016-03-04 3:12 ` [PATCH v2 7/10] cpufreq: Rework the scheduler hooks for triggering updates Rafael J. Wysocki
2016-03-04 3:14 ` [PATCH v2 8/10] cpufreq: Move scheduler-related code to the sched directory Rafael J. Wysocki
2016-03-04 3:18 ` [PATCH v2 9/10] cpufreq: sched: Re-introduce cpufreq_update_util() Rafael J. Wysocki
2016-03-04 10:50 ` Juri Lelli
2016-03-04 12:58 ` Rafael J. Wysocki
2016-03-04 13:30 ` [PATCH v3 " Rafael J. Wysocki
2016-03-04 21:21 ` Steve Muckle
2016-03-04 21:27 ` Rafael J. Wysocki
2016-03-04 21:36 ` Rafael J. Wysocki
2016-03-04 3:35 ` [PATCH v2 10/10] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-04 11:26 ` Juri Lelli
2016-03-04 13:19 ` Rafael J. Wysocki
2016-03-04 15:56 ` Srinivas Pandruvada
2016-03-08 2:23 ` [PATCH v3 0/7] cpufreq: schedutil governor Rafael J. Wysocki
2016-03-08 2:25 ` [PATCH v3 1/7][Resend] cpufreq: Rework the scheduler hooks for triggering updates Rafael J. Wysocki
2016-03-09 13:41 ` Peter Zijlstra
2016-03-09 14:02 ` Rafael J. Wysocki
2016-03-08 2:26 ` [PATCH v3 2/7][Resend] cpufreq: Move scheduler-related code to the sched directory Rafael J. Wysocki
2016-03-08 2:28 ` [PATCH v3 3/7][Resend] cpufreq: governor: New data type for management part of dbs_data Rafael J. Wysocki
2016-03-08 2:29 ` [PATCH v3 4/7][Resend] cpufreq: governor: Move abstract gov_attr_set code to seperate file Rafael J. Wysocki
2016-03-08 2:38 ` [PATCH v3 5/7] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-08 2:41 ` [PATCH v3 6/7] cpufreq: sched: Re-introduce cpufreq_update_util() Rafael J. Wysocki
2016-03-08 2:50 ` [PATCH v3 7/7] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-16 14:41 ` [PATCH v4 0/7] cpufreq: schedutil governor Rafael J. Wysocki
2016-03-16 14:43 ` [PATCH v4 1/7] cpufreq: sched: Helpers to add and remove update_util hooks Rafael J. Wysocki
2016-03-16 14:44 ` [PATCH v4 2/7] cpufreq: governor: New data type for management part of dbs_data Rafael J. Wysocki
2016-03-16 14:45 ` [PATCH v4 3/7] cpufreq: governor: Move abstract gov_attr_set code to seperate file Rafael J. Wysocki
2016-03-16 14:46 ` [PATCH v4 4/7] cpufreq: Move governor attribute set headers to cpufreq.h Rafael J. Wysocki
2016-03-16 14:47 ` [PATCH v4 5/7] cpufreq: Move governor symbols " Rafael J. Wysocki
2016-03-16 14:52 ` [PATCH v4 6/7] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-16 15:35 ` Peter Zijlstra
2016-03-16 16:58 ` Rafael J. Wysocki
2016-03-16 15:43 ` Peter Zijlstra
2016-03-16 16:58 ` Rafael J. Wysocki
2016-03-16 14:59 ` [PATCH v4 7/7] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-16 17:35 ` Peter Zijlstra
2016-03-16 21:42 ` Rafael J. Wysocki
2016-03-16 17:36 ` Peter Zijlstra
2016-03-16 21:34 ` Rafael J. Wysocki
2016-03-16 17:52 ` Peter Zijlstra
2016-03-16 21:38 ` Rafael J. Wysocki
2016-03-16 22:39 ` Peter Zijlstra
2016-03-16 17:53 ` Peter Zijlstra
2016-03-16 21:48 ` Rafael J. Wysocki
2016-03-16 18:14 ` Peter Zijlstra
2016-03-16 21:38 ` Rafael J. Wysocki
2016-03-16 22:40 ` Peter Zijlstra
2016-03-16 22:53 ` Rafael J. Wysocki
2016-03-16 15:27 ` [PATCH v4 0/7] cpufreq: schedutil governor Peter Zijlstra
2016-03-16 16:20 ` Rafael J. Wysocki
2016-03-16 23:51 ` [PATCH v5 6/7][Update] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-17 11:35 ` Juri Lelli
2016-03-17 11:40 ` Peter Zijlstra
2016-03-17 11:48 ` Juri Lelli
2016-03-17 12:53 ` Rafael J. Wysocki
2016-03-17 0:01 ` [PATCH v5 7/7][Update] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-17 11:30 ` Juri Lelli
2016-03-17 12:54 ` Rafael J. Wysocki
2016-03-17 11:36 ` Peter Zijlstra
2016-03-17 12:54 ` Rafael J. Wysocki
2016-03-17 15:54 ` [PATCH v6 6/7][Update] cpufreq: Support for fast frequency switching Rafael J. Wysocki
2016-03-17 16:01 ` [PATCH v6 7/7][Update] cpufreq: schedutil: New governor based on scheduler utilization data Rafael J. Wysocki
2016-03-18 12:34 ` Patrick Bellasi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1988425.XTpZIAJINa@vostro.rjw.lan \
--to=rjw@rjwysocki.net \
--cc=juri.lelli@arm.com \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=mturquette@baylibre.com \
--cc=peterz@infradead.org \
--cc=srinivas.pandruvada@linux.intel.com \
--cc=steve.muckle@linaro.org \
--cc=vincent.guittot@linaro.org \
--cc=viresh.kumar@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).