From: tip-bot for Thomas Gleixner <tipbot@zytor.com>
To: linux-tip-commits@vger.kernel.org
Cc: bigeasy@linutronix.de, rostedt@goodmis.org, peterz@infradead.org,
linux-kernel@vger.kernel.org, mingo@kernel.org, hpa@zytor.com,
tglx@linutronix.de
Subject: [tip:smp/hotplug] cpu/hotplug: Convert hotplug locking to percpu rwsem
Date: Thu, 20 Apr 2017 04:30:37 -0700 [thread overview]
Message-ID: <tip-d215aab82d81974f438bfbc80aa437132f3c37c3@git.kernel.org> (raw)
In-Reply-To: <20170418170554.382344438@linutronix.de>
Commit-ID: d215aab82d81974f438bfbc80aa437132f3c37c3
Gitweb: http://git.kernel.org/tip/d215aab82d81974f438bfbc80aa437132f3c37c3
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Tue, 18 Apr 2017 19:05:06 +0200
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitDate: Thu, 20 Apr 2017 13:08:58 +0200
cpu/hotplug: Convert hotplug locking to percpu rwsem
There are no more (known) nested calls to get_online_cpus() so it's
possible to remove the nested call magic and convert the mutex to a
percpu-rwsem, which speeds up get/put_online_cpus() significantly for the
uncontended case.
The contended case (write locked for hotplug operations) is slow anyway, so
the slightly more expensive down_write of the percpu rwsem does not matter.
[ peterz: Add lockdep assertions ]
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Siewior <bigeasy@linutronix.de>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/r/20170418170554.382344438@linutronix.de
---
include/linux/cpu.h | 2 +
kernel/cpu.c | 110 ++++++++------------------------------------------
kernel/jump_label.c | 2 +
kernel/padata.c | 1 +
kernel/stop_machine.c | 2 +
5 files changed, 23 insertions(+), 94 deletions(-)
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index f920812..83010c3 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -105,6 +105,7 @@ extern void cpu_hotplug_begin(void);
extern void cpu_hotplug_done(void);
extern void get_online_cpus(void);
extern void put_online_cpus(void);
+extern void lockdep_assert_hotplug_held(void);
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
void clear_tasks_mm_cpumask(int cpu);
@@ -118,6 +119,7 @@ static inline void cpu_hotplug_done(void) {}
#define put_online_cpus() do { } while (0)
#define cpu_hotplug_disable() do { } while (0)
#define cpu_hotplug_enable() do { } while (0)
+static inline void lockdep_assert_hotplug_held(void) {}
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_PM_SLEEP_SMP
diff --git a/kernel/cpu.c b/kernel/cpu.c
index f932e68..05341f7 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -27,6 +27,7 @@
#include <linux/smpboot.h>
#include <linux/relay.h>
#include <linux/slab.h>
+#include <linux/percpu-rwsem.h>
#include <trace/events/power.h>
#define CREATE_TRACE_POINTS
@@ -196,121 +197,41 @@ void cpu_maps_update_done(void)
mutex_unlock(&cpu_add_remove_lock);
}
-/* If set, cpu_up and cpu_down will return -EBUSY and do nothing.
+/*
+ * If set, cpu_up and cpu_down will return -EBUSY and do nothing.
* Should always be manipulated under cpu_add_remove_lock
*/
static int cpu_hotplug_disabled;
#ifdef CONFIG_HOTPLUG_CPU
-static struct {
- struct task_struct *active_writer;
- /* wait queue to wake up the active_writer */
- wait_queue_head_t wq;
- /* verifies that no writer will get active while readers are active */
- struct mutex lock;
- /*
- * Also blocks the new readers during
- * an ongoing cpu hotplug operation.
- */
- atomic_t refcount;
-
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- struct lockdep_map dep_map;
-#endif
-} cpu_hotplug = {
- .active_writer = NULL,
- .wq = __WAIT_QUEUE_HEAD_INITIALIZER(cpu_hotplug.wq),
- .lock = __MUTEX_INITIALIZER(cpu_hotplug.lock),
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- .dep_map = STATIC_LOCKDEP_MAP_INIT("cpu_hotplug.dep_map", &cpu_hotplug.dep_map),
-#endif
-};
-
-/* Lockdep annotations for get/put_online_cpus() and cpu_hotplug_begin/end() */
-#define cpuhp_lock_acquire_read() lock_map_acquire_read(&cpu_hotplug.dep_map)
-#define cpuhp_lock_acquire_tryread() \
- lock_map_acquire_tryread(&cpu_hotplug.dep_map)
-#define cpuhp_lock_acquire() lock_map_acquire(&cpu_hotplug.dep_map)
-#define cpuhp_lock_release() lock_map_release(&cpu_hotplug.dep_map)
-
+DEFINE_STATIC_PERCPU_RWSEM(cpu_hotplug_lock);
void get_online_cpus(void)
{
- might_sleep();
- if (cpu_hotplug.active_writer == current)
- return;
- cpuhp_lock_acquire_read();
- mutex_lock(&cpu_hotplug.lock);
- atomic_inc(&cpu_hotplug.refcount);
- mutex_unlock(&cpu_hotplug.lock);
+ percpu_down_read(&cpu_hotplug_lock);
}
EXPORT_SYMBOL_GPL(get_online_cpus);
void put_online_cpus(void)
{
- int refcount;
-
- if (cpu_hotplug.active_writer == current)
- return;
-
- refcount = atomic_dec_return(&cpu_hotplug.refcount);
- if (WARN_ON(refcount < 0)) /* try to fix things up */
- atomic_inc(&cpu_hotplug.refcount);
-
- if (refcount <= 0 && waitqueue_active(&cpu_hotplug.wq))
- wake_up(&cpu_hotplug.wq);
-
- cpuhp_lock_release();
-
+ percpu_up_read(&cpu_hotplug_lock);
}
EXPORT_SYMBOL_GPL(put_online_cpus);
-/*
- * This ensures that the hotplug operation can begin only when the
- * refcount goes to zero.
- *
- * Note that during a cpu-hotplug operation, the new readers, if any,
- * will be blocked by the cpu_hotplug.lock
- *
- * Since cpu_hotplug_begin() is always called after invoking
- * cpu_maps_update_begin(), we can be sure that only one writer is active.
- *
- * Note that theoretically, there is a possibility of a livelock:
- * - Refcount goes to zero, last reader wakes up the sleeping
- * writer.
- * - Last reader unlocks the cpu_hotplug.lock.
- * - A new reader arrives at this moment, bumps up the refcount.
- * - The writer acquires the cpu_hotplug.lock finds the refcount
- * non zero and goes to sleep again.
- *
- * However, this is very difficult to achieve in practice since
- * get_online_cpus() not an api which is called all that often.
- *
- */
void cpu_hotplug_begin(void)
{
- DEFINE_WAIT(wait);
-
- cpu_hotplug.active_writer = current;
- cpuhp_lock_acquire();
-
- for (;;) {
- mutex_lock(&cpu_hotplug.lock);
- prepare_to_wait(&cpu_hotplug.wq, &wait, TASK_UNINTERRUPTIBLE);
- if (likely(!atomic_read(&cpu_hotplug.refcount)))
- break;
- mutex_unlock(&cpu_hotplug.lock);
- schedule();
- }
- finish_wait(&cpu_hotplug.wq, &wait);
+ percpu_down_write(&cpu_hotplug_lock);
}
void cpu_hotplug_done(void)
{
- cpu_hotplug.active_writer = NULL;
- mutex_unlock(&cpu_hotplug.lock);
- cpuhp_lock_release();
+ percpu_up_write(&cpu_hotplug_lock);
+}
+
+void lockdep_assert_hotplug_held(void)
+{
+ percpu_rwsem_assert_held(&cpu_hotplug_lock);
}
/*
@@ -344,8 +265,6 @@ void cpu_hotplug_enable(void)
EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
#endif /* CONFIG_HOTPLUG_CPU */
-/* Notifier wrappers for transitioning to state machine */
-
static int bringup_wait_for_ap(unsigned int cpu)
{
struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
@@ -1482,6 +1401,8 @@ int __cpuhp_setup_state_cpuslocked(enum cpuhp_state state,
int cpu, ret = 0;
bool dynstate;
+ lockdep_assert_hotplug_held();
+
if (cpuhp_cb_check(state) || !name)
return -EINVAL;
@@ -1600,6 +1521,7 @@ void __cpuhp_remove_state_cpuslocked(enum cpuhp_state state, bool invoke)
int cpu;
BUG_ON(cpuhp_cb_check(state));
+ lockdep_assert_hotplug_held();
mutex_lock(&cpuhp_state_mutex);
if (sp->multi_instance) {
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 308b12e..9455d2b 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -130,6 +130,7 @@ void __static_key_slow_inc(struct static_key *key)
* the all CPUs, for that to be serialized against CPU hot-plug
* we need to avoid CPUs coming online.
*/
+ lockdep_assert_hotplug_held();
jump_label_lock();
if (atomic_read(&key->enabled) == 0) {
atomic_set(&key->enabled, -1);
@@ -158,6 +159,7 @@ EXPORT_SYMBOL_GPL(static_key_slow_inc_cpuslocked);
static void __static_key_slow_dec(struct static_key *key,
unsigned long rate_limit, struct delayed_work *work)
{
+ lockdep_assert_hotplug_held();
/*
* The negative count check is valid even when a negative
* key->enabled is in use by static_key_slow_inc(); a
diff --git a/kernel/padata.c b/kernel/padata.c
index e5ff511..dc74ddba 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -1013,6 +1013,7 @@ err:
*/
struct padata_instance *padata_alloc_possible(struct workqueue_struct *wq)
{
+ lockdep_assert_hotplug_held();
return padata_alloc(wq, cpu_possible_mask, cpu_possible_mask);
}
EXPORT_SYMBOL(padata_alloc_possible);
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index a8c5636..2390340 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -562,6 +562,8 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
.active_cpus = cpus,
};
+ lockdep_assert_hotplug_held();
+
if (!stop_machine_initialized) {
/*
* Handle the case where stop_machine() is called
next prev parent reply other threads:[~2017-04-20 11:33 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-04-18 17:04 [patch V2 00/24] cpu/hotplug: Convert get_online_cpus() to a percpu_rwsem Thomas Gleixner
2017-04-18 17:04 ` [patch V2 01/24] cpu/hotplug: Provide cpuhp_setup/remove_state[_nocalls]_cpuslocked() Thomas Gleixner
2017-04-20 11:18 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 02/24] stop_machine: Provide stop_machine_cpuslocked() Thomas Gleixner
2017-04-20 11:19 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 03/24] padata: Make padata_alloc() static Thomas Gleixner
2017-04-20 11:19 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:04 ` [patch V2 04/24] padata: Avoid nested calls to get_online_cpus() in pcrypt_init_padata() Thomas Gleixner
2017-04-20 11:20 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 05/24] x86/mtrr: Remove get_online_cpus() from mtrr_save_state() Thomas Gleixner
2017-04-20 11:20 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 06/24] cpufreq: Use cpuhp_setup_state_nocalls_cpuslocked() Thomas Gleixner
2017-04-20 11:21 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 07/24] KVM/PPC/Book3S HV: " Thomas Gleixner
2017-04-20 11:21 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 08/24] hwtracing/coresight-etm3x: " Thomas Gleixner
2017-04-20 11:22 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-20 15:14 ` [patch V2 08/24] " Mathieu Poirier
2017-04-20 15:32 ` Mathieu Poirier
2017-04-18 17:04 ` [patch V2 09/24] hwtracing/coresight-etm4x: " Thomas Gleixner
2017-04-20 11:22 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 10/24] perf/x86/intel/cqm: Use cpuhp_setup_state_cpuslocked() Thomas Gleixner
2017-04-20 11:23 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 11/24] ARM/hw_breakpoint: " Thomas Gleixner
2017-04-19 17:54 ` Mark Rutland
2017-04-19 18:20 ` Thomas Gleixner
2017-04-20 11:23 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 12/24] s390/kernel: Use stop_machine_cpuslocked() Thomas Gleixner
2017-04-20 11:24 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 13/24] powerpc/powernv: " Thomas Gleixner
2017-04-20 11:24 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 14/24] cpu/hotplug: Use stop_machine_cpuslocked() in takedown_cpu() Thomas Gleixner
2017-04-20 11:25 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 15/24] x86/perf: Drop EXPORT of perf_check_microcode Thomas Gleixner
2017-04-20 11:25 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:04 ` [patch V2 16/24] perf/x86/intel: Drop get_online_cpus() in intel_snb_check_microcode() Thomas Gleixner
2017-04-20 11:26 ` [tip:smp/hotplug] " tip-bot for Sebastian Andrzej Siewior
2017-04-18 17:04 ` [patch V2 17/24] PCI: Use cpu_hotplug_disable() instead of get_online_cpus() Thomas Gleixner
2017-04-20 11:27 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:05 ` [patch V2 18/24] PCI: Replace the racy recursion prevention Thomas Gleixner
2017-04-20 11:27 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:05 ` [patch V2 19/24] ACPI/processor: Use cpu_hotplug_disable() instead of get_online_cpus() Thomas Gleixner
2017-04-20 11:28 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:05 ` [patch V2 20/24] perf/core: Remove redundant get_online_cpus() Thomas Gleixner
2017-04-20 11:28 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
2017-04-18 17:05 ` [patch V2 21/24] jump_label: Pull get_online_cpus() into generic code Thomas Gleixner
2017-04-18 17:05 ` [patch V2 22/24] jump_label: Provide static_key_slow_inc_cpuslocked() Thomas Gleixner
2017-04-18 17:05 ` [patch V2 23/24] perf: Avoid cpu_hotplug_lock r-r recursion Thomas Gleixner
2017-04-18 17:05 ` [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Thomas Gleixner
2017-04-20 11:30 ` tip-bot for Thomas Gleixner [this message]
2017-05-10 4:59 ` Michael Ellerman
2017-05-10 8:49 ` Thomas Gleixner
2017-05-10 16:30 ` Steven Rostedt
2017-05-10 17:15 ` Steven Rostedt
2017-05-11 5:49 ` Michael Ellerman
2017-04-25 16:10 ` [patch V2 00/24] cpu/hotplug: Convert get_online_cpus() to a percpu_rwsem Mark Rutland
2017-04-25 17:28 ` Sebastian Siewior
2017-04-26 8:59 ` Mark Rutland
2017-04-26 9:40 ` Suzuki K Poulose
2017-04-26 10:32 ` Mark Rutland
2017-04-27 8:27 ` Sebastian Siewior
2017-04-27 9:57 ` Mark Rutland
2017-04-27 10:01 ` Thomas Gleixner
2017-04-27 12:30 ` Mark Rutland
2017-04-27 15:48 ` [PATCH] arm64: cpufeature: use static_branch_enable_cpuslocked() (was: Re: [patch V2 00/24] cpu/hotplug: Convert get_online_cpus() to a percpu_rwsem) Mark Rutland
2017-04-27 16:35 ` Suzuki K Poulose
2017-04-27 17:03 ` [PATCH] arm64: cpufeature: use static_branch_enable_cpuslocked() Suzuki K Poulose
2017-04-27 17:17 ` Mark Rutland
2017-04-28 14:24 ` [RFC PATCH] trace/perf: cure locking issue in perf_event_open() error path Sebastian Siewior
2017-04-28 14:27 ` Sebastian Siewior
2017-05-01 12:57 ` [tip:smp/hotplug] perf: Reorder cpu hotplug rwsem against cred_guard_mutex tip-bot for Thomas Gleixner
2017-05-01 12:58 ` [tip:smp/hotplug] perf: Push hotplug protection down to callers tip-bot for Thomas Gleixner
2017-05-24 8:15 [patch V3 29/32] cpu/hotplug: Convert hotplug locking to percpu rwsem Thomas Gleixner
2017-05-26 8:47 ` [tip:smp/hotplug] " tip-bot for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=tip-d215aab82d81974f438bfbc80aa437132f3c37c3@git.kernel.org \
--to=tipbot@zytor.com \
--cc=bigeasy@linutronix.de \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).