From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: linux-kernel@vger.kernel.org
Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com,
akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca,
josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de,
peterz@infradead.org, rostedt@goodmis.org,
Valdis.Kletnieks@vt.edu, dhowells@redhat.com,
eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com,
patches@linaro.org, "Paul E. McKenney" <paul.mckenney@linaro.org>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Subject: [PATCH RFC tip/core/rcu 38/41] rcu: Rework detection of use of RCU by offline CPUs
Date: Wed, 1 Feb 2012 11:41:56 -0800 [thread overview]
Message-ID: <1328125319-5205-38-git-send-email-paulmck@linux.vnet.ibm.com> (raw)
In-Reply-To: <1328125319-5205-1-git-send-email-paulmck@linux.vnet.ibm.com>
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Because newly offlined CPUs continue executing after completing the
CPU_DYING notifiers, they legitimately enter the scheduler and use
RCU while appearing to be offline. This calls for a more sophisticated
approach as follows:
1. RCU marks the CPU online during the CPU_UP_PREPARE phase.
2. RCU marks the CPU offline during the CPU_DEAD phase.
3. Diagnostics regarding use of read-side RCU by offline CPUs use
RCU's accounting rather than the cpu_online_map. (Note that
__call_rcu() still uses cpu_online_map to detect illegal
invocations within CPU_DYING notifiers.)
4. Offline CPUs are prevented from hanging the system by
force_quiescent_state(), which pays attention to cpu_online_map.
Some additional work (in a later commit) will be needed to
guarantee that force_quiescent_state() waits a full jiffy before
assuming that a CPU is offline, for example, when called from
idle entry.
This approach avoids the false positives encountered when attempting to
use more exact classification of CPU online/offline state.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 92 ++++++++++++++++++++++++++++++----------------
kernel/rcutree_plugin.h | 2 +-
2 files changed, 61 insertions(+), 33 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 632b1c3..ce39431 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -604,19 +604,33 @@ EXPORT_SYMBOL(rcu_is_cpu_idle);
* this task being preempted, its old CPU being taken offline, resuming
* on some other CPU, then determining that its old CPU is now offline.
* It is OK to use RCU on an offline processor during initial boot, hence
- * the check for rcu_scheduler_fully_active.
+ * the check for rcu_scheduler_fully_active. Note also that it is OK
+ * for a CPU coming online to use RCU for one jiffy prior to marking itself
+ * online in the cpu_online_mask. Similarly, it is OK for a CPU going
+ * offline to continue to use RCU for one jiffy after marking itself
+ * offline in the cpu_online_mask. This leniency is necessary given the
+ * non-atomic nature of the online and offline processing, for example,
+ * the fact that a CPU enters the scheduler after completing the CPU_DYING
+ * notifiers.
+ *
+ * This is also why RCU internally marks CPUs online during the
+ * CPU_UP_PREPARE phase and offline during the CPU_DEAD phase.
*
* Disable checking if in an NMI handler because we cannot safely report
* errors from NMI handlers anyway.
*/
bool rcu_lockdep_current_cpu_online(void)
{
+ struct rcu_data *rdp;
+ struct rcu_node *rnp;
bool ret;
if (in_nmi())
return 1;
preempt_disable();
- ret = cpu_online(smp_processor_id()) ||
+ rdp = &__get_cpu_var(rcu_sched_data);
+ rnp = rdp->mynode;
+ ret = (rdp->grpmask & rnp->qsmaskinit) ||
!rcu_scheduler_fully_active;
preempt_enable();
return ret;
@@ -1311,14 +1325,12 @@ rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
*/
static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
{
- unsigned long flags;
int i;
unsigned long mask;
- int need_report;
int receive_cpu = cpumask_any(cpu_online_mask);
struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
struct rcu_data *receive_rdp = per_cpu_ptr(rsp->rda, receive_cpu);
- struct rcu_node *rnp = rdp->mynode; /* For dying CPU. */
+ RCU_TRACE(struct rcu_node *rnp = rdp->mynode); /* For dying CPU. */
/* First, adjust the counts. */
if (rdp->nxtlist != NULL) {
@@ -1384,32 +1396,6 @@ static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
"cpuofl");
rcu_report_qs_rdp(smp_processor_id(), rsp, rdp, rsp->gpnum);
/* Note that rcu_report_qs_rdp() might call trace_rcu_grace_period(). */
-
- /*
- * Remove the dying CPU from the bitmasks in the rcu_node
- * hierarchy. Because we are in stop_machine() context, we
- * automatically exclude ->onofflock critical sections.
- */
- do {
- raw_spin_lock_irqsave(&rnp->lock, flags);
- rnp->qsmaskinit &= ~mask;
- if (rnp->qsmaskinit != 0) {
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- break;
- }
- if (rnp == rdp->mynode) {
- need_report = rcu_preempt_offline_tasks(rsp, rnp, rdp);
- if (need_report & RCU_OFL_TASKS_NORM_GP)
- rcu_report_unblock_qs_rnp(rnp, flags);
- else
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- if (need_report & RCU_OFL_TASKS_EXP_GP)
- rcu_report_exp_rnp(rsp, rnp, true);
- } else
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- mask = rnp->grpmask;
- rnp = rnp->parent;
- } while (rnp != NULL);
}
/*
@@ -1420,11 +1406,53 @@ static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
*/
static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)
{
+ unsigned long flags;
+ unsigned long mask;
+ int need_report = 0;
struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
- struct rcu_node *rnp = rdp->mynode;
+ struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rnp. */
+ /* Adjust any no-longer-needed kthreads. */
rcu_stop_cpu_kthread(cpu);
rcu_node_kthread_setaffinity(rnp, -1);
+
+ /* Remove the dying CPU from the bitmasks in the rcu_node hierarchy. */
+
+ /* Exclude any attempts to start a new grace period. */
+ raw_spin_lock_irqsave(&rsp->onofflock, flags);
+
+ /* Remove the outgoing CPU from the masks in the rcu_node hierarchy. */
+ mask = rdp->grpmask; /* rnp->grplo is constant. */
+ do {
+ raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ rnp->qsmaskinit &= ~mask;
+ if (rnp->qsmaskinit != 0) {
+ if (rnp != rdp->mynode)
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ break;
+ }
+ if (rnp == rdp->mynode)
+ need_report = rcu_preempt_offline_tasks(rsp, rnp, rdp);
+ else
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ mask = rnp->grpmask;
+ rnp = rnp->parent;
+ } while (rnp != NULL);
+
+ /*
+ * We still hold the leaf rcu_node structure lock here, and
+ * irqs are still disabled. The reason for this subterfuge is
+ * because invoking rcu_report_unblock_qs_rnp() with ->onofflock
+ * held leads to deadlock.
+ */
+ raw_spin_unlock(&rsp->onofflock); /* irqs remain disabled. */
+ rnp = rdp->mynode;
+ if (need_report & RCU_OFL_TASKS_NORM_GP)
+ rcu_report_unblock_qs_rnp(rnp, flags);
+ else
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ if (need_report & RCU_OFL_TASKS_EXP_GP)
+ rcu_report_exp_rnp(rsp, rnp, true);
}
#else /* #ifdef CONFIG_HOTPLUG_CPU */
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 3dc58cc..56a63bc 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -610,7 +610,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
* absolutely necessary, but this is a good performance/complexity
* tradeoff.
*/
- if (rcu_preempt_blocked_readers_cgp(rnp))
+ if (rcu_preempt_blocked_readers_cgp(rnp) && rnp->qsmask == 0)
retval |= RCU_OFL_TASKS_NORM_GP;
if (rcu_preempted_readers_exp(rnp))
retval |= RCU_OFL_TASKS_EXP_GP;
--
1.7.8
next prev parent reply other threads:[~2012-02-01 19:51 UTC|newest]
Thread overview: 104+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-01 19:41 [PATCH RFC 0/41] RCU commits for 3.4 Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 01/41] rcu: Bring RTFP.txt up to date Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 02/41] rcu: Improve synchronize_rcu() diagnostics Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 03/41] rcu: Add lockdep-RCU checks for simple self-deadlock Paul E. McKenney
2012-02-02 0:55 ` Josh Triplett
2012-02-02 16:20 ` Paul E. McKenney
2012-02-02 19:56 ` Josh Triplett
2012-02-02 20:42 ` Paul E. McKenney
2012-02-03 9:04 ` Josh Triplett
2012-02-03 18:05 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 04/41] rcu: Add diagnostic for misaligned rcu_head structures Paul E. McKenney
2012-02-02 1:00 ` Josh Triplett
2012-02-02 16:22 ` Paul E. McKenney
2012-02-02 20:11 ` Josh Triplett
2012-02-02 1:01 ` Josh Triplett
2012-02-02 16:27 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 05/41] rcu: Avoid waking up CPUs having only kfree_rcu() callbacks Paul E. McKenney
2012-02-02 1:15 ` Josh Triplett
2012-02-02 16:34 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 06/41] rcu: Move RCU_TRACE to lib/Kconfig.debug Paul E. McKenney
2012-02-02 1:39 ` Josh Triplett
2012-02-02 17:05 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 07/41] s390: Convert call_rcu() to kfree_rcu() Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 08/41] tcm_fc: " Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 09/41] ipv4: " Paul E. McKenney
2012-02-01 19:49 ` David Miller
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 10/41] " Paul E. McKenney
2012-02-01 19:50 ` David Miller
2012-02-02 0:24 ` Josh Triplett
2012-02-02 15:56 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 11/41] mac80211: " Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 12/41] rcu: Simplify offline processing Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 13/41] rcu: Make rcutorture flag online/offline failures Paul E. McKenney
2012-02-02 1:46 ` Josh Triplett
2012-02-02 17:08 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 14/41] rcu: Limit lazy-callback duration Paul E. McKenney
2012-02-02 2:03 ` Josh Triplett
2012-02-02 17:13 ` Paul E. McKenney
2012-02-03 4:07 ` Josh Triplett
2012-02-03 5:54 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 15/41] rcu: Check for callback invocation from offline CPUs Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 16/41] rcu: Don't make callbacks go through second full grace period Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 17/41] rcu: Remove single-rcu_node optimization in rcu_start_gp() Paul E. McKenney
2012-02-02 2:13 ` Josh Triplett
2012-02-02 17:16 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 18/41] rcu: Protect __rcu_read_unlock() against scheduler-using irq handlers Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 19/41] rcu: Streamline code produced by __rcu_read_unlock() Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 20/41] rcu: Prevent RCU callbacks from executing before scheduler initialized Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 21/41] rcu: Inform RCU of irq_exit() activity Paul E. McKenney
2012-02-02 2:30 ` Josh Triplett
2012-02-02 17:30 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 22/41] rcu: Simplify unboosting checks Paul E. McKenney
2012-02-02 2:38 ` Josh Triplett
2012-02-02 17:48 ` Paul E. McKenney
2012-02-03 4:23 ` Josh Triplett
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 23/41] rcu: Clean up straggling rcu_preempt_needs_cpu() name Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 24/41] rcu: Check for idle-loop entry while in RCU read-side critical section Paul E. McKenney
2012-02-02 5:13 ` Josh Triplett
2012-02-02 17:50 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 25/41] rcu: Make rcu_sleep_check() also check rcu_lock_map Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 26/41] rcu: Note that rcu_access_pointer() can be used for teardown Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 27/41] rcu: Remove #ifdef CONFIG_SMP from TREE_RCU Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 28/41] rcu: Set RCU CPU stall times via sysfs Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 29/41] rcu: Print scheduling-clock information on RCU CPU stall-warning messages Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 30/41] rcutorture: Permit holding off CPU-hotplug operations during boot Paul E. McKenney
2012-02-02 5:43 ` Josh Triplett
2012-02-02 17:56 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 31/41] rcu: Add CPU-stall capability to rcutorture Paul E. McKenney
2012-02-02 5:53 ` Josh Triplett
2012-02-02 9:15 ` Julia Lawall
2012-02-02 18:03 ` Paul E. McKenney
2012-02-02 18:00 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 32/41] rcu: Update stall-warning documentation Paul E. McKenney
2012-02-02 5:56 ` Josh Triplett
2012-02-02 18:18 ` Paul E. McKenney
2012-02-03 5:42 ` Josh Triplett
2012-02-03 5:58 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 33/41] rcu: Make boolean rcutorture parameters be of type "bool" Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 34/41] rcu: Check for illegal use of RCU from offlined CPUs Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 35/41] rcu: Move synchronize_sched_expedited() to rcutree.c Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 36/41] rcu: No interrupt disabling for rcu_prepare_for_idle() Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 37/41] lockdep: Add CPU-idle/offline warning to lockdep-RCU splat Paul E. McKenney
2012-02-02 6:07 ` Josh Triplett
2012-02-02 18:30 ` Paul E. McKenney
2012-02-03 6:12 ` Josh Triplett
2012-02-01 19:41 ` Paul E. McKenney [this message]
2012-02-02 6:11 ` [PATCH RFC tip/core/rcu 38/41] rcu: Rework detection of use of RCU by offline CPUs Josh Triplett
2012-02-02 18:31 ` Paul E. McKenney
2012-02-03 9:17 ` Josh Triplett
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 39/41] rcu: Wait at least a jiffy before declaring a CPU to be offline Paul E. McKenney
2012-02-02 6:12 ` Josh Triplett
2012-02-02 18:27 ` Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 40/41] rcu: Call out dangers of expedited RCU primitives Paul E. McKenney
2012-02-01 19:41 ` [PATCH RFC tip/core/rcu 41/41] rcu: Trace only after NULL-pointer check Paul E. McKenney
2012-02-02 0:18 ` [PATCH RFC tip/core/rcu 01/41] rcu: Bring RTFP.txt up to date Josh Triplett
2012-02-02 1:33 ` Paul E. McKenney
2012-02-02 2:01 ` Josh Triplett
2012-02-02 16:47 ` Paul E. McKenney
2012-02-02 22:32 ` Josh Triplett
2012-02-03 18:00 ` Paul E. McKenney
2012-02-02 22:47 ` [PATCH RFC 0/41] RCU commits for 3.4 Kevin Hilman
2012-02-02 23:58 ` Paul E. McKenney
2012-02-03 19:54 ` Kevin Hilman
2012-02-06 7:04 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1328125319-5205-38-git-send-email-paulmck@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=Valdis.Kletnieks@vt.edu \
--cc=akpm@linux-foundation.org \
--cc=darren@dvhart.com \
--cc=dhowells@redhat.com \
--cc=dipankar@in.ibm.com \
--cc=eric.dumazet@gmail.com \
--cc=fweisbec@gmail.com \
--cc=josh@joshtriplett.org \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@polymtl.ca \
--cc=mingo@elte.hu \
--cc=niv@us.ibm.com \
--cc=patches@linaro.org \
--cc=paul.mckenney@linaro.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).