archive mirror
 help / color / mirror / Atom feed
From: "Paul E. McKenney" <>
	"Paul E. McKenney" <>
Subject: [PATCH RFC v2 tip/core/rcu 3/4] rcu: Only do rcu_read_unlock_special() wakeups if expedited
Date: Tue,  2 Apr 2019 07:38:41 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Currently, rcu_read_unlock_special() will do wakeups whenever it is safe
to do so.  However, wakeups are expensive, and they are only really
needed when the just-ended RCU read-side critical section is blocking
an expedited grace period (in which case speed is of the essence)
or on a nohz_full CPU (where it might be a good long time before an
interrupt arrives).  This commit therefore checks for these conditions,
and does the expensive wakeups only if doing so would be useful.

Note it can be rather expensive to determine whether or not the current
task (as opposed to the current CPU) is blocking the current expedited
grace period.  Doing so requires traversing the ->blkd_tasks list, which
can be quite long.  This commit therefore cheats:  If the current task
is on a given ->blkd_tasks list, and some task on that list is blocking
the current expedited grace period, the code assumes that the current
task is blocking that expedited grace period.

Reported-by: Peter Zijlstra <>
Signed-off-by: Paul E. McKenney <>
 kernel/rcu/tree_plugin.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 75110ea75d01..d15cdab6aeb4 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -606,20 +606,28 @@ static void rcu_read_unlock_special(struct task_struct *t)
 	irqs_were_disabled = irqs_disabled_flags(flags);
 	if (preempt_bh_were_disabled || irqs_were_disabled) {
+		bool exp;
+		struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+		struct rcu_node *rnp = rdp->mynode;
 		t->rcu_read_unlock_special.b.exp_hint = false;
+		exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) ||
+		      (rdp->grpmask & rnp->expmask) ||
+		      tick_nohz_full_cpu(rdp->cpu);
 		// Need to defer quiescent state until everything is enabled.
-		if (irqs_were_disabled && use_softirq &&
+		if (exp && irqs_were_disabled && use_softirq &&
 		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
 			// Using softirq, safe to awaken, and we get
 			// no help from enabling irqs, unlike bh/preempt.
-		} else if (irqs_were_disabled && !use_softirq &&
+		} else if (exp && irqs_were_disabled && !use_softirq &&
 			   !t->rcu_read_unlock_special.b.deferred_qs) {
 			// Safe to awaken and we get no help from enabling
 			// irqs, unlike bh/preempt.
 		} else {
 			// Enabling BH or preempt does reschedule, so...
+			// Also if no expediting or NO_HZ_FULL, slow is OK.

  parent reply	other threads:[~2019-04-02 14:38 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-02 14:37 [PATCH RFC v2 tip/core/rcu 0/4] Real-time elimination of RCU_SOFTIRQ Paul E. McKenney
2019-04-02 14:38 ` [PATCH RFC v2 tip/core/rcu 1/4] rcu: Enable elimination of Tree-RCU softirq processing Paul E. McKenney
2019-04-02 14:38 ` [PATCH RFC v2 tip/core/rcu 2/4] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special() Paul E. McKenney
2019-04-02 14:38 ` Paul E. McKenney [this message]
2019-04-02 14:38 ` [PATCH RFC v2 tip/core/rcu 4/4] rcu: Allow rcu_read_unlock_special() to raise_softirq() if in_irq() Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH RFC v2 tip/core/rcu 3/4] rcu: Only do rcu_read_unlock_special() wakeups if expedited' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).