From: "Paul E. McKenney" <paulmck@kernel.org>
To: rcu@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com,
mingo@kernel.org, jiangshanlai@gmail.com,
akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org,
rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com,
fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org,
"Paul E. McKenney" <paulmck@kernel.org>
Subject: [PATCH tip/core/rcu 22/26] rcutorture: Don't count CPU-stalled time against priority boosting
Date: Tue, 11 May 2021 16:12:19 -0700 [thread overview]
Message-ID: <20210511231223.2895398-22-paulmck@kernel.org> (raw)
In-Reply-To: <20210511231149.GA2895263@paulmck-ThinkPad-P17-Gen-1>
It will frequently be the case that rcu_torture_boost() will get a
->start_gp_poll() cookie that needs almost all of the current grace period
plus an additional grace period to elapse before ->poll_gp_state() will
return true. It is quite possible that the current grace period will have
(say) two seconds of stall by a CPU failing to pass through a quiescent
state, followed by 300 milliseconds of delay due to a preempted reader.
The next grace period might suffer only one second of stall by a CPU,
followed by another 300 milliseconds of delay due to a preempted reader.
This is an example of RCU priority boosting doing its job, but the full
elapsed time of 3.6 seconds exceeds the 3.5-second limit. In addition,
there is no CPU stall in force at the 3.5-second mark, so this would
nevertheless currently be counted as an RCU priority boosting failure.
This commit therefore avoids this sort of false positive by resetting
the gp_state_time timestamp any time that the current grace period is
being blocked by a CPU. This results in extremely frequent calls to
the ->check_boost_failed() function, so this commit provides a lockless
fastpath that is selected by supplying a NULL CPU-number pointer.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/rcutorture.c | 13 ++++++++-----
kernel/rcu/tree_stall.h | 10 ++++++++++
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 5ae4dcc6ba27..8b347b9659aa 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -918,17 +918,18 @@ static void rcu_torture_enable_rt_throttle(void)
old_rt_runtime = -1;
}
-static bool rcu_torture_boost_failed(unsigned long gp_state, unsigned long start, unsigned long end)
+static bool rcu_torture_boost_failed(unsigned long gp_state, unsigned long *start)
{
int cpu;
static int dbg_done;
+ unsigned long end = jiffies;
bool gp_done;
unsigned long j;
static unsigned long last_persist;
unsigned long lp;
unsigned long mininterval = test_boost_duration * HZ - HZ / 2;
- if (end - start > mininterval) {
+ if (end - *start > mininterval) {
// Recheck after checking time to avoid false positives.
smp_mb(); // Time check before grace-period check.
if (cur_ops->poll_gp_state(gp_state))
@@ -945,7 +946,7 @@ static bool rcu_torture_boost_failed(unsigned long gp_state, unsigned long start
n_rcu_torture_boost_failure++;
if (!xchg(&dbg_done, 1) && cur_ops->gp_kthread_dbg) {
pr_info("Boost inversion thread ->rt_priority %u gp_state %lu jiffies %lu\n",
- current->rt_priority, gp_state, end - start);
+ current->rt_priority, gp_state, end - *start);
cur_ops->gp_kthread_dbg();
// Recheck after print to flag grace period ending during splat.
gp_done = cur_ops->poll_gp_state(gp_state);
@@ -955,6 +956,8 @@ static bool rcu_torture_boost_failed(unsigned long gp_state, unsigned long start
}
return true; // failed
+ } else if (cur_ops->check_boost_failed && !cur_ops->check_boost_failed(gp_state, NULL)) {
+ *start = jiffies;
}
return false; // passed
@@ -995,7 +998,7 @@ static int rcu_torture_boost(void *arg)
while (time_before(jiffies, endtime)) {
// Has current GP gone too long?
if (gp_initiated && !failed && !cur_ops->poll_gp_state(gp_state))
- failed = rcu_torture_boost_failed(gp_state, gp_state_time, jiffies);
+ failed = rcu_torture_boost_failed(gp_state, &gp_state_time);
// If we don't have a grace period in flight, start one.
if (!gp_initiated || cur_ops->poll_gp_state(gp_state)) {
gp_state = cur_ops->start_gp_poll();
@@ -1016,7 +1019,7 @@ static int rcu_torture_boost(void *arg)
// In case the grace period extended beyond the end of the loop.
if (gp_initiated && !failed && !cur_ops->poll_gp_state(gp_state))
- rcu_torture_boost_failed(gp_state, gp_state_time, jiffies);
+ rcu_torture_boost_failed(gp_state, &gp_state_time);
/*
* Set the start time of the next test interval.
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index af92d9fee0d4..8bde1b53b0c9 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -723,6 +723,10 @@ static void check_cpu_stall(struct rcu_data *rdp)
* count this as an RCU priority boosting failure. A return of true says
* RCU priority boosting is to blame, and false says otherwise. If false
* is returned, the first of the CPUs to blame is stored through cpup.
+ *
+ * If cpup is NULL, then a lockless quick check is carried out, suitable
+ * for high-rate usage. On the other hand, if cpup is non-NULL, each
+ * rcu_node structure's ->lock is acquired, ruling out high-rate usage.
*/
bool rcu_check_boost_fail(unsigned long gp_state, int *cpup)
{
@@ -731,6 +735,12 @@ bool rcu_check_boost_fail(unsigned long gp_state, int *cpup)
struct rcu_node *rnp;
rcu_for_each_leaf_node(rnp) {
+ if (!cpup) {
+ if (READ_ONCE(rnp->qsmask))
+ return false;
+ else
+ continue;
+ }
raw_spin_lock_irqsave_rcu_node(rnp, flags);
if (!rnp->qsmask) {
// No CPUs without quiescent states for this rnp.
--
2.31.1.189.g2e36527f23
next prev parent reply other threads:[~2021-05-11 23:12 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-11 23:11 [PATCH tip/core/rcu 0/26] Torture-test updates for v5.14 Paul E. McKenney
2021-05-11 23:11 ` [PATCH tip/core/rcu 01/26] torture: Fix remaining erroneous torture.sh instance of $* Paul E. McKenney
2021-05-11 23:11 ` [PATCH tip/core/rcu 02/26] torture: Add "scenarios" option to kvm.sh --dryrun parameter Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 03/26] torture: Make kvm-again.sh use "scenarios" rather than "batches" file Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 04/26] refscale: Allow CPU hotplug to be enabled Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 05/26] rcuscale: " Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 06/26] torture: Add kvm-remote.sh script for distributed rcutorture test runs Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 07/26] refscale: Add acqrel, lock, and lock-irq Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 08/26] rcutorture: Abstract read-lock-held checks Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 09/26] torture: Fix grace-period rate output Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 10/26] torture: Abstract end-of-run summary Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 11/26] torture: Make kvm.sh use abstracted kvm-end-run-stats.sh Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 12/26] torture: Make the build machine control N in "make -jN" Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 13/26] torture: Make kvm-find-errors.sh account for kvm-remote.sh Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 14/26] rcutorture: Judge RCU priority boosting on grace periods, not callbacks Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 15/26] torture: Correctly fetch number of CPUs for non-English languages Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 16/26] torture: Set kvm.sh language to English Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 17/26] rcutorture: Delay-based false positives for RCU priority boosting tests Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 18/26] rcutorture: Consolidate rcu_torture_boost() timing and statistics Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 19/26] rcutorture: Make rcu_torture_boost_failed() check for GP end Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 20/26] rcutorture: Add BUSTED-BOOST to test RCU priority boosting tests Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 21/26] rcutorture: Forgive RCU boost failures when CPUs don't pass through QS Paul E. McKenney
2021-05-11 23:12 ` Paul E. McKenney [this message]
2021-05-11 23:12 ` [PATCH tip/core/rcu 23/26] torture: Make kvm-remote.sh account for network failure in pathname checks Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 24/26] torture: Don't cap remote runs by build-system number of CPUs Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 25/26] rcutorture: Move mem_dump_obj() tests into separate function Paul E. McKenney
2021-05-11 23:12 ` [PATCH tip/core/rcu 26/26] rcu: Don't penalize priority boosting when there is nothing to boost Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210511231223.2895398-22-paulmck@kernel.org \
--to=paulmck@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=dhowells@redhat.com \
--cc=edumazet@google.com \
--cc=fweisbec@gmail.com \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).