From: "tip-bot2 for Thomas Gleixner" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
x86@kernel.org, linux-kernel@vger.kernel.org
Subject: [tip: timers/core] tick/sched: Use tick_next_period for lockless quick check
Date: Thu, 19 Nov 2020 09:55:09 -0000 [thread overview]
Message-ID: <160577970969.11244.10855382455981949914.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20201117132006.337366695@linutronix.de>
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 372acbbaa80940189593f9d69c7c069955f24f7a
Gitweb: https://git.kernel.org/tip/372acbbaa80940189593f9d69c7c069955f24f7a
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Tue, 17 Nov 2020 14:19:45 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Thu, 19 Nov 2020 10:48:29 +01:00
tick/sched: Use tick_next_period for lockless quick check
No point in doing calculations.
tick_next_period = last_jiffies_update + tick_period
Just check whether now is before tick_next_period to figure out whether
jiffies need an update.
Add a comment why the intentional data race in the quick check is safe or
not so safe in a 32bit corner case and why we don't worry about it.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20201117132006.337366695@linutronix.de
---
kernel/time/tick-sched.c | 46 +++++++++++++++++++++++++++------------
1 file changed, 33 insertions(+), 13 deletions(-)
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 15360e6..b4b6abc 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -59,11 +59,29 @@ static void tick_do_update_jiffies64(ktime_t now)
ktime_t delta;
/*
- * Do a quick check without holding jiffies_lock:
- * The READ_ONCE() pairs with two updates done later in this function.
+ * Do a quick check without holding jiffies_lock. The READ_ONCE()
+ * pairs with the update done later in this function.
+ *
+ * This is also an intentional data race which is even safe on
+ * 32bit in theory. If there is a concurrent update then the check
+ * might give a random answer. It does not matter because if it
+ * returns then the concurrent update is already taking care, if it
+ * falls through then it will pointlessly contend on jiffies_lock.
+ *
+ * Though there is one nasty case on 32bit due to store tearing of
+ * the 64bit value. If the first 32bit store makes the quick check
+ * return on all other CPUs and the writing CPU context gets
+ * delayed to complete the second store (scheduled out on virt)
+ * then jiffies can become stale for up to ~2^32 nanoseconds
+ * without noticing. After that point all CPUs will wait for
+ * jiffies lock.
+ *
+ * OTOH, this is not any different than the situation with NOHZ=off
+ * where one CPU is responsible for updating jiffies and
+ * timekeeping. If that CPU goes out for lunch then all other CPUs
+ * will operate on stale jiffies until it decides to come back.
*/
- delta = ktime_sub(now, READ_ONCE(last_jiffies_update));
- if (delta < tick_period)
+ if (ktime_before(now, READ_ONCE(tick_next_period)))
return;
/* Reevaluate with jiffies_lock held */
@@ -74,9 +92,8 @@ static void tick_do_update_jiffies64(ktime_t now)
if (delta >= tick_period) {
delta = ktime_sub(delta, tick_period);
- /* Pairs with the lockless read in this function. */
- WRITE_ONCE(last_jiffies_update,
- ktime_add(last_jiffies_update, tick_period));
+ last_jiffies_update = ktime_add(last_jiffies_update,
+ tick_period);
/* Slow path for long timeouts */
if (unlikely(delta >= tick_period)) {
@@ -84,15 +101,18 @@ static void tick_do_update_jiffies64(ktime_t now)
ticks = ktime_divns(delta, incr);
- /* Pairs with the lockless read in this function. */
- WRITE_ONCE(last_jiffies_update,
- ktime_add_ns(last_jiffies_update,
- incr * ticks));
+ last_jiffies_update = ktime_add_ns(last_jiffies_update,
+ incr * ticks);
}
do_timer(++ticks);
- /* Keep the tick_next_period variable up to date */
- tick_next_period = ktime_add(last_jiffies_update, tick_period);
+ /*
+ * Keep the tick_next_period variable up to date.
+ * WRITE_ONCE() pairs with the READ_ONCE() in the lockless
+ * quick check above.
+ */
+ WRITE_ONCE(tick_next_period,
+ ktime_add(last_jiffies_update, tick_period));
} else {
write_seqcount_end(&jiffies_seq);
raw_spin_unlock(&jiffies_lock);
next prev parent reply other threads:[~2020-11-19 9:55 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-17 13:19 [patch 0/7] tick: Cleanups and reduce jiffies_seq held times Thomas Gleixner
2020-11-17 13:19 ` [patch 1/7] tick/broadcast: Serialize access to tick_next_period Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Thomas Gleixner
2020-11-17 13:19 ` [patch 2/7] tick: Document protections for tick related data Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Thomas Gleixner
2020-11-17 13:19 ` [patch 3/7] tick/sched: Use tick_next_period for lockless quick check Thomas Gleixner
2020-11-19 9:55 ` tip-bot2 for Thomas Gleixner [this message]
2020-11-17 13:19 ` [patch 4/7] tick/sched: Reduce seqcount held scope in tick_do_update_jiffies64() Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Yunfeng Ye
2020-11-17 13:19 ` [patch 5/7] tick/sched: Optimize tick_do_update_jiffies64() further Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Thomas Gleixner
2020-11-17 13:19 ` [patch 6/7] tick/sched: Release seqcount before invoking calc_load_global() Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Yunfeng Ye
2020-11-17 13:19 ` [patch 7/7] tick: Get rid of tick_period Thomas Gleixner
2020-11-19 9:55 ` [tip: timers/core] " tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=160577970969.11244.10855382455981949914.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).