From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Paul McKenney <paulmck@linux.vnet.ibm.com>,
Sebastian Siewior <bigeasy@linutronix.de>
Subject: [patch 6/8] sched: Make RCU nest depth distinct in __might_resched()
Date: Thu, 23 Sep 2021 18:54:43 +0200 (CEST) [thread overview]
Message-ID: <20210923165358.243232823@linutronix.de> (raw)
In-Reply-To: 20210923164145.466686140@linutronix.de
For !RT kernels RCU nest depth in __might_resched() is always expected to
be 0, but on RT kernels it can be non zero while the preempt count is
expected to be always 0.
Instead of playing magic games in interpreting the 'preempt_offset'
argument, rename it to 'offsets' and use the lower 8 bits for the expected
preempt count, allow to hand in the expected RCU nest depth in the upper
bits and adopt the __might_resched() code and related checks and printks.
The affected call sites are updated in subsequent steps.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
include/linux/kernel.h | 4 ++--
include/linux/sched.h | 3 +++
kernel/sched/core.c | 28 ++++++++++++++++------------
3 files changed, 21 insertions(+), 14 deletions(-)
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -111,7 +111,7 @@ static __always_inline void might_resche
#endif /* CONFIG_PREEMPT_* */
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
-extern void __might_resched(const char *file, int line, int preempt_offset);
+extern void __might_resched(const char *file, int line, unsigned int offsets);
extern void __might_sleep(const char *file, int line);
extern void __cant_sleep(const char *file, int line, int preempt_offset);
extern void __cant_migrate(const char *file, int line);
@@ -169,7 +169,7 @@ extern void __cant_migrate(const char *f
# define non_block_end() WARN_ON(current->non_block_count-- == 0)
#else
static inline void __might_resched(const char *file, int line,
- int preempt_offset) { }
+ unsigned int offsets) { }
static inline void __might_sleep(const char *file, int line) { }
# define might_sleep() do { might_resched(); } while (0)
# define cant_sleep() do { } while (0)
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2049,6 +2049,9 @@ extern int __cond_resched_lock(spinlock_
extern int __cond_resched_rwlock_read(rwlock_t *lock);
extern int __cond_resched_rwlock_write(rwlock_t *lock);
+#define MIGHT_RESCHED_RCU_SHIFT 8
+#define MIGHT_RESCHED_PREEMPT_MASK ((1U << MIGHT_RESCHED_RCU_SHIFT) - 1)
+
#define cond_resched_lock(lock) ({ \
__might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \
__cond_resched_lock(lock); \
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9425,12 +9425,6 @@ void __init sched_init(void)
}
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
-static inline int preempt_count_equals(int preempt_offset)
-{
- int nested = preempt_count() + rcu_preempt_depth();
-
- return (nested == preempt_offset);
-}
void __might_sleep(const char *file, int line)
{
@@ -9462,7 +9456,16 @@ static void print_preempt_disable_ip(int
print_ip_sym(KERN_ERR, ip);
}
-void __might_resched(const char *file, int line, int preempt_offset)
+static inline bool resched_offsets_ok(unsigned int offsets)
+{
+ unsigned int nested = preempt_count();
+
+ nested += rcu_preempt_depth() << MIGHT_RESCHED_RCU_SHIFT;
+
+ return nested == offsets;
+}
+
+void __might_resched(const char *file, int line, unsigned int offsets)
{
/* Ratelimiting timestamp: */
static unsigned long prev_jiffy;
@@ -9472,7 +9475,7 @@ void __might_resched(const char *file, i
/* WARN_ON_ONCE() by default, no rate limit required: */
rcu_sleep_check();
- if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
+ if ((resched_offsets_ok(offsets) && !irqs_disabled() &&
!is_idle_task(current) && !current->non_block_count) ||
system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING ||
oops_in_progress)
@@ -9491,11 +9494,11 @@ void __might_resched(const char *file, i
in_atomic(), irqs_disabled(), current->non_block_count,
current->pid, current->comm);
pr_err("preempt_count: %x, expected: %x\n", preempt_count(),
- preempt_offset);
+ offsets & MIGHT_RESCHED_PREEMPT_MASK);
if (IS_ENABLED(CONFIG_PREEMPT_RCU)) {
- pr_err("RCU nest depth: %d, expected: 0\n",
- rcu_preempt_depth());
+ pr_err("RCU nest depth: %d, expected: %u\n",
+ rcu_preempt_depth(), offsets >> MIGHT_RESCHED_RCU_SHIFT);
}
if (task_stack_end_corrupted(current))
@@ -9505,7 +9508,8 @@ void __might_resched(const char *file, i
if (irqs_disabled())
print_irqtrace_events(current);
- print_preempt_disable_ip(preempt_offset, preempt_disable_ip);
+ print_preempt_disable_ip(offsets & MIGHT_RESCHED_PREEMPT_MASK,
+ preempt_disable_ip);
dump_stack();
add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
next prev parent reply other threads:[~2021-09-23 16:54 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-23 16:54 [patch 0/8] sched: Clean up might_sleep() and make it RT aware Thomas Gleixner
2021-09-23 16:54 ` [patch 1/8] sched: Clean up the might_sleep() underscore zoo Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 2/8] sched: Make cond_resched_*lock() variants consistent vs. might_sleep() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 3/8] sched: Remove preempt_offset argument from __might_sleep() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 4/8] sched: Cleanup might_sleep() printks Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 5/8] sched: Make might_sleep() output less confusing Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` Thomas Gleixner [this message]
2021-10-01 15:05 ` [tip: locking/core] sched: Make RCU nest depth distinct in __might_resched() tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 7/8] sched: Make cond_resched_lock() variants RT aware Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 8/8] locking/rt: Take RCU nesting into account for __might_resched() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210923165358.243232823@linutronix.de \
--to=tglx@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.