From: Frederic Weisbecker <frederic@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>,
"Paul E . McKenney" <paulmck@kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <frederic@kernel.org>,
"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
Ingo Molnar <mingo@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
stable@vger.kernel.org
Subject: [RFC PATCH 6/8] sched: Report local wake up on resched blind zone within idle loop
Date: Sat, 9 Jan 2021 03:05:34 +0100 [thread overview]
Message-ID: <20210109020536.127953-7-frederic@kernel.org> (raw)
In-Reply-To: <20210109020536.127953-1-frederic@kernel.org>
The idle loop has several need_resched() checks that make sure we don't
miss a rescheduling request. This means that any wake up performed on
the local runqueue after the last generic need_resched() check is going
to have its rescheduling silently ignored. This has happened in the
past with rcu kthreads awaken from rcu_idle_enter() for example.
Perform sanity checks to report these situations.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar<mingo@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
include/linux/sched.h | 11 +++++++++++
kernel/sched/core.c | 42 ++++++++++++++++++++++++++++++++++++++++++
kernel/sched/idle.c | 3 +++
kernel/sched/sched.h | 3 +++
4 files changed, 59 insertions(+)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6e3a5eeec509..83fedda54943 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1917,6 +1917,17 @@ static __always_inline bool need_resched(void)
return unlikely(tif_need_resched());
}
+#ifdef CONFIG_SCHED_DEBUG
+extern void sched_resched_local_allow(void);
+extern void sched_resched_local_forbid(void);
+extern void sched_resched_local_assert_allowed(void);
+#else
+static inline void sched_resched_local_allow(void) { }
+static inline void sched_resched_local_forbid(void) { }
+static inline void sched_resched_local_assert_allowed(void) { }
+#endif
+
+
/*
* Wrappers for p->thread_info->cpu access. No-op on UP.
*/
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 15d2562118d1..6056f0374674 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -591,6 +591,44 @@ void wake_up_q(struct wake_q_head *head)
}
}
+#ifdef CONFIG_SCHED_DEBUG
+void noinstr sched_resched_local_allow(void)
+{
+ this_rq()->resched_local_allow = 1;
+}
+
+void noinstr sched_resched_local_forbid(void)
+{
+ this_rq()->resched_local_allow = 0;
+}
+
+void noinstr sched_resched_local_assert_allowed(void)
+{
+ if (this_rq()->resched_local_allow)
+ return;
+
+ /*
+ * Idle interrupts break the CPU from its pause and
+ * rescheduling happens on idle loop exit.
+ */
+ if (in_hardirq())
+ return;
+
+ /*
+ * What applies to hardirq also applies to softirq as
+ * we assume they execute on hardirq tail. Ksoftirqd
+ * shouldn't have resched_local_allow == 0.
+ * We also assume that no local_bh_enable() call may
+ * execute softirqs inline on fragile idle/entry
+ * path...
+ */
+ if (in_serving_softirq())
+ return;
+
+ WARN_ONCE(1, "Late current task rescheduling may be lost\n");
+}
+#endif
+
/*
* resched_curr - mark rq's current task 'to be rescheduled now'.
*
@@ -613,6 +651,7 @@ void resched_curr(struct rq *rq)
if (cpu == smp_processor_id()) {
set_tsk_need_resched(curr);
set_preempt_need_resched();
+ sched_resched_local_assert_allowed();
return;
}
@@ -7796,6 +7835,9 @@ void __init sched_init(void)
#endif /* CONFIG_SMP */
hrtick_rq_init(rq);
atomic_set(&rq->nr_iowait, 0);
+#ifdef CONFIG_SCHED_DEBUG
+ rq->resched_local_allow = 1;
+#endif
}
set_load_weight(&init_task, false);
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index b601a3aa2152..cdffd32812bd 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -185,6 +185,8 @@ static void cpuidle_idle_call(void)
return;
}
+ sched_resched_local_forbid();
+
/*
* The RCU framework needs to be told that we are entering an idle
* section, so no more rcu read side critical sections and one more
@@ -247,6 +249,7 @@ static void cpuidle_idle_call(void)
}
exit_idle:
+ sched_resched_local_allow();
__current_set_polling();
/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 12ada79d40f3..a9416c383451 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1060,6 +1060,9 @@ struct rq {
#endif
unsigned int push_busy;
struct cpu_stop_work push_work;
+#ifdef CONFIG_SCHED_DEBUG
+ unsigned int resched_local_allow;
+#endif
};
#ifdef CONFIG_FAIR_GROUP_SCHED
--
2.25.1
next prev parent reply other threads:[~2021-01-09 2:07 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-09 2:05 [RFC PATCH 0/8] rcu/sched: Fix ignored rescheduling after rcu_eqs_enter() v3 Frederic Weisbecker
2021-01-09 2:05 ` [RFC PATCH 1/8] rcu: Remove superfluous rdp fetch Frederic Weisbecker
2021-01-09 9:03 ` Greg KH
2021-01-10 10:59 ` Frederic Weisbecker
2021-01-09 2:05 ` [RFC PATCH 2/8] rcu: Pull deferred rcuog wake up to rcu_eqs_enter() callers Frederic Weisbecker
2021-01-09 2:05 ` [RFC PATCH 3/8] rcu/nocb: Perform deferred wake up before last idle's need_resched() check Frederic Weisbecker
2021-01-09 2:05 ` [RFC PATCH 4/8] rcu/nocb: Trigger self-IPI on late deferred wake up before user resume Frederic Weisbecker
2021-01-11 12:04 ` Peter Zijlstra
2021-01-11 12:35 ` Frederic Weisbecker
2021-01-09 2:05 ` [RFC PATCH 5/8] entry: Explicitly flush pending rcuog wakeup before last rescheduling points Frederic Weisbecker
2021-01-11 0:40 ` Frederic Weisbecker
2021-01-11 5:13 ` Paul E. McKenney
2021-01-11 11:50 ` Frederic Weisbecker
2021-01-11 12:08 ` Peter Zijlstra
2021-01-11 12:54 ` Frederic Weisbecker
2021-01-09 2:05 ` Frederic Weisbecker [this message]
2021-01-11 12:25 ` [RFC PATCH 6/8] sched: Report local wake up on resched blind zone within idle loop Peter Zijlstra
2021-01-11 12:56 ` Frederic Weisbecker
2021-01-12 8:24 ` [sched] 9720a64438: WARNING:at_kernel/sched/core.c:#sched_resched_local_assert_allowed kernel test robot
[not found] ` <161062476680.19482.8402362019173198799@build.alporthouse.com>
2021-01-14 15:19 ` [RFC PATCH 6/8] sched: Report local wake up on resched blind zone within idle loop Paul E. McKenney
2021-01-09 2:05 ` [RFC PATCH 7/8] entry: Report local wake up on resched blind zone while resuming to user Frederic Weisbecker
2021-01-13 2:46 ` [entry] 8e01c5f104: unixbench.score -2.2% regression kernel test robot
2021-01-09 2:05 ` [RFC PATCH 8/8] timer: Report ignored local enqueue in nohz mode Frederic Weisbecker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210109020536.127953-7-frederic@kernel.org \
--to=frederic@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=rafael.j.wysocki@intel.com \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).