* BUG-RT: scheduling while in atomic in the watchdog's hrtimer
@ 2019-03-21 21:27 Daniel Bristot de Oliveira
2019-03-21 21:46 ` Peter Zijlstra
0 siblings, 1 reply; 2+ messages in thread
From: Daniel Bristot de Oliveira @ 2019-03-21 21:27 UTC (permalink / raw)
To: linux-rt-users
Cc: Sebastian Andrzej Siewior, Peter Zijlstra, Clark Williams,
Tommaso Cucinotta, Rômulo Silva de Oliveira, LKML
Hi,
In the 4.19/5.0-rt configured !SMP, the following chain of events can
take place:
------------------------------ %< -------------------------------------
smp_apic_timer_interrupt(){
hrtimer_interrupt() {
__hrtimer_run_queues() {
watchdog_timer_fn() {
stop_one_cpu_nowait() {
#ifdef !CONFIG_SMP
schedule_work() {
queue_work() {
queue_work_on() {
/* phew, long, ah!?.... */
local_lock_irqsave() {
__local_lock_irqsave() {
__local_lock_irq() {
spin_lock_irqsave() {
rt_spin_lock()...
/* from here on you already know...*/
/* a lot of } */
------------------------------ >% -------------------------------------
And this can cause a scheduling while in atomic.
Tracking down, the commit that introduced the stop_one_cpu_nowait()->
queue_work_on() to the path was:
------------------------------ %< -------------------------------------
commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra
Date: Thu Jun 7 10:52:03 2018 +0200
watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
Oleg suggested to replace the "watchdog/%u" threads with
cpu_stop_work. That removes one thread per CPU while at the same time
fixes softlockup vs SCHED_DEADLINE.
But more importantly, it does away with the single
smpboot_update_cpumask_percpu_thread() user, which allows
cleanups/shrinkage of the smpboot interface.
Suggested-by: Oleg Nesterov
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------
Later modified by:
------------------------------ %< -------------------------------------
commit be45bf5395e0886a93fc816bbe41a008ec2e42e2
Author: Peter Zijlstra
Date: Fri Jul 13 12:42:08 2018 +0200
watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug
When scheduling is delayed for longer than the softlockup interrupt
period it is possible to double-queue the cpu_stop_work, causing list
corruption.
Cure this by adding a completion to track the cpu_stop_work's
progress.
Reported-by: kernel test robot
Tested-by: Rong Chen
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work")
Link: http://lkml.kernel.org/r/20180713104208.GW2494@hirez.programming.kicks-ass.net
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------
I hit it while trying the automata model against the 4.19-rt/5.0-rt
kernels.
You can find more about it here:
http://bristot.me/another-bug-found-with-the-model/
Thanks
-- Daniel
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: BUG-RT: scheduling while in atomic in the watchdog's hrtimer
2019-03-21 21:27 BUG-RT: scheduling while in atomic in the watchdog's hrtimer Daniel Bristot de Oliveira
@ 2019-03-21 21:46 ` Peter Zijlstra
0 siblings, 0 replies; 2+ messages in thread
From: Peter Zijlstra @ 2019-03-21 21:46 UTC (permalink / raw)
To: Daniel Bristot de Oliveira
Cc: linux-rt-users, Sebastian Andrzej Siewior, Clark Williams,
Tommaso Cucinotta, Rômulo Silva de Oliveira, LKML
On Thu, Mar 21, 2019 at 10:27:30PM +0100, Daniel Bristot de Oliveira wrote:
> Hi,
>
> In the 4.19/5.0-rt configured !SMP, the following chain of events can
> take place:
>
> ------------------------------ %< -------------------------------------
> smp_apic_timer_interrupt(){
> hrtimer_interrupt() {
> __hrtimer_run_queues() {
> watchdog_timer_fn() {
> stop_one_cpu_nowait() {
> #ifdef !CONFIG_SMP
ARGGHH..
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2019-03-21 21:46 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-21 21:27 BUG-RT: scheduling while in atomic in the watchdog's hrtimer Daniel Bristot de Oliveira
2019-03-21 21:46 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).