[4.19.103-rt42] rcu_preempt detected stalls

[4.19.103-rt42] rcu_preempt detected stalls

[Tim Blechmann]

hi all,

updating the RT kernel from 4.14 to 4.19, we're experiencing a lockup of
of RT threads:

"Worker 1" and "Worker 2" are my RT threads, which run at high
SCHED_FIFO priorities. they are both synchronised via the same futex.
wildly speculating, we might run into a race condition between
`futex_wake` while `try_to_wake_up -> futex_wait`

is this a known issue (it seems that rt48 has some RT related changes)?
or is there a good way to trace this down? cyclictest doesn't seem to
trigger this situation, with our real-world load, it takes hours to
trigger it.

thanks a lot,
tim

---

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu:         Tasks blocked on level-0 rcu_node (CPUs 0-3): P693 P692
rcu:         (detected by 0, t=21002 jiffies, g=463681, q=2032)
Worker 2   D    0   693      1 0x00000000
Call Trace:
 ? __schedule+0x2cf/0x770
 schedule+0x43/0xe0
 rt_spin_lock_slowlock_locked+0x11b/0x2c0
 rt_spin_lock_slowlock+0x41/0x60
 futex_wake+0xa2/0x150
 do_futex+0x942/0xd90
 ? trace_hardirqs_on_thunk+0x1a/0x1c
 ? do_preempt_schedule_irq+0x17/0x17
 ? do_futex+0xd90/0xd90
 __x64_sys_futex+0x136/0x14f
 ? do_syscall_64+0x1e/0x3f0
 do_syscall_64+0x5d/0x3f0
 ? trace_hardirqs_off_thunk+0x1a/0x1c
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f9f9e9a0311
Code: ff ff ff 7f 74 4b 48 8d 50 01 f0 48 0f b1 17 75 ee 48 c1 e8 20 74
1b 40 80 f6 81 45 31 d2 ba 01 00 00 00 b8 ca 00 00 00 0f 05 <48> 3d 00
f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 83 c0 16 83 e0 f7 74
RSP: 002b:00007f9f7affca28 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9f9e9a0311
RDX: 0000000000000001 RSI: 0000000000000081 RDI: 000055b8f8c20a88
RBP: 0000000000000001 R08: 000055b9072f2820 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000055b8f8c20a88
R13: 00007f9f7affca97 R14: 00007f9f7affca98 R15: 00007f9f7affcaa0
Worker 1   R  running task        0   692      1 0x00000000
Call Trace:
 ? __schedule+0x2cf/0x770
 ? ___preempt_schedule+0x16/0x18
 preempt_schedule_common+0x32/0xc0
 ___preempt_schedule+0x16/0x18
 ? migrate_disable+0x38/0xc0
 ? migrate_disable+0x38/0xc0
 migrate_disable+0x6c/0xc0
 get_futex_value_locked+0x14/0x60
 futex_wait_setup+0x80/0x100
 ? futex_wait_queue_me+0xbf/0x120
 futex_wait+0xb9/0x210
 ? try_to_wake_up+0x230/0x6c0
 do_futex+0x134/0xd90
 ? trace_hardirqs_on_thunk+0x1a/0x1c
 ? do_preempt_schedule_irq+0x17/0x17
 ? trace_hardirqs_on_thunk+0x1a/0x1c
 ? trace_hardirqs_on_caller+0x32/0x130
 ? trace_hardirqs_on_thunk+0x1a/0x1c
 ? do_preempt_schedule_irq+0x17/0x17
 __x64_sys_futex+0x136/0x14f
 ? do_syscall_64+0x1e/0x3f0
 do_syscall_64+0x5d/0x3f0
 ? trace_hardirqs_off_thunk+0x1a/0x1c
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f9f9e99fd54
Code: e8 41 0b 00 00 b9 ca 00 00 00 41 89 c4 89 de 41 b9 ff ff ff ff 45
31 c0 81 f6 89 01 00 00 45 31 d2 31 d2 48 89 ef 89 c8 0f 05 <48> 3d 00
f0 ff ff 77 14 44 89 e7 e8 6c 0b 00 00 5b 31 c0 5d 41 5c
RSP: 002b:00007f9f7b7fda80 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9f9e99fd54
RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055b8f8c20a88
RBP: 000055b8f8c20a88 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: fffffffeffffffff R14: 000055b8f8c20a88 R15: 00007f9f7b7fe700