rcu.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RCU: rcu stall issues and an approach to the fix
@ 2021-07-22 20:08 donghai qiao
  2021-07-22 20:41 ` Paul E. McKenney
  2021-07-23  0:29 ` Boqun Feng
  0 siblings, 2 replies; 26+ messages in thread
From: donghai qiao @ 2021-07-22 20:08 UTC (permalink / raw)
  To: rcu, donghai qiao

RCU experts,

When you reply, please also keep me CC'ed.

The problem of RCU stall might be an old problem and it can happen quite often.
As I have observed, when the problem occurs,  at least one CPU in the system
on which its rdp->gp_seq falls behind others by 4 (qs).

e.g.  On CPU 0, rdp->gp_seq = 0x13889d, but on other CPUs, their
rdp->gp_seq = 0x1388a1.

Because RCU stall issues can last a long period of time, the number of callbacks
in the list rdp->cblist of all CPUs can accumulate to thousands. In
the worst case,
it triggers panic.

When looking into the problem further, I'd think the problem is related to the
Linux scheduler. When the RCU core detects the stall on a CPU, rcu_gp_kthread
would send a rescheduling request via send_IPI to that CPU to try to force a
context switch to make some progress. However, at least one situation can fail
this effort, which is when the CPU is running a user thread and it is the only
user thread in the rq, then this attempted context switching will not happen
immediately. In particular if the system is also configured with NOHZ_FULL for
the CPU and as long as the user thread is running, the forced context
switch will
never happen unless the user thread volunteers to yield the CPU. I think this
should be one of the major root causes of these RCU stall issues. Even if
NOHZ_FULL is not configured, there will be at least 1 tick delay which can
affect the realtime kernel, by the way.

But it seems not a good idea to craft a fix from the scheduler side because
this has to invalidate some existing scheduling optimizations. The current
scheduler is deliberately optimized to avoid such context switching.  So my
question is why the RCU core cannot effectively update qs for the stalled CPU
when it detects that the stalled CPU is running a user thread?  The reason
is pretty obvious because when a CPU is running a user thread, it must not
be in any kernel read-side critical sections. So it should be safe to close
its current RCU grace period on this CPU. Also, with this approach we can make
RCU work more efficiently than the approach of context switch which needs to
go through an IPI interrupt and the destination CPU needs to wake up its
ksoftirqd or wait for the next scheduling cycle.

If my suggested approach makes sense, I can go ahead to fix it that way.

Thanks
Donghai

^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, other threads:[~2021-10-21 16:44 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-22 20:08 RCU: rcu stall issues and an approach to the fix donghai qiao
2021-07-22 20:41 ` Paul E. McKenney
2021-07-23  0:29 ` Boqun Feng
2021-07-23  3:49   ` Paul E. McKenney
     [not found]     ` <CAOzhvcOLFzFGZAptOTrP9Xne1-LiO8jka1sPF6+0=WiLh-cQUA@mail.gmail.com>
2021-07-23 17:25       ` Paul E. McKenney
2021-07-23 18:41         ` donghai qiao
2021-07-23 19:06           ` Paul E. McKenney
2021-07-24  0:01             ` donghai qiao
2021-07-25  0:48               ` Paul E. McKenney
2021-07-27 22:34                 ` donghai qiao
2021-07-28  0:10                   ` Paul E. McKenney
2021-10-04 21:22                     ` donghai qiao
2021-10-05  0:59                       ` Paul E. McKenney
2021-10-05 16:10                         ` donghai qiao
2021-10-05 16:39                           ` Paul E. McKenney
2021-10-06  0:25                             ` donghai qiao
2021-10-18 21:18                               ` donghai qiao
2021-10-18 23:46                                 ` Paul E. McKenney
2021-10-20 17:48                                   ` donghai qiao
2021-10-20 18:37                                     ` Paul E. McKenney
2021-10-20 20:05                                       ` donghai qiao
2021-10-20 21:33                                         ` Paul E. McKenney
2021-10-21  3:25                                           ` Zhouyi Zhou
2021-10-21  4:17                                             ` Paul E. McKenney
2021-10-21 16:44                                           ` donghai qiao
2021-07-23 17:25     ` donghai qiao

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).