Re: RCU stall query

From: "Paul E. McKenney" <paulmck@kernel.org>
To: John Garry <john.garry@huawei.com>
Cc: "josh@joshtriplett.org" <josh@joshtriplett.org>,
	"rostedt@goodmis.org" <rostedt@goodmis.org>,
	"mathieu.desnoyers@efficios.com" <mathieu.desnoyers@efficios.com>,
	"jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
	Joel Fernandes <joel@joelfernandes.org>,
	"rcu@vger.kernel.org" <rcu@vger.kernel.org>
Subject: Re: RCU stall query
Date: Thu, 22 Apr 2021 09:02:33 -0700	[thread overview]
Message-ID: <20210422160233.GF975577@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <412926e8-d3e1-3071-8cb9-098a7f49b64c@huawei.com>

On Thu, Apr 22, 2021 at 04:45:00PM +0100, John Garry wrote:
> On 22/04/2021 15:35, Paul E. McKenney wrote:
> > On Thu, Apr 22, 2021 at 10:20:51AM +0100, John Garry wrote:
> > > Hi RCU experts,
> > > 
> 
> Thanks Paul
> 
> > > Recently I have noticed that I can trigger an RCU stall quite easily on my
> > > system under specific conditions.
> > > 
> > > I have a fair idea why it happens, but need to analyze a proper solution
> > > further. It looks like a hard IRQ handler and threaded part are tied to
> > > specific CPU and getting swamped and not relinquishing.

I should hasten to confirm that saturating a CPU with interrupts can
also result in RCU CPU stall warnings, so please do continue your
efforts fixing this as well.

> > > However, mixed in the RCU splats, I have noticed many BUG logs, like:
> > > 
> > > [  207.788748] BUG: spinlock recursion on CPU#46, fio/1470
> > This is a self-deadlock.  Given that deadlock, and given that spinlocks
> > disable preemption, the RCU CPU stall warnings are expected behavior.
> > After all, your code really is grabbing a CPU by the throat and shaking
> > it indefinitely.
> > 
> > Please build your kernel with CONFIG_PROVE_LOCKING=y and then fix the
> > issues it calls out.  Then please also fix the bugs resulting in the
> > "sleeping function called from invalid context" and in the "scheduling
> > while atomic".
> 
> Here's the rub, the issue goes away with CONFIG_PROVE_LOCKING and all the
> extra debugging it adds. Hmmm.

That can happpen.  You have enough going on that fixing what you already
know about might eventually get things to where CONFIG_PROVE_LOCKING
does something useful to you.

							Thanx, Paul

> But I get the point that these are separate and need to be fixed also.
> 
> > 
> > In addition, there are quite a few idle tasks called out in your list of
> > stalled CPUs.  This is often due to RCU's grace-period kthread (named
> > "rcu_preempt" in this case) not getting any CPU time.  This is not
> > unexpected given the "RT throttling activated".  If you are going to run
> > code at real-time priorities, you must ensure that any number of kernel
> > kthreads get the CPU time that they need.  As Spiderman's uncle said
> > "With great power comes great responsibility".
> 
> OK, I need to check on that separately also.
> 
> Cheers,
> John