Re: [PATCH rcu 3/3] rcu: Allow expedited RCU grace periods on incoming CPUs

From: "Paul E. McKenney" <paulmck@kernel.org>
To: kernel-team@fb.com, linux-kernel@vger.kernel.org,
	quic_mojha@quicinc.com, rcu@vger.kernel.org, rostedt@goodmis.org,
	tj@kernel.org, Peter Zijlstra <peterz@infradead.org>
Subject: Re: [PATCH rcu 3/3] rcu: Allow expedited RCU grace periods on incoming CPUs
Date: Mon, 14 Feb 2022 08:44:35 -0800	[thread overview]
Message-ID: <20220214164435.GA2805255@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <20220209233811.GC557593@lothringen>

On Thu, Feb 10, 2022 at 12:38:11AM +0100, Frederic Weisbecker wrote:
> On Fri, Feb 04, 2022 at 02:55:07PM -0800, Paul E. McKenney wrote:
> > Although it is usually safe to invoke synchronize_rcu_expedited() from a
> > preemption-enabled CPU-hotplug notifier, if it is invoked from a notifier
> > between CPUHP_AP_RCUTREE_ONLINE and CPUHP_AP_ACTIVE, its attempts to
> > invoke a workqueue handler will hang due to RCU waiting on a CPU that
> > the scheduler is not paying attention to.  This commit therefore expands
> > use of the existing workqueue-independent synchronize_rcu_expedited()
> > from early boot to also include CPUs that are being hotplugged.
> > 
> > Link: https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@quicinc.com/
> > Reported-by: Mukesh Ojha <quic_mojha@quicinc.com>
> > Cc: Tejun Heo <tj@kernel.org>
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> 
> I'm surprised by this scheduler behaviour.
> 
> Since sched_cpu_activate() hasn't been called yet,
> rq->balance_callback = balance_push_callback. As a result, balance_push() should
> be called at the end of schedule() when the workqueue is picked as the next task.
> Then eventually the workqueue should be immediately preempted by the stop task to
> be migrated elsewhere.
> 
> So I must be missing something. For the fun, I booted the following and it
> didn't produce any issue:
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 80faf2273ce9..b1e74a508881 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4234,6 +4234,8 @@ int rcutree_online_cpu(unsigned int cpu)
>  
>  	// Stop-machine done, so allow nohz_full to disable tick.
>  	tick_dep_clear(TICK_DEP_BIT_RCU);
> +	if (cpu != 0)
> +		synchronize_rcu_expedited();
>  	return 0;
>  }

That does seem compelling.  And others have argued that the workqueue
system's handling of offline CPUs should deal with this.

Mukesh, was this a theoretical bug, or did you actually make it happen?
If you made it happen, as seems to have been the case given your original
email [1], could you please post your reproducer?

							Thanx, Paul

[1] https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@quicinc.com/