Re: [PATCH RFC tip/core/rcu] Eliminate deadlock between CPU hotplug and expedited grace periods

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org,
	laijs@cn.fujitsu.com, dipankar@in.ibm.com,
	akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
	josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org,
	dhowells@redhat.com, edumazet@google.com, dvhart@linux.intel.com,
	fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com,
	<""@rjwysocki.net>,
	tianyu.lan@intel.com
Subject: Re: [PATCH RFC tip/core/rcu] Eliminate deadlock between CPU hotplug and expedited grace periods
Date: Wed, 3 Sep 2014 09:38:52 -0700	[thread overview]
Message-ID: <20140903163852.GY5001@linux.vnet.ibm.com> (raw)
In-Reply-To: <20140903152850.GQ4783@worktop.ger.corp.intel.com>

On Wed, Sep 03, 2014 at 05:28:50PM +0200, Peter Zijlstra wrote:
> On Wed, Sep 03, 2014 at 08:03:06AM -0700, Paul E. McKenney wrote:
> > > > Normal RCU grace periods avoid this by synchronizing on a lock acquired by
> > > > the RCU CPU-hotplug notifiers, but this does not work for the expedited
> > > > grace periods because the outgoing CPU can be running random tasks for
> > > > quite some time after RCU's notifier executes.  So the fix is just to
> > > > drop back to a normal grace period when there is a CPU-hotplug operation
> > > > in progress.
> > > 
> > > So why are we 'normally' doing an expedited call here anyhow? 
> > 
> > Presumably because they set either the boot parameter or
> > the sysfs variable that causes synchronize_sched() to so
> > synchronize_sched_expedited().
> 
> That's not a why but a how. Why does that option exist, why are we doing
> this?

As you say below, to reduce RCU grace-period latency on small systems.
And one level of abstraction's why is another level's how.  ;-)

> I cannot actually find a sysfs variable that controls this though; only
> the rcu_pm_notifier. It seems to favour doing an expedited call when
> suspending on 'small' machines.

See rcu_expedited_store() in kernel/ksysfs.c.  Or the rcu_expedited
module_param() in kernel/rcu/update.c.

> > > But those are not within hotplug bits. Also weren't we removing them? I
> > > thought we didn't appreciate spraying IPIs like they do?
> > 
> > I hadn't heard anything about removing them, but making the
> > expedited primitives a bit less IPI-happy is on my list.
> 
> I had some recollections of removing a fair number of expedited calls,
> but its was a long while ago so what do I know ;-)

If a given use case can tolerate the latency of normal calls, then the
normal calls certainly are preferable, no two ways about it.

> Making them less IPI happy would be good indeed.

Priority of this task duly increased.  ;-)

							Thanx, Paul