Re: [PATCH tip/core/rcu 04/13] rcu: Make RCU_FANOUT_LEAF help text more explicit about skew_tick

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org,
	jiangshanlai@gmail.com, dipankar@in.ibm.com,
	akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
	josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org,
	dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com,
	oleg@redhat.com, bobby.prani@gmail.com
Subject: Re: [PATCH tip/core/rcu 04/13] rcu: Make RCU_FANOUT_LEAF help text more explicit about skew_tick
Date: Thu, 13 Apr 2017 10:31:00 -0700	[thread overview]
Message-ID: <20170413173100.GL3956@linux.vnet.ibm.com> (raw)
In-Reply-To: <20170413170434.xk4zq3p75pu3ubxw@hirez.programming.kicks-ass.net>

On Thu, Apr 13, 2017 at 07:04:34PM +0200, Peter Zijlstra wrote:
> On Thu, Apr 13, 2017 at 09:55:16AM -0700, Paul E. McKenney wrote:
> > > > To avoid people tuning huge machines having to wait for me to give
> > > > them an answer as to why they are suffering lock contention after
> > > > cranking up the value of RCU_FANOUT_LEAF.
> 
> So is there a good reason to increase FANOUT_LEAF ?

Increasing it reduces the number of rcu_node structures, and thus the
number of cache misses during grace-period initialization and cleanup.
This has proven necessary in the past on large machines having long
memory latencies.  And there are starting to be some pretty big machines
running in production, and even for typical commerical workloads.

> > > > Or am I missing your point?
> > > 
> > > Your answer should be: don't do that then. Not provide them a shady work
> > > around.
> > > 
> > > tick skew isn't pretty and has other problems (there's a reason its not
> > > on by default). You're then doing two things you shouldn't.
> > 
> > The tick skew problem that I know of is energy efficiency for light
> > workloads.  This doesn't normally apply to the large heavily loaded
> > systems on which people skew ticks.
> > 
> > So what are the other problems?
> 
> If the jiffy updater bounces between CPUs (as is not uncommon) the
> duration of the jiffy becomes an average (I think we fixed it where it
> could go too fast by always jumping to a CPU which has a short jiffy,
> but I'm not sure).
> 
> This further complicates some of the jiffy based loops (which arguably
> should go away anyway).

Fair enough.  And I am OK with jiffies going away as long as there
is a low-overhead rough-and-ready timing mechanism replacing it.

> And I have vague memories of it actually causing lock contention, but
> I've forgotten how that worked.

That is a new one on me.  I can easily see how not skewing ticks could
cause serious lock contention, but am missing how skewed ticks would
do so.

							Thanx, Paul