Re: [PATCH rcu 11/14] rcu: Make rcu_normal_after_boot writable again

From: "Paul E. McKenney" <paulmck@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: rcu@vger.kernel.org, linux-kernel@vger.kernel.org,
	kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com,
	akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
	josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org,
	dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com,
	oleg@redhat.com, joel@joelfernandes.org,
	Juri Lelli <juri.lelli@redhat.com>
Subject: Re: [PATCH rcu 11/14] rcu: Make rcu_normal_after_boot writable again
Date: Thu, 16 Sep 2021 06:57:12 -0700	[thread overview]
Message-ID: <20210916135712.GB4156@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <YULyiT+RbAgHxO7u@hirez.programming.kicks-ass.net>

On Thu, Sep 16, 2021 at 09:30:17AM +0200, Peter Zijlstra wrote:
> On Wed, Sep 15, 2021 at 04:33:40PM -0700, Paul E. McKenney wrote:
> > From: Juri Lelli <juri.lelli@redhat.com>
> > 
> > Certain configurations (e.g., systems that make heavy use of netns)
> > need to use synchronize_rcu_expedited() to service RCU grace periods
> > even after boot.
> > 
> > Even though synchronize_rcu_expedited() has been traditionally
> > considered harmful for RT for the heavy use of IPIs, it is perfectly
> > usable under certain conditions (e.g. nohz_full).
> > 
> > Make rcupdate.rcu_normal_after_boot= again writeable on RT (if NO_HZ_
> > FULL is defined), but keep its default value to 1 (enabled) to avoid
> > regressions. Users who need synchronize_rcu_expedited() will boot with
> > rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline.
> > 
> > Reflect the change in synchronize_rcu_expedited_wait() by removing the
> > WARN related to CONFIG_PREEMPT_RT.
> > 
> > Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > ---
> >  kernel/rcu/tree_exp.h | 1 -
> >  kernel/rcu/update.c   | 2 +-
> >  2 files changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > index 2796084ef85a..d9e4f8eb9ae2 100644
> > --- a/kernel/rcu/tree_exp.h
> > +++ b/kernel/rcu/tree_exp.h
> > @@ -512,7 +512,6 @@ static void synchronize_rcu_expedited_wait(void)
> >  		j = READ_ONCE(jiffies_till_first_fqs);
> >  		if (synchronize_rcu_expedited_wait_once(j + HZ))
> >  			return;
> > -		WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT));
> >  	}
> >  
> >  	for (;;) {
> 
> Urgh... I really don't like this. That would force RT applications to
> always use NOHZ_FULL which isn't always possible or desired.

???

RT systems they avoid expedited grace periods by booting with either
the rcupdate.rcu_normal or the rcupdate.rcu_normal_after_boot kernel
boot parameters.  And here is the definition for the latter:

static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);

In other words, RT systems shut off expedited grace periods by default,
and are thus free to use nohz_full CPU or not, as they choose.  When using
nohz_full, they can also enable expedited grace periods by booting with
rcupdate.rcu_normal_after_boot=0.  Or not, sysadm's choice.

So I am not seeing a problem here.  What am I missing?

> WTH does that netns crud need expedited so much?

Large systems with lots of networking configurations can take quite some
time to set things up due to waiting for many grace periods.  In addition,
last I checked, expedited grace periods were used to reduce contention
on RTNL.

If you have a modest-sized system with less elaborate networking setup
and make use of the batched network-configuration mechanisms, there
should be no problem using normal grace periods.

Which has been the set of constraints that RT has been living with for
quite some time.

Again, what am I missing?

							Thanx, Paul