[PATCH] rcu: Make rcu_normal_after_boot writable on RT

[PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Juri Lelli]

Certain configurations (e.g., systems that make heavy use of netns)
need to use synchronize_rcu_expedited() to service RCU grace periods
even after boot.

Even though synchronize_rcu_expedited() has been traditionally
considered harmful for RT for the heavy use of IPIs, it is perfectly
usable under certain conditions (e.g. nohz_full).

Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep
its default value to 1 (enabled) to avoid regressions. Users who need
synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_
boot=0 in the kernel cmdline.

Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
---
 kernel/rcu/update.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index c21b38cc25e9..0fdbf937edac 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -57,9 +57,7 @@
 module_param(rcu_expedited, int, 0);
 module_param(rcu_normal, int, 0);
 static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
-#ifndef CONFIG_PREEMPT_RT
 module_param(rcu_normal_after_boot, int, 0);
-#endif
 #endif /* #ifndef CONFIG_TINY_RCU */
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
-- 
2.31.1

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Paul E. McKenney]

On Thu, Aug 05, 2021 at 10:01:23AM +0200, Juri Lelli wrote:
> Certain configurations (e.g., systems that make heavy use of netns)
> need to use synchronize_rcu_expedited() to service RCU grace periods
> even after boot.
> 
> Even though synchronize_rcu_expedited() has been traditionally
> considered harmful for RT for the heavy use of IPIs, it is perfectly
> usable under certain conditions (e.g. nohz_full).
> 
> Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep
> its default value to 1 (enabled) to avoid regressions. Users who need
> synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_
> boot=0 in the kernel cmdline.
> 
> Signed-off-by: Juri Lelli <juri.lelli@redhat.com>

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by?
For example, maybe they would prefer this kernel boot parameter to be
exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
situations where rcu_normal_after_boot makes sense?

							Thanx, Paul

> ---
>  kernel/rcu/update.c | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
> index c21b38cc25e9..0fdbf937edac 100644
> --- a/kernel/rcu/update.c
> +++ b/kernel/rcu/update.c
> @@ -57,9 +57,7 @@
>  module_param(rcu_expedited, int, 0);
>  module_param(rcu_normal, int, 0);
>  static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
> -#ifndef CONFIG_PREEMPT_RT
>  module_param(rcu_normal_after_boot, int, 0);
> -#endif
>  #endif /* #ifndef CONFIG_TINY_RCU */
>  
>  #ifdef CONFIG_DEBUG_LOCK_ALLOC
> -- 
> 2.31.1
> 

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Paul E. McKenney]

On Thu, Aug 05, 2021 at 09:03:37AM -0700, Paul E. McKenney wrote:
> On Thu, Aug 05, 2021 at 10:01:23AM +0200, Juri Lelli wrote:
> > Certain configurations (e.g., systems that make heavy use of netns)
> > need to use synchronize_rcu_expedited() to service RCU grace periods
> > even after boot.
> > 
> > Even though synchronize_rcu_expedited() has been traditionally
> > considered harmful for RT for the heavy use of IPIs, it is perfectly
> > usable under certain conditions (e.g. nohz_full).
> > 
> > Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep
> > its default value to 1 (enabled) to avoid regressions. Users who need
> > synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_
> > boot=0 in the kernel cmdline.
> > 
> > Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
> 
> Makes sense to me!
> 
> But would another of the -rt people be willing to give an Acked-by?
> For example, maybe they would prefer this kernel boot parameter to be
> exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
> situations where rcu_normal_after_boot makes sense?

Ah, and this will also need to be reflected in the WARN_ON_ONCE()
in synchronize_rcu_expedited_wait() in kernel/rcu/tree_exp.h.

							Thanx, Paul

> > ---
> >  kernel/rcu/update.c | 2 --
> >  1 file changed, 2 deletions(-)
> > 
> > diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
> > index c21b38cc25e9..0fdbf937edac 100644
> > --- a/kernel/rcu/update.c
> > +++ b/kernel/rcu/update.c
> > @@ -57,9 +57,7 @@
> >  module_param(rcu_expedited, int, 0);
> >  module_param(rcu_normal, int, 0);
> >  static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
> > -#ifndef CONFIG_PREEMPT_RT
> >  module_param(rcu_normal_after_boot, int, 0);
> > -#endif
> >  #endif /* #ifndef CONFIG_TINY_RCU */
> >  
> >  #ifdef CONFIG_DEBUG_LOCK_ALLOC
> > -- 
> > 2.31.1
> > 

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Juri Lelli]

Hi,

Thanks for reviewing Paul.

On 05/08/21 14:08, Paul E. McKenney wrote:
> On Thu, Aug 05, 2021 at 09:03:37AM -0700, Paul E. McKenney wrote:
> > On Thu, Aug 05, 2021 at 10:01:23AM +0200, Juri Lelli wrote:
> > > Certain configurations (e.g., systems that make heavy use of netns)
> > > need to use synchronize_rcu_expedited() to service RCU grace periods
> > > even after boot.
> > > 
> > > Even though synchronize_rcu_expedited() has been traditionally
> > > considered harmful for RT for the heavy use of IPIs, it is perfectly
> > > usable under certain conditions (e.g. nohz_full).
> > > 
> > > Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep
> > > its default value to 1 (enabled) to avoid regressions. Users who need
> > > synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_
> > > boot=0 in the kernel cmdline.
> > > 
> > > Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
> > 
> > Makes sense to me!
> > 
> > But would another of the -rt people be willing to give an Acked-by?
> > For example, maybe they would prefer this kernel boot parameter to be
> > exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
> > situations where rcu_normal_after_boot makes sense?
> 
> Ah, and this will also need to be reflected in the WARN_ON_ONCE()
> in synchronize_rcu_expedited_wait() in kernel/rcu/tree_exp.h.

Indeed. Will add the change as soon as I receive indication about your
first point.

Best,
Juri

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Sebastian Andrzej Siewior]

On 2021-08-05 09:03:37 [-0700], Paul E. McKenney wrote:
> Makes sense to me!
> 
> But would another of the -rt people be willing to give an Acked-by?
> For example, maybe they would prefer this kernel boot parameter to be
> exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
> situations where rcu_normal_after_boot makes sense?

Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more
and more synchronize_rcu_expedited() users popping up. I would like to
keep that part (default value) since it good to have for most users.

I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate
use cases for using "rcu_normal_after_boot = 0".
Paul suggested initially to restrict that option for PREEMPT_RT and I
would follow here Paul's guidance to either remove it or restrict it to
NO_HZ_FULL in RT's case (as suggested).

> 							Thanx, Paul

Sebastian

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Paul E. McKenney]

On Fri, Aug 06, 2021 at 10:04:55AM +0200, Sebastian Andrzej Siewior wrote:
> On 2021-08-05 09:03:37 [-0700], Paul E. McKenney wrote:
> > Makes sense to me!
> > 
> > But would another of the -rt people be willing to give an Acked-by?
> > For example, maybe they would prefer this kernel boot parameter to be
> > exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
> > situations where rcu_normal_after_boot makes sense?
> 
> Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more
> and more synchronize_rcu_expedited() users popping up. I would like to
> keep that part (default value) since it good to have for most users.
> 
> I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate
> use cases for using "rcu_normal_after_boot = 0".
> Paul suggested initially to restrict that option for PREEMPT_RT and I
> would follow here Paul's guidance to either remove it or restrict it to
> NO_HZ_FULL in RT's case (as suggested).

Given what I know now, I suggest the following:

o	Restrict the option to !PREEMPT_RT unless NO_HZ_FULL.
	Maybe "!defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL)".

	If there is some non-NO_HZ_FULL PREEMPT_RT configuration that
	tolerates expedited grace periods, this would need to change.

o	Change the permissions from "0" to "0444", if desired.	If you
	would rather not, I can do this in a follow-up patch.  (No idea
	why I let such an ugly serviceability issue through, but the
	previous pair of module_param() instances have the same problem.)

Anything I am missing?

							Thanx, Paul

Re: [PATCH] rcu: Make rcu_normal_after_boot writable on RT

[Juri Lelli]

On 06/08/21 10:44, Paul E. McKenney wrote:
> On Fri, Aug 06, 2021 at 10:04:55AM +0200, Sebastian Andrzej Siewior wrote:
> > On 2021-08-05 09:03:37 [-0700], Paul E. McKenney wrote:
> > > Makes sense to me!
> > > 
> > > But would another of the -rt people be willing to give an Acked-by?
> > > For example, maybe they would prefer this kernel boot parameter to be
> > > exposed only if (!PREEMPT_RT || NO_HZ_FULL).  Or are there !NO_HZ_FULL
> > > situations where rcu_normal_after_boot makes sense?
> > 
> > Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more
> > and more synchronize_rcu_expedited() users popping up. I would like to
> > keep that part (default value) since it good to have for most users.
> > 
> > I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate
> > use cases for using "rcu_normal_after_boot = 0".
> > Paul suggested initially to restrict that option for PREEMPT_RT and I
> > would follow here Paul's guidance to either remove it or restrict it to
> > NO_HZ_FULL in RT's case (as suggested).
> 
> Given what I know now, I suggest the following:
> 
> o	Restrict the option to !PREEMPT_RT unless NO_HZ_FULL.
> 	Maybe "!defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL)".
> 
> 	If there is some non-NO_HZ_FULL PREEMPT_RT configuration that
> 	tolerates expedited grace periods, this would need to change.
> 
> o	Change the permissions from "0" to "0444", if desired.	If you
> 	would rather not, I can do this in a follow-up patch.  (No idea
> 	why I let such an ugly serviceability issue through, but the
> 	previous pair of module_param() instances have the same problem.)
> 
> Anything I am missing?

Not that I can think of right now. :)

Will implement your suggestions and submit v2 soon. Thank again to you
and Sebastian for the review!

Best,
Juri