linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
@ 2015-10-26 19:14 Josh Cartwright
  2015-10-27  0:44 ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Josh Cartwright @ 2015-10-26 19:14 UTC (permalink / raw)
  To: tglx, bigeasy
  Cc: linux-rt-users, linux-kernel, Eric Dumazet, Paul E. McKenney,
	David S. Miller

This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41.

While the use of synchronize_rcu_expedited() might make
synchronize_net() "faster", it does so at significant cost on RT
systems, as expediting a grace period forcibly preempts any
high-priority RT tasks (via the stop_machine() mechanism).

Without be3fc413da9e reverted, we can observe a latency spike up to 30us
with cyclictest by rapidly unplugging/reestablishing an ethernet link.

Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Signed-off-by: Josh Cartwright <joshc@ni.com>
---
 net/core/dev.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index f8c23de..869ef62 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -6969,10 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
 void synchronize_net(void)
 {
 	might_sleep();
-	if (rtnl_is_locked())
-		synchronize_rcu_expedited();
-	else
-		synchronize_rcu();
+	synchronize_rcu();
 }
 EXPORT_SYMBOL(synchronize_net);
 
-- 
2.5.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-26 19:14 [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" Josh Cartwright
@ 2015-10-27  0:44 ` Paul E. McKenney
  2015-10-27 12:31   ` Josh Cartwright
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2015-10-27  0:44 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: tglx, bigeasy, linux-rt-users, linux-kernel, Eric Dumazet,
	David S. Miller

On Mon, Oct 26, 2015 at 02:14:55PM -0500, Josh Cartwright wrote:
> This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41.
> 
> While the use of synchronize_rcu_expedited() might make
> synchronize_net() "faster", it does so at significant cost on RT
> systems, as expediting a grace period forcibly preempts any
> high-priority RT tasks (via the stop_machine() mechanism).
> 
> Without be3fc413da9e reverted, we can observe a latency spike up to 30us
> with cyclictest by rapidly unplugging/reestablishing an ethernet link.
> 
> Cc: Eric Dumazet <eric.dumazet@gmail.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: David S. Miller <davem@davemloft.net>
> Signed-off-by: Josh Cartwright <joshc@ni.com>

Hmmm...  If I remember correctly, using expedited here resulted
in impressive performance improvements in some important cases.
Perhaps things have changed (I must defer to Eric), but if not, how
about something like this instead?

	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREMPT_RT_FULL))
		synchronize_rcu_expedited();
	else
		synchronize_rcu();

Alternatively, a boot-time option could be used:

	if (rtnl_is_locked() && !some_rt_boot_parameter)
		synchronize_rcu_expedited();
	else
		synchronize_rcu();

I believe that the first alternative is better because it does the right
thing without user intervention.  The second would be preferred should
distros want to offer full RT by default, but I am guessing thta most
distros would be reluctant to do this for some time to come.

Either way, these approaches have the advantage of giving RT users the
latency they need, even in the face of networking configuration changes,
while giving non-RT users the required performance of the networking
configuration changes themselves.

							Thanx, Paul

> ---
>  net/core/dev.c | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/net/core/dev.c b/net/core/dev.c
> index f8c23de..869ef62 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -6969,10 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
>  void synchronize_net(void)
>  {
>  	might_sleep();
> -	if (rtnl_is_locked())
> -		synchronize_rcu_expedited();
> -	else
> -		synchronize_rcu();
> +	synchronize_rcu();
>  }
>  EXPORT_SYMBOL(synchronize_net);
> 
> -- 
> 2.5.1
> 


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27  0:44 ` Paul E. McKenney
@ 2015-10-27 12:31   ` Josh Cartwright
  2015-10-27 14:18     ` Eric Dumazet
  0 siblings, 1 reply; 10+ messages in thread
From: Josh Cartwright @ 2015-10-27 12:31 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: tglx, bigeasy, linux-rt-users, linux-kernel, Eric Dumazet,
	David S. Miller

On Mon, Oct 26, 2015 at 05:44:22PM -0700, Paul E. McKenney wrote:
> On Mon, Oct 26, 2015 at 02:14:55PM -0500, Josh Cartwright wrote:
> > This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41.
> > 
> > While the use of synchronize_rcu_expedited() might make
> > synchronize_net() "faster", it does so at significant cost on RT
> > systems, as expediting a grace period forcibly preempts any
> > high-priority RT tasks (via the stop_machine() mechanism).
> > 
> > Without be3fc413da9e reverted, we can observe a latency spike up to 30us
> > with cyclictest by rapidly unplugging/reestablishing an ethernet link.
[..]
> 
> Hmmm...  If I remember correctly, using expedited here resulted
> in impressive performance improvements in some important cases.
> Perhaps things have changed (I must defer to Eric), but if not, how
> about something like this instead?
> 
> 	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREMPT_RT_FULL))
> 		synchronize_rcu_expedited();
> 	else
> 		synchronize_rcu();
> 
> Alternatively, a boot-time option could be used:
> 
> 	if (rtnl_is_locked() && !some_rt_boot_parameter)
> 		synchronize_rcu_expedited();
> 	else
> 		synchronize_rcu();
> 
> I believe that the first alternative is better because it does the right
> thing without user intervention.  The second would be preferred should
> distros want to offer full RT by default, but I am guessing thta most
> distros would be reluctant to do this for some time to come.
> 
> Either way, these approaches have the advantage of giving RT users the
> latency they need, even in the face of networking configuration changes,
> while giving non-RT users the required performance of the networking
> configuration changes themselves.

Okay, yes, I like the first suggestion better as well, I've included a
patch below that does just that.  I hope you don't mind me turning it
into a Suggested-by :).

Thanks for taking a look!
  Josh

-- 8< --
From: Josh Cartwright <joshc@ni.com>
Subject: [PATCH] net: make synchronize_rcu_expedited() conditional on RT_FULL

While the use of synchronize_rcu_expedited() might make
synchronize_net() "faster", it does so at significant cost on RT
systems, as expediting a grace period forcibly preempts any
high-priority RT tasks (via the stop_machine() mechanism).

Without this change, we can observe a latency spike up to 30us with
cyclictest by rapidly unplugging/reestablishing an ethernet link.

Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Suggested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Josh Cartwright <joshc@ni.com>
---
 net/core/dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index f8c23de..16fbef8 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
 void synchronize_net(void)
 {
 	might_sleep();
-	if (rtnl_is_locked())
+	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
 		synchronize_rcu_expedited();
 	else
 		synchronize_rcu();
-- 
2.5.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 12:31   ` Josh Cartwright
@ 2015-10-27 14:18     ` Eric Dumazet
  2015-10-27 15:02       ` Arnaldo Carvalho de Melo
  2015-10-30  9:16       ` David Miller
  0 siblings, 2 replies; 10+ messages in thread
From: Eric Dumazet @ 2015-10-27 14:18 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: Paul E. McKenney, tglx, bigeasy, linux-rt-users, linux-kernel,
	David S. Miller

On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote:

> Okay, yes, I like the first suggestion better as well, I've included a
> patch below that does just that.  I hope you don't mind me turning it
> into a Suggested-by :).
> 
> Thanks for taking a look!
>   Josh


> @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
>  void synchronize_net(void)
>  {
>  	might_sleep();
> -	if (rtnl_is_locked())
> +	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
>  		synchronize_rcu_expedited();
>  	else
>  		synchronize_rcu();

No objection from me. Thanks.

Acked-by: Eric Dumazet <edumazet@google.com>



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 14:18     ` Eric Dumazet
@ 2015-10-27 15:02       ` Arnaldo Carvalho de Melo
  2015-10-27 15:27         ` Eric Dumazet
  2015-10-30  9:16       ` David Miller
  1 sibling, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-10-27 15:02 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Josh Cartwright, Paul E. McKenney, tglx, bigeasy, linux-rt-users,
	linux-kernel, David S. Miller, Clark Williams

Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu:
> On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote:
> 
> > Okay, yes, I like the first suggestion better as well, I've included a
> > patch below that does just that.  I hope you don't mind me turning it
> > into a Suggested-by :).
> > 
> > Thanks for taking a look!
> >   Josh
> 
> 
> > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
> >  void synchronize_net(void)
> >  {
> >  	might_sleep();
> > -	if (rtnl_is_locked())
> > +	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
> >  		synchronize_rcu_expedited();
> >  	else
> >  		synchronize_rcu();
> 
> No objection from me. Thanks.
> 
> Acked-by: Eric Dumazet <edumazet@google.com>

The first suggestion, with it disabled by default seems to be the most
flexible tho, i.e, Paul's original message plus the boot parameter line:

Alternatively, a boot-time option could be used:

int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT;

        if (rtnl_is_locked() && !some_rt_boot_parameter)
                synchronize_rcu_expedited();
        else
                synchronize_rcu();

Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT
set to 1, while upstream would have this default to 0.

RT oriented kernel users could try using this in some scenarios where
networking is not the critical path.

- Arnaldo

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 15:02       ` Arnaldo Carvalho de Melo
@ 2015-10-27 15:27         ` Eric Dumazet
  2015-10-27 23:15           ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Eric Dumazet @ 2015-10-27 15:27 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Josh Cartwright, Paul E. McKenney, tglx, bigeasy, linux-rt-users,
	linux-kernel, David S. Miller, Clark Williams

On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu:
> > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote:
> > 
> > > Okay, yes, I like the first suggestion better as well, I've included a
> > > patch below that does just that.  I hope you don't mind me turning it
> > > into a Suggested-by :).
> > > 
> > > Thanks for taking a look!
> > >   Josh
> > 
> > 
> > > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
> > >  void synchronize_net(void)
> > >  {
> > >  	might_sleep();
> > > -	if (rtnl_is_locked())
> > > +	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
> > >  		synchronize_rcu_expedited();
> > >  	else
> > >  		synchronize_rcu();
> > 
> > No objection from me. Thanks.
> > 
> > Acked-by: Eric Dumazet <edumazet@google.com>
> 
> The first suggestion, with it disabled by default seems to be the most
> flexible tho, i.e, Paul's original message plus the boot parameter line:
> 
> Alternatively, a boot-time option could be used:
> 
> int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT;
> 
>         if (rtnl_is_locked() && !some_rt_boot_parameter)
>                 synchronize_rcu_expedited();
>         else
>                 synchronize_rcu();
> 
> Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT
> set to 1, while upstream would have this default to 0.
> 
> RT oriented kernel users could try using this in some scenarios where
> networking is not the critical path.


Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe
a generic solution would make synchronize_rcu_expedited() to fallback
synchronize_rcu() after boot time on RT.

Not sure why networking use of synchronize_rcu_expedited() would be
problematic, and not the others.

scripts/checkpatch.pl has this comment about this :

# Check for expedited grace periods that interrupt non-idle non-nohz
# online CPUs.  These expedited can therefore degrade real-time response
# if used carelessly, and should be avoided where not absolutely
# needed.  It is always OK to use synchronize_rcu_expedited() and
# synchronize_sched_expedited() at boot time (before real-time applications
# start) and in error situations where real-time response is compromised in
# any case.  Note that synchronize_srcu_expedited() does -not- interrupt
# other CPUs, so don't warn on uses of synchronize_srcu_expedited().
# Of course, nothing comes for free, and srcu_read_lock() and
# srcu_read_unlock() do contain full memory barriers in payment for
# synchronize_srcu_expedited() non-interruption properties.





^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 15:27         ` Eric Dumazet
@ 2015-10-27 23:15           ` Paul E. McKenney
  2015-10-28  8:34             ` Josh Cartwright
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2015-10-27 23:15 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Arnaldo Carvalho de Melo, Josh Cartwright, tglx, bigeasy,
	linux-rt-users, linux-kernel, David S. Miller, Clark Williams

On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote:
> On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu:
> > > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote:
> > > 
> > > > Okay, yes, I like the first suggestion better as well, I've included a
> > > > patch below that does just that.  I hope you don't mind me turning it
> > > > into a Suggested-by :).
> > > > 
> > > > Thanks for taking a look!
> > > >   Josh
> > > 
> > > 
> > > > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
> > > >  void synchronize_net(void)
> > > >  {
> > > >  	might_sleep();
> > > > -	if (rtnl_is_locked())
> > > > +	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
> > > >  		synchronize_rcu_expedited();
> > > >  	else
> > > >  		synchronize_rcu();
> > > 
> > > No objection from me. Thanks.
> > > 
> > > Acked-by: Eric Dumazet <edumazet@google.com>
> > 
> > The first suggestion, with it disabled by default seems to be the most
> > flexible tho, i.e, Paul's original message plus the boot parameter line:
> > 
> > Alternatively, a boot-time option could be used:
> > 
> > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT;
> > 
> >         if (rtnl_is_locked() && !some_rt_boot_parameter)
> >                 synchronize_rcu_expedited();
> >         else
> >                 synchronize_rcu();

This could be OK, but why not start with something very simple and automatic?
We can always add more knobs when and if they actually prove necessary.
In contrast, unnecessary knobs can cause confusion and might at the same time
get locked into some misbegotten userspace application, which would make the
unnecessary knob really hard to get rid of.

> > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT
> > set to 1, while upstream would have this default to 0.
> > 
> > RT oriented kernel users could try using this in some scenarios where
> > networking is not the critical path.
> 
> Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe
> a generic solution would make synchronize_rcu_expedited() to fallback
> synchronize_rcu() after boot time on RT.
> 
> Not sure why networking use of synchronize_rcu_expedited() would be
> problematic, and not the others.

>From what I can see, their testing just happened to run into this one.
Perhaps further testing will run into others, or perhaps the others are
off in code paths that should not be exercised while running RT apps.

I doubt that it is anything personal.  ;-)

> scripts/checkpatch.pl has this comment about this :
> 
> # Check for expedited grace periods that interrupt non-idle non-nohz
> # online CPUs.  These expedited can therefore degrade real-time response
> # if used carelessly, and should be avoided where not absolutely
> # needed.  It is always OK to use synchronize_rcu_expedited() and
> # synchronize_sched_expedited() at boot time (before real-time applications
> # start) and in error situations where real-time response is compromised in
> # any case.  Note that synchronize_srcu_expedited() does -not- interrupt
> # other CPUs, so don't warn on uses of synchronize_srcu_expedited().
> # Of course, nothing comes for free, and srcu_read_lock() and
> # srcu_read_unlock() do contain full memory barriers in payment for
> # synchronize_srcu_expedited() non-interruption properties.

As the author of that paragraph, I hereby declare that synchronize_net()'s
use of synchronize_rcu_expedited() clears the high absolutely-needed bar.
The point of that paragraph is to get people to think hard about alternatives,
for example, call_rcu().  However, as I understand it, call_rcu() would not
be particularly helpful in the synchronize_net() case.

							Thanx, Paul


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 23:15           ` Paul E. McKenney
@ 2015-10-28  8:34             ` Josh Cartwright
  2015-10-28 12:27               ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Josh Cartwright @ 2015-10-28  8:34 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: Eric Dumazet, Arnaldo Carvalho de Melo, tglx, bigeasy,
	linux-rt-users, linux-kernel, David S. Miller, Clark Williams

On Tue, Oct 27, 2015 at 04:15:59PM -0700, Paul E. McKenney wrote:
> On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote:
> > On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote:
[..]
> > > The first suggestion, with it disabled by default seems to be the most
> > > flexible tho, i.e, Paul's original message plus the boot parameter line:
> > > 
> > > Alternatively, a boot-time option could be used:
> > > 
> > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT;
> > > 
> > >         if (rtnl_is_locked() && !some_rt_boot_parameter)
> > >                 synchronize_rcu_expedited();
> > >         else
> > >                 synchronize_rcu();
> 
> This could be OK, but why not start with something very simple and automatic?
> We can always add more knobs when and if they actually prove necessary.

I suppose the question is if, for acme's usecases the answer to "when
it's proven necessary" is "now".

> In contrast, unnecessary knobs can cause confusion and might at the same time
> get locked into some misbegotten userspace application, which would make the
> unnecessary knob really hard to get rid of.

I think I would make a stronger statement; the CONFIG_SYNC_NET_DEFAULT
proposed option would be a boot/compile time parameter which says "I
require networking (and network configuration) in my critical path", why
don't we have these flags for other I/O subsystems?  What's special
about networking?

We don't because applications can make use of thread priorities to
express exactly which tasks should be more important than others.  So
perhaps the failure here is that RCU (and networking, by implication)
doesn't (can't?) take into consideration the calling thread's priority?
(And, there may be a cascade of other problems as well, like deferred
work pushed to a waitqueue, and thus losing the callers priority, etc)

(I will admit that RCU is a black box to me, so it is entirely possible
it's already capable of this, or it's fundamentally impossible, or
somewhere in between :)

> > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT
> > > set to 1, while upstream would have this default to 0.
> > > 
> > > RT oriented kernel users could try using this in some scenarios where
> > > networking is not the critical path.
> > 
> > Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe
> > a generic solution would make synchronize_rcu_expedited() to fallback
> > synchronize_rcu() after boot time on RT.
> > 
> > Not sure why networking use of synchronize_rcu_expedited() would be
> > problematic, and not the others.
> 
> From what I can see, their testing just happened to run into this one.
> Perhaps further testing will run into others, or perhaps the others are
> off in code paths that should not be exercised while running RT apps.

I accidentally ran into this issue when I was doing testing with an
ethernet cable w/ a broken RJ-45 connector (without the tab, that I was
just too lazy to replace), and I kept accidentally knocking it out.  :)

Regardless, industrial automation environments aren't known for having
the most stable network environments; there may be deployed systems
doing high priority motion control tasks, we'd want to ensure that the
poor network technician sent in to repair a defective network switch
wouldn't end up being mangled.

> > scripts/checkpatch.pl has this comment about this :

Also, Documentation/RCU/checklist.txt mentions:

	Use of the expedited primitives should be restricted to rare
	configuration-change operations that would not normally be
	undertaken while a real-time..

I think it could have been argued at the time, that operations under
rtnl_lock() were "configuration-change" operations.  However, for our
use cases, it's not, as link changes are external events beyond control.

Thanks again,
  Josh

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-28  8:34             ` Josh Cartwright
@ 2015-10-28 12:27               ` Paul E. McKenney
  0 siblings, 0 replies; 10+ messages in thread
From: Paul E. McKenney @ 2015-10-28 12:27 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: Eric Dumazet, Arnaldo Carvalho de Melo, tglx, bigeasy,
	linux-rt-users, linux-kernel, David S. Miller, Clark Williams

On Wed, Oct 28, 2015 at 03:34:00AM -0500, Josh Cartwright wrote:
> On Tue, Oct 27, 2015 at 04:15:59PM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote:
> > > On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote:
> [..]
> > > > The first suggestion, with it disabled by default seems to be the most
> > > > flexible tho, i.e, Paul's original message plus the boot parameter line:
> > > > 
> > > > Alternatively, a boot-time option could be used:
> > > > 
> > > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT;
> > > > 
> > > >         if (rtnl_is_locked() && !some_rt_boot_parameter)
> > > >                 synchronize_rcu_expedited();
> > > >         else
> > > >                 synchronize_rcu();
> > 
> > This could be OK, but why not start with something very simple and automatic?
> > We can always add more knobs when and if they actually prove necessary.
> 
> I suppose the question is if, for acme's usecases the answer to "when
> it's proven necessary" is "now".
> 
> > In contrast, unnecessary knobs can cause confusion and might at the same time
> > get locked into some misbegotten userspace application, which would make the
> > unnecessary knob really hard to get rid of.
> 
> I think I would make a stronger statement; the CONFIG_SYNC_NET_DEFAULT
> proposed option would be a boot/compile time parameter which says "I
> require networking (and network configuration) in my critical path", why
> don't we have these flags for other I/O subsystems?  What's special
> about networking?
> 
> We don't because applications can make use of thread priorities to
> express exactly which tasks should be more important than others.  So
> perhaps the failure here is that RCU (and networking, by implication)
> doesn't (can't?) take into consideration the calling thread's priority?
> (And, there may be a cascade of other problems as well, like deferred
> work pushed to a waitqueue, and thus losing the callers priority, etc)
> 
> (I will admit that RCU is a black box to me, so it is entirely possible
> it's already capable of this, or it's fundamentally impossible, or
> somewhere in between :)

CONFIG_RCU_KTHREAD_PRIO=nn, where 0 says SCHED_OTHER and 0 < nn <= 99
says SCHED_FIFO with RT priority nn.

> > > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT
> > > > set to 1, while upstream would have this default to 0.
> > > > 
> > > > RT oriented kernel users could try using this in some scenarios where
> > > > networking is not the critical path.
> > > 
> > > Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe
> > > a generic solution would make synchronize_rcu_expedited() to fallback
> > > synchronize_rcu() after boot time on RT.
> > > 
> > > Not sure why networking use of synchronize_rcu_expedited() would be
> > > problematic, and not the others.
> > 
> > From what I can see, their testing just happened to run into this one.
> > Perhaps further testing will run into others, or perhaps the others are
> > off in code paths that should not be exercised while running RT apps.
> 
> I accidentally ran into this issue when I was doing testing with an
> ethernet cable w/ a broken RJ-45 connector (without the tab, that I was
> just too lazy to replace), and I kept accidentally knocking it out.  :)
> 
> Regardless, industrial automation environments aren't known for having
> the most stable network environments; there may be deployed systems
> doing high priority motion control tasks, we'd want to ensure that the
> poor network technician sent in to repair a defective network switch
> wouldn't end up being mangled.
> 
> > > scripts/checkpatch.pl has this comment about this :
> 
> Also, Documentation/RCU/checklist.txt mentions:
> 
> 	Use of the expedited primitives should be restricted to rare
> 	configuration-change operations that would not normally be
> 	undertaken while a real-time..
> 
> I think it could have been argued at the time, that operations under
> rtnl_lock() were "configuration-change" operations.  However, for our
> use cases, it's not, as link changes are external events beyond control.

Certainly the variety of operations that people are willing to run
concurrently with real-time applications seems to be steadily growing
over time...  But much depends on the RT deadlines.

							Thanx, Paul


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()"
  2015-10-27 14:18     ` Eric Dumazet
  2015-10-27 15:02       ` Arnaldo Carvalho de Melo
@ 2015-10-30  9:16       ` David Miller
  1 sibling, 0 replies; 10+ messages in thread
From: David Miller @ 2015-10-30  9:16 UTC (permalink / raw)
  To: eric.dumazet; +Cc: joshc, paulmck, tglx, bigeasy, linux-rt-users, linux-kernel

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 27 Oct 2015 07:18:01 -0700

> On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote:
> 
>> Okay, yes, I like the first suggestion better as well, I've included a
>> patch below that does just that.  I hope you don't mind me turning it
>> into a Suggested-by :).
>> 
>> Thanks for taking a look!
>>   Josh
> 
> 
>> @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev);
>>  void synchronize_net(void)
>>  {
>>  	might_sleep();
>> -	if (rtnl_is_locked())
>> +	if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
>>  		synchronize_rcu_expedited();
>>  	else
>>  		synchronize_rcu();
> 
> No objection from me. Thanks.
> 
> Acked-by: Eric Dumazet <edumazet@google.com>

I agree with the sentiment to do the simple thing here first before
adding potentially useless knobs.

Signed-off-by: David S. Miller <davem@davemloft.net>

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2015-10-30  9:16 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-10-26 19:14 [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" Josh Cartwright
2015-10-27  0:44 ` Paul E. McKenney
2015-10-27 12:31   ` Josh Cartwright
2015-10-27 14:18     ` Eric Dumazet
2015-10-27 15:02       ` Arnaldo Carvalho de Melo
2015-10-27 15:27         ` Eric Dumazet
2015-10-27 23:15           ` Paul E. McKenney
2015-10-28  8:34             ` Josh Cartwright
2015-10-28 12:27               ` Paul E. McKenney
2015-10-30  9:16       ` David Miller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).