[PATCH tip/core/rcu 0/20] Miscellaneous fixes for 4.11

[PATCH tip/core/rcu 0/20] Miscellaneous fixes for 4.11

[Paul E. McKenney]

Hello!

This series provides miscellaneous fixes:

1.	Make RCU_EXPEDITE_BOOT be the default in order to speed up
	boot, courtesy of Sebastian Andrzej Siewior.

2.	Make RCU suspicious-access lockdep splats use pr_err().

3.	Disable sys_membarrier when nohz_full is enabled, courtesy
	of Mathieu Desnoyers.

4.	Only dump stalled-tasks stacks if there was a real stall, courtesy
	of Byungchul Park.

5.	Remove unneeded rcu_process_callbacks() declarations.

6.	Remove unused but set "mask" variable, courtesy of Tobias Klauser.

7.	Remove short-term CPU kicking.

8.	Add long-term CPU kicking.

9.	Once again use NMI-based stack traces in stall warnings.

10.	Add functions to test for trivial grace periods.

11.	Make cond_resched() provide RCU quiescent state, which will
	hopefully lead to the removal of cond_resched_rcu_qs().

12.	Re-enable TASKS_RCU for User Mode Linux.

13.	Don't wake rcuc/X kthreads on NOCB CPUs.

14.	Add comment headers to expedited-grace-period counter functions.

15.	Make rcu_cpu_starting() use its "cpu" argument.

16.	Enable RCU tracepoints by default to aid in debugging, courtesy
	of Matt Fleming.

17.	Fix comment in rcu_organize_nocb_kthreads(), which no longer
	spawns kthreads.

18.	Clarify comments about when llist locking is needed, courtesy
	of Joel Fernandes.

19.	Eliminate unused expedited_normal counter.

20.	Add lockdep checks to synchronous expedited primitives.

							Thanx, Paul

------------------------------------------------------------------------

 Documentation/RCU/trace.txt |    5 ---
 include/linux/llist.h       |   37 +++++++++++++++-----------
 include/linux/rcupdate.h    |    8 +++++
 include/linux/rcutiny.h     |   10 +++++++
 include/linux/rcutree.h     |    2 +
 include/linux/sched.h       |    3 +-
 init/Kconfig                |   14 ----------
 kernel/locking/lockdep.c    |   12 ++++----
 kernel/membarrier.c         |    4 ++
 kernel/rcu/tiny.c           |    2 -
 kernel/rcu/tree.c           |   61 +++++++++++++++++++++++++++++++-------------
 kernel/rcu/tree.h           |    1 
 kernel/rcu/tree_exp.h       |   28 +++++++++++++++++++-
 kernel/rcu/tree_plugin.h    |   16 ++++++++++-
 kernel/rcu/tree_trace.c     |    3 --
 kernel/rcu/update.c         |    6 +---
 kernel/sched/core.c         |    1 
 lib/Kconfig.debug           |    1 
 18 files changed, 143 insertions(+), 71 deletions(-)

[PATCH tip/core/rcu 01/20] rcu: update: Make RCU_EXPEDITE_BOOT be the default

[Paul E. McKenney]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

RCU_EXPEDITE_BOOT should speed up the boot process by enforcing
synchronize_rcu_expedited() instead of synchronize_rcu() during the boot
process. There should be no reason why one does not want this and there
is no need worry about real time latency at this point.
Therefore make it default.

Note that users wishing to avoid expediting entirely, for example when
bringing up new hardware possibly having flaky IPIs, can use the
rcu_normal boot parameter to override boot-time expediting.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ paulmck: Reworded commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 init/Kconfig        | 13 -------------
 kernel/rcu/update.c |  6 ++----
 2 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 223b734abccd..96e6d56acd50 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -781,19 +781,6 @@ config RCU_NOCB_CPU_ALL
 
 endchoice
 
-config RCU_EXPEDITE_BOOT
-	bool
-	default n
-	help
-	  This option enables expedited grace periods at boot time,
-	  as if rcu_expedite_gp() had been invoked early in boot.
-	  The corresponding rcu_unexpedite_gp() is invoked from
-	  rcu_end_inkernel_boot(), which is intended to be invoked
-	  at the end of the kernel-only boot sequence, just before
-	  init is exec'ed.
-
-	  Accept the default if unsure.
-
 endmenu # "RCU Subsystem"
 
 config BUILD_BIN2C
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 4f6db7e6a117..9e03db9ea9c0 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -132,8 +132,7 @@ bool rcu_gp_is_normal(void)
 }
 EXPORT_SYMBOL_GPL(rcu_gp_is_normal);
 
-static atomic_t rcu_expedited_nesting =
-	ATOMIC_INIT(IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT) ? 1 : 0);
+static atomic_t rcu_expedited_nesting = ATOMIC_INIT(1);
 
 /*
  * Should normal grace-period primitives be expedited?  Intended for
@@ -182,8 +181,7 @@ EXPORT_SYMBOL_GPL(rcu_unexpedite_gp);
  */
 void rcu_end_inkernel_boot(void)
 {
-	if (IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT))
-		rcu_unexpedite_gp();
+	rcu_unexpedite_gp();
 	if (rcu_normal_after_boot)
 		WRITE_ONCE(rcu_normal, 1);
 }
-- 
2.5.2

[PATCH tip/core/rcu 02/20] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

This commit switches RCU suspicious-access splats use pr_err()
instead of the current INFO printk()s.  This change makes it easier
to automatically classify splats.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/locking/lockdep.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index 7c38f8f3d97b..844cd04bb453 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
 	printk("\n");
-	printk("===============================\n");
-	printk("[ INFO: suspicious RCU usage. ]\n");
+	pr_err("===============================\n");
+	pr_err("suspicious RCU usage. ]\n");
 	print_kernel_ident();
-	printk("-------------------------------\n");
-	printk("%s:%d %s!\n", file, line, s);
-	printk("\nother info that might help us debug this:\n\n");
-	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
+	pr_err("-------------------------------\n");
+	pr_err("%s:%d %s!\n", file, line, s);
+	pr_err("\nother info that might help us debug this:\n\n");
+	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
 	       !rcu_lockdep_current_cpu_online()
 			? "RCU used illegally from offline CPU!\n"
 			: !rcu_is_watching()
-- 
2.5.2

Re: [PATCH tip/core/rcu 02/20] lockdep: Make RCU suspicious-access splats use pr_err

[Josh Triplett]

On Sat, Jan 14, 2017 at 01:13:03AM -0800, Paul E. McKenney wrote:
> This commit switches RCU suspicious-access splats use pr_err()
> instead of the current INFO printk()s.  This change makes it easier
> to automatically classify splats.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

One comment below.

>  kernel/locking/lockdep.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> index 7c38f8f3d97b..844cd04bb453 100644
> --- a/kernel/locking/lockdep.c
> +++ b/kernel/locking/lockdep.c
> @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
>  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
>  	/* Note: the following can be executed concurrently, so be careful. */
>  	printk("\n");
> -	printk("===============================\n");
> -	printk("[ INFO: suspicious RCU usage. ]\n");
> +	pr_err("===============================\n");
> +	pr_err("suspicious RCU usage. ]\n");

This message change seems unrelated, and potentially unintentional.

>  	print_kernel_ident();
> -	printk("-------------------------------\n");
> -	printk("%s:%d %s!\n", file, line, s);
> -	printk("\nother info that might help us debug this:\n\n");
> -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> +	pr_err("-------------------------------\n");
> +	pr_err("%s:%d %s!\n", file, line, s);
> +	pr_err("\nother info that might help us debug this:\n\n");
> +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
>  	       !rcu_lockdep_current_cpu_online()
>  			? "RCU used illegally from offline CPU!\n"
>  			: !rcu_is_watching()
> -- 
> 2.5.2
> 

Re: [PATCH tip/core/rcu 02/20] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

On Sun, Jan 15, 2017 at 11:53:02PM -0800, Josh Triplett wrote:
> On Sat, Jan 14, 2017 at 01:13:03AM -0800, Paul E. McKenney wrote:
> > This commit switches RCU suspicious-access splats use pr_err()
> > instead of the current INFO printk()s.  This change makes it easier
> > to automatically classify splats.
> > 
> > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> One comment below.

Good catch, fixed!

							Thanx, Paul

> >  kernel/locking/lockdep.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> > index 7c38f8f3d97b..844cd04bb453 100644
> > --- a/kernel/locking/lockdep.c
> > +++ b/kernel/locking/lockdep.c
> > @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
> >  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
> >  	/* Note: the following can be executed concurrently, so be careful. */
> >  	printk("\n");
> > -	printk("===============================\n");
> > -	printk("[ INFO: suspicious RCU usage. ]\n");
> > +	pr_err("===============================\n");
> > +	pr_err("suspicious RCU usage. ]\n");
> 
> This message change seems unrelated, and potentially unintentional.
> 
> >  	print_kernel_ident();
> > -	printk("-------------------------------\n");
> > -	printk("%s:%d %s!\n", file, line, s);
> > -	printk("\nother info that might help us debug this:\n\n");
> > -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> > +	pr_err("-------------------------------\n");
> > +	pr_err("%s:%d %s!\n", file, line, s);
> > +	pr_err("\nother info that might help us debug this:\n\n");
> > +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> >  	       !rcu_lockdep_current_cpu_online()
> >  			? "RCU used illegally from offline CPU!\n"
> >  			: !rcu_is_watching()
> > -- 
> > 2.5.2
> > 
> 

Re: [PATCH tip/core/rcu 02/20] lockdep: Make RCU suspicious-access splats use pr_err

[Peter Zijlstra]

On Sat, Jan 14, 2017 at 01:13:03AM -0800, Paul E. McKenney wrote:
> This commit switches RCU suspicious-access splats use pr_err()
> instead of the current INFO printk()s.  This change makes it easier
> to automatically classify splats.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/locking/lockdep.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> index 7c38f8f3d97b..844cd04bb453 100644
> --- a/kernel/locking/lockdep.c
> +++ b/kernel/locking/lockdep.c
> @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
>  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
>  	/* Note: the following can be executed concurrently, so be careful. */
>  	printk("\n");
> -	printk("===============================\n");
> -	printk("[ INFO: suspicious RCU usage. ]\n");
> +	pr_err("===============================\n");
> +	pr_err("suspicious RCU usage. ]\n");
>  	print_kernel_ident();
> -	printk("-------------------------------\n");
> -	printk("%s:%d %s!\n", file, line, s);
> -	printk("\nother info that might help us debug this:\n\n");
> -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> +	pr_err("-------------------------------\n");
> +	pr_err("%s:%d %s!\n", file, line, s);
> +	pr_err("\nother info that might help us debug this:\n\n");
> +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
>  	       !rcu_lockdep_current_cpu_online()
>  			? "RCU used illegally from offline CPU!\n"
>  			: !rcu_is_watching()


This makes it inconsistent with the rest of lockdep; why are these more
important?

Re: [PATCH tip/core/rcu 02/20] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

On Mon, Jan 16, 2017 at 06:21:29PM +0100, Peter Zijlstra wrote:
> On Sat, Jan 14, 2017 at 01:13:03AM -0800, Paul E. McKenney wrote:
> > This commit switches RCU suspicious-access splats use pr_err()
> > instead of the current INFO printk()s.  This change makes it easier
> > to automatically classify splats.
> > 
> > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/locking/lockdep.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> > index 7c38f8f3d97b..844cd04bb453 100644
> > --- a/kernel/locking/lockdep.c
> > +++ b/kernel/locking/lockdep.c
> > @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
> >  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
> >  	/* Note: the following can be executed concurrently, so be careful. */
> >  	printk("\n");
> > -	printk("===============================\n");
> > -	printk("[ INFO: suspicious RCU usage. ]\n");
> > +	pr_err("===============================\n");
> > +	pr_err("suspicious RCU usage. ]\n");
> >  	print_kernel_ident();
> > -	printk("-------------------------------\n");
> > -	printk("%s:%d %s!\n", file, line, s);
> > -	printk("\nother info that might help us debug this:\n\n");
> > -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> > +	pr_err("-------------------------------\n");
> > +	pr_err("%s:%d %s!\n", file, line, s);
> > +	pr_err("\nother info that might help us debug this:\n\n");
> > +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> >  	       !rcu_lockdep_current_cpu_online()
> >  			? "RCU used illegally from offline CPU!\n"
> >  			: !rcu_is_watching()
> 
> 
> This makes it inconsistent with the rest of lockdep; why are these more
> important?

No idea.  The checkpatch script whined piteously so I changed them.

							Thanx, Paul

[PATCH tip/core/rcu 03/20] Fix: Disable sys_membarrier when nohz_full is enabled

[Paul E. McKenney]

From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Userspace applications should be allowed to expect the membarrier system
call with MEMBARRIER_CMD_SHARED command to issue memory barriers on
nohz_full CPUs, but synchronize_sched() does not take those into
account.

Given that we do not want unrelated processes to be able to affect
real-time sensitive nohz_full CPUs, simply return ENOSYS when membarrier
is invoked on a kernel with enabled nohz_full CPUs.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Josh Triplett <josh@joshtriplett.org>
CC: Steven Rostedt <rostedt@goodmis.org>
CC: <stable@vger.kernel.org>	[3.10+]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Chris Metcalf <cmetcalf@mellanox.com>
Cc: Rik van Riel <riel@redhat.com>
Acked-by: Lai Jiangshan <jiangshanlai@gmail.com>
---
 kernel/membarrier.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/membarrier.c b/kernel/membarrier.c
index 536c727a56e9..9f9284f37f8d 100644
--- a/kernel/membarrier.c
+++ b/kernel/membarrier.c
@@ -16,6 +16,7 @@
 
 #include <linux/syscalls.h>
 #include <linux/membarrier.h>
+#include <linux/tick.h>
 
 /*
  * Bitmask made from a "or" of all commands within enum membarrier_cmd,
@@ -51,6 +52,9 @@
  */
 SYSCALL_DEFINE2(membarrier, int, cmd, int, flags)
 {
+	/* MEMBARRIER_CMD_SHARED is not compatible with nohz_full. */
+	if (tick_nohz_full_enabled())
+		return -ENOSYS;
 	if (unlikely(flags))
 		return -EINVAL;
 	switch (cmd) {
-- 
2.5.2

[PATCH tip/core/rcu 04/20] rcu: Only dump stalled-tasks stacks if there was a real stall

[Paul E. McKenney]

From: Byungchul Park <byungchul.park@lge.com>

The print_other_cpu_stall() function currently unconditionally invokes
rcu_print_detail_task_stall().  This is OK because if there was a stall
sufficient to cause print_other_cpu_stall() to be invoked, that stall
is very likely to persist through the entire print_other_cpu_stall()
execution.  However, if the stall did not persist, the variable ndetected
will be zero, and that variable is already tested in an "if" statement.
Therefore, this commit moves the call to rcu_print_detail_task_stall()
under that pre-existing "if" to improve readability, with a very rare
reduction in overhead.

Signed-off-by: Byungchul Park <byungchul.park@lge.com>
[ paulmck: Reworked commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cb4e2056ccf3..6232d2f9a84e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1379,6 +1379,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 	       (long)rsp->gpnum, (long)rsp->completed, totqlen);
 	if (ndetected) {
 		rcu_dump_cpu_stacks(rsp);
+
+		/* Complain about tasks blocking the grace period. */
+		rcu_print_detail_task_stall(rsp);
 	} else {
 		if (READ_ONCE(rsp->gpnum) != gpnum ||
 		    READ_ONCE(rsp->completed) == gpnum) {
@@ -1395,9 +1398,6 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 		}
 	}
 
-	/* Complain about tasks blocking the grace period. */
-	rcu_print_detail_task_stall(rsp);
-
 	rcu_check_gp_kthread_starvation(rsp);
 
 	panic_on_rcu_stall();
-- 
2.5.2

[PATCH tip/core/rcu 05/20] rcu: Remove unneeded rcu_process_callbacks() declarations

[Paul E. McKenney]

The declarations of __rcu_process_callbacks() and rcu_process_callbacks()
are not needed, as the definition of both of these functions appear before
any uses.  This commit therefore removes both declarations.

Reported-by: "Ahmed, Iftekhar" <ahmedi@oregonstate.edu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tiny.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index b23a4d076f3d..fa6a48d3917b 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -41,8 +41,6 @@
 
 /* Forward declarations for tiny_plugin.h. */
 struct rcu_ctrlblk;
-static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp);
-static void rcu_process_callbacks(struct softirq_action *unused);
 static void __call_rcu(struct rcu_head *head,
 		       rcu_callback_t func,
 		       struct rcu_ctrlblk *rcp);
-- 
2.5.2

[PATCH tip/core/rcu 06/20] rcu: Remove unused but set variable

[Paul E. McKenney]

From: Tobias Klauser <tklauser@distanz.ch>

Since commit 7ec99de36f40 ("rcu: Provide exact CPU-online tracking for
RCU"), the variable mask in rcu_init_percpu_data is set but no longer
used. Remove it to fix the following warning when building with 'W=1':

  kernel/rcu/tree.c: In function ‘rcu_init_percpu_data’:
  kernel/rcu/tree.c:3765:16: warning: variable ‘mask’ set but not used [-Wunused-but-set-variable]

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6232d2f9a84e..83bf054e194e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3765,7 +3765,6 @@ static void
 rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
-	unsigned long mask;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rcu_get_root(rsp);
 
@@ -3788,7 +3787,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 	 * of the next grace period.
 	 */
 	rnp = rdp->mynode;
-	mask = rdp->grpmask;
 	raw_spin_lock_rcu_node(rnp);		/* irqs already disabled. */
 	if (!rdp->beenonline)
 		WRITE_ONCE(rsp->ncpus, READ_ONCE(rsp->ncpus) + 1);
-- 
2.5.2

[PATCH tip/core/rcu 07/20] rcu: Remove short-term CPU kicking

[Paul E. McKenney]

Commit 4914950aaa12d ("rcu: Stop treating in-kernel CPU-bound workloads
as errors") added a (relatively) short-timeout call to resched_cpu().
This was inspired by as issue that was fixed by b7e7ade34e61 ("sched/core:
Fix remote wakeups").  But given that this issue was fixed, it is time
for the current commit to remove this call to resched_cpu().

Reported-by: Byungchul Park <byungchul.park@lge.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 83bf054e194e..3c0d6c129718 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1220,12 +1220,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
 	}
 
-	/* And if it has been a really long time, kick the CPU as well. */
-	if (ULONG_CMP_GE(jiffies,
-			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
-	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
-		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
-
 	return 0;
 }
 
-- 
2.5.2

[PATCH tip/core/rcu 08/20] rcu: Add long-term CPU kicking

[Paul E. McKenney]

Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
frequent calls to resched_cpu(), which means that the only time
resched_cpu() is invoked is after an RCU CPU stall warning.  Although
this is good from an avoid-IPIs perspective, we should try to break
things loose -before- splatting.  This commit therefore starts invoking
resched_cpu() for each holdout at each force-quiescent-state interval
that is more than halfway through the stall-warning interval.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3c0d6c129718..5a4aaad75e76 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1220,6 +1220,13 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
 	}
 
+	/*
+	 * If more than halfway to RCU CPU stall-warning time, do
+	 * a resched_cpu() to try to loosen things up a bit.
+	 */
+	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
+		resched_cpu(rdp->cpu);
+
 	return 0;
 }
 
-- 
2.5.2

Re: [PATCH tip/core/rcu 08/20] rcu: Add long-term CPU kicking

[Josh Triplett]

On Sat, Jan 14, 2017 at 01:13:09AM -0800, Paul E. McKenney wrote:
> Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> frequent calls to resched_cpu(), which means that the only time
> resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> this is good from an avoid-IPIs perspective, we should try to break
> things loose -before- splatting.  This commit therefore starts invoking
> resched_cpu() for each holdout at each force-quiescent-state interval
> that is more than halfway through the stall-warning interval.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Could you please reverse the order of patches 7 and 8, so that
unnecessary splats don't occur between the two?

For the patch itself:
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcu/tree.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 3c0d6c129718..5a4aaad75e76 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1220,6 +1220,13 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
>  		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
>  	}
>  
> +	/*
> +	 * If more than halfway to RCU CPU stall-warning time, do
> +	 * a resched_cpu() to try to loosen things up a bit.
> +	 */
> +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> +		resched_cpu(rdp->cpu);
> +
>  	return 0;
>  }
>  
> -- 
> 2.5.2
> 

Re: [PATCH tip/core/rcu 08/20] rcu: Add long-term CPU kicking

[Paul E. McKenney]

On Sun, Jan 15, 2017 at 11:58:07PM -0800, Josh Triplett wrote:
> On Sat, Jan 14, 2017 at 01:13:09AM -0800, Paul E. McKenney wrote:
> > Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> > frequent calls to resched_cpu(), which means that the only time
> > resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> > this is good from an avoid-IPIs perspective, we should try to break
> > things loose -before- splatting.  This commit therefore starts invoking
> > resched_cpu() for each holdout at each force-quiescent-state interval
> > that is more than halfway through the stall-warning interval.
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Could you please reverse the order of patches 7 and 8, so that
> unnecessary splats don't occur between the two?

Done!

							Thanx, Paul

> For the patch itself:
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> 
> >  kernel/rcu/tree.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 3c0d6c129718..5a4aaad75e76 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -1220,6 +1220,13 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
> >  		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
> >  	}
> >  
> > +	/*
> > +	 * If more than halfway to RCU CPU stall-warning time, do
> > +	 * a resched_cpu() to try to loosen things up a bit.
> > +	 */
> > +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> > +		resched_cpu(rdp->cpu);
> > +
> >  	return 0;
> >  }
> >  
> > -- 
> > 2.5.2
> > 
> 

[PATCH tip/core/rcu 09/20] rcu: Once again use NMI-based stack traces in stall warnings

[Paul E. McKenney]

This commit is for all intents and purposes a revert of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks").  The reason to suppose
that this can now safely be reverted is the presence of 42a0bb3f7138
("printk/nmi: generic solution for safe printk in NMI"), which is said
to have made NMI-based stack dumps safe.

However, this reversion keeps one nice property of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks"), namely that
only those CPUs blocking the grace period are dumped.  The new
trigger_single_cpu_backtrace() is used to make this happen, as
suggested by Josh Poimboeuf.

Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
---
 kernel/rcu/tree.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 5a4aaad75e76..d7b63b88434b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1278,7 +1278,10 @@ static void rcu_check_gp_kthread_starvation(struct rcu_state *rsp)
 }
 
 /*
- * Dump stacks of all tasks running on stalled CPUs.
+ * Dump stacks of all tasks running on stalled CPUs.  First try using
+ * NMIs, but fall back to manual remote stack tracing on architectures
+ * that don't support NMI-based stack dumps.  The NMI-triggered stack
+ * traces are more accurate because they are printed by the target CPU.
  */
 static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 {
@@ -1288,11 +1291,10 @@ static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 
 	rcu_for_each_leaf_node(rsp, rnp) {
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-		if (rnp->qsmask != 0) {
-			for_each_leaf_node_possible_cpu(rnp, cpu)
-				if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+		for_each_leaf_node_possible_cpu(rnp, cpu)
+			if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+				if (!trigger_single_cpu_backtrace(cpu))
 					dump_cpu_task(cpu);
-		}
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	}
 }
-- 
2.5.2

[PATCH tip/core/rcu 10/20] rcu: Add functions to test for trivial grace periods

[Paul E. McKenney]

Under some circumstances, RCU grace periods are zero cost.  For
RCU-preempt, this is the case during boot, and for RCU-bh and RCU-sched,
this is the case if there is only one CPU.  This means that RCU users
might wish to dispense with grace-period-avoidance strategies when
grace periods are zero cost, so this commit adds rcu_trivial_gp(),
rcu_bh_trivial_gp(), and rcu_sched_trivial_gp() to test for these
conditions.  Because the conditions leading to zero-cost grace periods
can change at any time (for example, when a second CPU is onlined), these
functions should be used as performance hints, and must not be relied
on for correctness.  For example, even if rcu_trivial_gp() returns true,
you are required to invoke synchronize_rcu().

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |  8 ++++++++
 include/linux/rcutiny.h  | 10 ++++++++++
 include/linux/rcutree.h  |  2 ++
 kernel/rcu/tree.c        | 24 ++++++++++++++++++++++++
 kernel/rcu/tree_plugin.h | 11 +++++++++++
 5 files changed, 55 insertions(+)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 01f71e1d2e94..a6222478b87d 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -293,6 +293,7 @@ void __rcu_read_lock(void);
 void __rcu_read_unlock(void);
 void rcu_read_unlock_special(struct task_struct *t);
 void synchronize_rcu(void);
+bool rcu_trivial_gp(void);
 
 /*
  * Defined as a macro as it is a very low level header included from
@@ -448,6 +449,13 @@ bool __rcu_is_watching(void);
 #define RCU_SCHEDULER_INIT	1
 #define RCU_SCHEDULER_RUNNING	2
 
+#ifndef CONFIG_PREEMPT_RCU
+static inline bool rcu_trivial_gp(void)
+{
+	return rcu_sched_trivial_gp();
+}
+#endif /* #ifndef CONFIG_PREEMPT_RCU */
+
 /*
  * init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic
  * initialization and destruction of rcu_head on the stack. rcu_head structures
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index ac81e4063b40..a77dafe79813 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -82,6 +82,16 @@ static inline void synchronize_sched_expedited(void)
 	synchronize_sched();
 }
 
+static inline bool rcu_sched_trivial_gp(void)
+{
+	return true;
+}
+
+static inline bool rcu_bh_trivial_gp(void)
+{
+	return true;
+}
+
 static inline void kfree_call_rcu(struct rcu_head *head,
 				  rcu_callback_t func)
 {
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 63a4e4cf40a5..fcd61cb08851 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -47,6 +47,8 @@ static inline void rcu_virt_note_context_switch(int cpu)
 void synchronize_rcu_bh(void);
 void synchronize_sched_expedited(void);
 void synchronize_rcu_expedited(void);
+bool rcu_sched_trivial_gp(void);
+bool rcu_bh_trivial_gp(void);
 
 void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
 
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d7b63b88434b..ed5a17aca281 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3293,6 +3293,18 @@ void synchronize_sched(void)
 EXPORT_SYMBOL_GPL(synchronize_sched);
 
 /**
+ * rcu_sched_trivial_gp - Are RCU-sched grace periods trivially zero cost?
+ *
+ * Returns true if RCU-sched grace periods are currently zero cost, which
+ * they are if there is only one CPU.  Note that unless you take steps to
+ * prevent it, the number of CPUs might change at any time.
+ */
+bool rcu_sched_trivial_gp(void)
+{
+	return rcu_blocking_is_gp();
+}
+
+/**
  * synchronize_rcu_bh - wait until an rcu_bh grace period has elapsed.
  *
  * Control will return to the caller some time after a full rcu_bh grace
@@ -3320,6 +3332,18 @@ void synchronize_rcu_bh(void)
 EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
 
 /**
+ * rcu_bh_trivial_gp - Are RCU-bh grace periods trivially zero cost?
+ *
+ * Returns true if RCU-bh grace periods are currently zero cost, which
+ * they are if there is only one CPU.  Note that unless you take steps to
+ * prevent it, the number of CPUs might change at any time.
+ */
+bool rcu_bh_trivial_gp(void)
+{
+	return rcu_blocking_is_gp();
+}
+
+/**
  * get_state_synchronize_rcu - Snapshot current RCU state
  *
  * Returns a cookie that is used by a later call to cond_synchronize_rcu()
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 56583e764ebf..e92d67a3fad2 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -680,6 +680,17 @@ void synchronize_rcu(void)
 EXPORT_SYMBOL_GPL(synchronize_rcu);
 
 /**
+ * rcu_trivial_gp - Are RCU grace periods trivially zero cost?
+ *
+ * Returns true if RCU grace periods are currently zero cost, which
+ * they are during boot.
+ */
+bool rcu_trivial_gp(void)
+{
+	return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
+}
+
+/**
  * rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
  *
  * Note that this primitive does not necessarily wait for an RCU grace period
-- 
2.5.2

Re: [PATCH tip/core/rcu 10/20] rcu: Add functions to test for trivial grace periods

[Josh Triplett]

On Sat, Jan 14, 2017 at 01:13:11AM -0800, Paul E. McKenney wrote:
> Under some circumstances, RCU grace periods are zero cost.  For
> RCU-preempt, this is the case during boot, and for RCU-bh and RCU-sched,
> this is the case if there is only one CPU.  This means that RCU users
> might wish to dispense with grace-period-avoidance strategies when
> grace periods are zero cost, so this commit adds rcu_trivial_gp(),
> rcu_bh_trivial_gp(), and rcu_sched_trivial_gp() to test for these
> conditions.  Because the conditions leading to zero-cost grace periods
> can change at any time (for example, when a second CPU is onlined), these
> functions should be used as performance hints, and must not be relied
> on for correctness.  For example, even if rcu_trivial_gp() returns true,
> you are required to invoke synchronize_rcu().
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Do you have anything planned that uses these functions?

Exposing this information, rather than just letting callers call
synchronize_rcu() and sometimes get "free" grace periods, seems
potentially error-prone.

If you keep these functions, please expand the comments above them to
explicitly include the explanation in the commit message, and
specifically that you must always call the corresponding synchronize
function even if these functions return true.

>  include/linux/rcupdate.h |  8 ++++++++
>  include/linux/rcutiny.h  | 10 ++++++++++
>  include/linux/rcutree.h  |  2 ++
>  kernel/rcu/tree.c        | 24 ++++++++++++++++++++++++
>  kernel/rcu/tree_plugin.h | 11 +++++++++++
>  5 files changed, 55 insertions(+)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 01f71e1d2e94..a6222478b87d 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -293,6 +293,7 @@ void __rcu_read_lock(void);
>  void __rcu_read_unlock(void);
>  void rcu_read_unlock_special(struct task_struct *t);
>  void synchronize_rcu(void);
> +bool rcu_trivial_gp(void);
>  
>  /*
>   * Defined as a macro as it is a very low level header included from
> @@ -448,6 +449,13 @@ bool __rcu_is_watching(void);
>  #define RCU_SCHEDULER_INIT	1
>  #define RCU_SCHEDULER_RUNNING	2
>  
> +#ifndef CONFIG_PREEMPT_RCU
> +static inline bool rcu_trivial_gp(void)
> +{
> +	return rcu_sched_trivial_gp();
> +}
> +#endif /* #ifndef CONFIG_PREEMPT_RCU */
> +
>  /*
>   * init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic
>   * initialization and destruction of rcu_head on the stack. rcu_head structures
> diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> index ac81e4063b40..a77dafe79813 100644
> --- a/include/linux/rcutiny.h
> +++ b/include/linux/rcutiny.h
> @@ -82,6 +82,16 @@ static inline void synchronize_sched_expedited(void)
>  	synchronize_sched();
>  }
>  
> +static inline bool rcu_sched_trivial_gp(void)
> +{
> +	return true;
> +}
> +
> +static inline bool rcu_bh_trivial_gp(void)
> +{
> +	return true;
> +}
> +
>  static inline void kfree_call_rcu(struct rcu_head *head,
>  				  rcu_callback_t func)
>  {
> diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> index 63a4e4cf40a5..fcd61cb08851 100644
> --- a/include/linux/rcutree.h
> +++ b/include/linux/rcutree.h
> @@ -47,6 +47,8 @@ static inline void rcu_virt_note_context_switch(int cpu)
>  void synchronize_rcu_bh(void);
>  void synchronize_sched_expedited(void);
>  void synchronize_rcu_expedited(void);
> +bool rcu_sched_trivial_gp(void);
> +bool rcu_bh_trivial_gp(void);
>  
>  void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
>  
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index d7b63b88434b..ed5a17aca281 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -3293,6 +3293,18 @@ void synchronize_sched(void)
>  EXPORT_SYMBOL_GPL(synchronize_sched);
>  
>  /**
> + * rcu_sched_trivial_gp - Are RCU-sched grace periods trivially zero cost?
> + *
> + * Returns true if RCU-sched grace periods are currently zero cost, which
> + * they are if there is only one CPU.  Note that unless you take steps to
> + * prevent it, the number of CPUs might change at any time.
> + */
> +bool rcu_sched_trivial_gp(void)
> +{
> +	return rcu_blocking_is_gp();
> +}
> +
> +/**
>   * synchronize_rcu_bh - wait until an rcu_bh grace period has elapsed.
>   *
>   * Control will return to the caller some time after a full rcu_bh grace
> @@ -3320,6 +3332,18 @@ void synchronize_rcu_bh(void)
>  EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
>  
>  /**
> + * rcu_bh_trivial_gp - Are RCU-bh grace periods trivially zero cost?
> + *
> + * Returns true if RCU-bh grace periods are currently zero cost, which
> + * they are if there is only one CPU.  Note that unless you take steps to
> + * prevent it, the number of CPUs might change at any time.
> + */
> +bool rcu_bh_trivial_gp(void)
> +{
> +	return rcu_blocking_is_gp();
> +}
> +
> +/**
>   * get_state_synchronize_rcu - Snapshot current RCU state
>   *
>   * Returns a cookie that is used by a later call to cond_synchronize_rcu()
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 56583e764ebf..e92d67a3fad2 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -680,6 +680,17 @@ void synchronize_rcu(void)
>  EXPORT_SYMBOL_GPL(synchronize_rcu);
>  
>  /**
> + * rcu_trivial_gp - Are RCU grace periods trivially zero cost?
> + *
> + * Returns true if RCU grace periods are currently zero cost, which
> + * they are during boot.
> + */
> +bool rcu_trivial_gp(void)
> +{
> +	return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
> +}
> +
> +/**
>   * rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
>   *
>   * Note that this primitive does not necessarily wait for an RCU grace period
> -- 
> 2.5.2
> 

Re: [PATCH tip/core/rcu 10/20] rcu: Add functions to test for trivial grace periods

[Paul E. McKenney]

On Mon, Jan 16, 2017 at 12:01:29AM -0800, Josh Triplett wrote:
> On Sat, Jan 14, 2017 at 01:13:11AM -0800, Paul E. McKenney wrote:
> > Under some circumstances, RCU grace periods are zero cost.  For
> > RCU-preempt, this is the case during boot, and for RCU-bh and RCU-sched,
> > this is the case if there is only one CPU.  This means that RCU users
> > might wish to dispense with grace-period-avoidance strategies when
> > grace periods are zero cost, so this commit adds rcu_trivial_gp(),
> > rcu_bh_trivial_gp(), and rcu_sched_trivial_gp() to test for these
> > conditions.  Because the conditions leading to zero-cost grace periods
> > can change at any time (for example, when a second CPU is onlined), these
> > functions should be used as performance hints, and must not be relied
> > on for correctness.  For example, even if rcu_trivial_gp() returns true,
> > you are required to invoke synchronize_rcu().
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Do you have anything planned that uses these functions?
> 
> Exposing this information, rather than just letting callers call
> synchronize_rcu() and sometimes get "free" grace periods, seems
> potentially error-prone.
> 
> If you keep these functions, please expand the comments above them to
> explicitly include the explanation in the commit message, and
> specifically that you must always call the corresponding synchronize
> function even if these functions return true.

Hmmm...

Someone asked for this, I failed to add their Reported-by, and
I do not remember who it was.  I have tagged this commit with
trivial_gp.2017.01.17a and am dropping it from my commits for the 4.11
merge window.  If someone speaks up for it, I can always add it in later.

The use case was something about doing batching if !rcu_trivial_gp(),
but not bothering if rcu_trivial_gp(), which means that grace periods
are free anyway.  So performance, no risk of incorrectness.  If used
correctly, anyway...

							Thanx, Paul

> >  include/linux/rcupdate.h |  8 ++++++++
> >  include/linux/rcutiny.h  | 10 ++++++++++
> >  include/linux/rcutree.h  |  2 ++
> >  kernel/rcu/tree.c        | 24 ++++++++++++++++++++++++
> >  kernel/rcu/tree_plugin.h | 11 +++++++++++
> >  5 files changed, 55 insertions(+)
> > 
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index 01f71e1d2e94..a6222478b87d 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -293,6 +293,7 @@ void __rcu_read_lock(void);
> >  void __rcu_read_unlock(void);
> >  void rcu_read_unlock_special(struct task_struct *t);
> >  void synchronize_rcu(void);
> > +bool rcu_trivial_gp(void);
> >  
> >  /*
> >   * Defined as a macro as it is a very low level header included from
> > @@ -448,6 +449,13 @@ bool __rcu_is_watching(void);
> >  #define RCU_SCHEDULER_INIT	1
> >  #define RCU_SCHEDULER_RUNNING	2
> >  
> > +#ifndef CONFIG_PREEMPT_RCU
> > +static inline bool rcu_trivial_gp(void)
> > +{
> > +	return rcu_sched_trivial_gp();
> > +}
> > +#endif /* #ifndef CONFIG_PREEMPT_RCU */
> > +
> >  /*
> >   * init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic
> >   * initialization and destruction of rcu_head on the stack. rcu_head structures
> > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > index ac81e4063b40..a77dafe79813 100644
> > --- a/include/linux/rcutiny.h
> > +++ b/include/linux/rcutiny.h
> > @@ -82,6 +82,16 @@ static inline void synchronize_sched_expedited(void)
> >  	synchronize_sched();
> >  }
> >  
> > +static inline bool rcu_sched_trivial_gp(void)
> > +{
> > +	return true;
> > +}
> > +
> > +static inline bool rcu_bh_trivial_gp(void)
> > +{
> > +	return true;
> > +}
> > +
> >  static inline void kfree_call_rcu(struct rcu_head *head,
> >  				  rcu_callback_t func)
> >  {
> > diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> > index 63a4e4cf40a5..fcd61cb08851 100644
> > --- a/include/linux/rcutree.h
> > +++ b/include/linux/rcutree.h
> > @@ -47,6 +47,8 @@ static inline void rcu_virt_note_context_switch(int cpu)
> >  void synchronize_rcu_bh(void);
> >  void synchronize_sched_expedited(void);
> >  void synchronize_rcu_expedited(void);
> > +bool rcu_sched_trivial_gp(void);
> > +bool rcu_bh_trivial_gp(void);
> >  
> >  void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
> >  
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index d7b63b88434b..ed5a17aca281 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -3293,6 +3293,18 @@ void synchronize_sched(void)
> >  EXPORT_SYMBOL_GPL(synchronize_sched);
> >  
> >  /**
> > + * rcu_sched_trivial_gp - Are RCU-sched grace periods trivially zero cost?
> > + *
> > + * Returns true if RCU-sched grace periods are currently zero cost, which
> > + * they are if there is only one CPU.  Note that unless you take steps to
> > + * prevent it, the number of CPUs might change at any time.
> > + */
> > +bool rcu_sched_trivial_gp(void)
> > +{
> > +	return rcu_blocking_is_gp();
> > +}
> > +
> > +/**
> >   * synchronize_rcu_bh - wait until an rcu_bh grace period has elapsed.
> >   *
> >   * Control will return to the caller some time after a full rcu_bh grace
> > @@ -3320,6 +3332,18 @@ void synchronize_rcu_bh(void)
> >  EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
> >  
> >  /**
> > + * rcu_bh_trivial_gp - Are RCU-bh grace periods trivially zero cost?
> > + *
> > + * Returns true if RCU-bh grace periods are currently zero cost, which
> > + * they are if there is only one CPU.  Note that unless you take steps to
> > + * prevent it, the number of CPUs might change at any time.
> > + */
> > +bool rcu_bh_trivial_gp(void)
> > +{
> > +	return rcu_blocking_is_gp();
> > +}
> > +
> > +/**
> >   * get_state_synchronize_rcu - Snapshot current RCU state
> >   *
> >   * Returns a cookie that is used by a later call to cond_synchronize_rcu()
> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > index 56583e764ebf..e92d67a3fad2 100644
> > --- a/kernel/rcu/tree_plugin.h
> > +++ b/kernel/rcu/tree_plugin.h
> > @@ -680,6 +680,17 @@ void synchronize_rcu(void)
> >  EXPORT_SYMBOL_GPL(synchronize_rcu);
> >  
> >  /**
> > + * rcu_trivial_gp - Are RCU grace periods trivially zero cost?
> > + *
> > + * Returns true if RCU grace periods are currently zero cost, which
> > + * they are during boot.
> > + */
> > +bool rcu_trivial_gp(void)
> > +{
> > +	return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
> > +}
> > +
> > +/**
> >   * rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
> >   *
> >   * Note that this primitive does not necessarily wait for an RCU grace period
> > -- 
> > 2.5.2
> > 
> 

[PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Paul E. McKenney]

There is some confusion as to which of cond_resched() or
cond_resched_rcu_qs() should be added to long in-kernel loops.
This commit therefore eliminates the decision by adding RCU
quiescent states to cond_resched().

Warning: This is a prototype.  For example, it does not correctly
handle Tasks RCU.  Which is OK for the moment, given that no one
actually uses Tasks RCU yet.

Reported-by: Michal Hocko <mhocko@kernel.org>
Not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
 include/linux/sched.h | 3 ++-
 kernel/sched/core.c   | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4d1905245c7a..1531c48f56e2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -3352,10 +3352,11 @@ static inline int signal_pending_state(long state, struct task_struct *p)
  * cond_resched_lock() will drop the spinlock before scheduling,
  * cond_resched_softirq() will enable bhs before scheduling.
  */
+void rcu_all_qs(void);
 #ifndef CONFIG_PREEMPT
 extern int _cond_resched(void);
 #else
-static inline int _cond_resched(void) { return 0; }
+static inline int _cond_resched(void) { rcu_all_qs(); return 0; }
 #endif
 
 #define cond_resched() ({			\
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c56fb57f2991..b442f8918eb9 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
 		preempt_schedule_common();
 		return 1;
 	}
+	rcu_all_qs();
 	return 0;
 }
 EXPORT_SYMBOL(_cond_resched);
-- 
2.5.2

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Josh Triplett]

On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> There is some confusion as to which of cond_resched() or
> cond_resched_rcu_qs() should be added to long in-kernel loops.
> This commit therefore eliminates the decision by adding RCU
> quiescent states to cond_resched().
> 
> Warning: This is a prototype.  For example, it does not correctly
> handle Tasks RCU.  Which is OK for the moment, given that no one
> actually uses Tasks RCU yet.
> 
> Reported-by: Michal Hocko <mhocko@kernel.org>
> Not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>

Do you intend to merge this patch, or have you just posted it for
review?

You may want to remove it from this 20-patch series and post it as a
separate RFC PATCH, to ensure that it doesn't get merged while still a
prototype.

>  include/linux/sched.h | 3 ++-
>  kernel/sched/core.c   | 1 +
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 4d1905245c7a..1531c48f56e2 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -3352,10 +3352,11 @@ static inline int signal_pending_state(long state, struct task_struct *p)
>   * cond_resched_lock() will drop the spinlock before scheduling,
>   * cond_resched_softirq() will enable bhs before scheduling.
>   */
> +void rcu_all_qs(void);
>  #ifndef CONFIG_PREEMPT
>  extern int _cond_resched(void);
>  #else
> -static inline int _cond_resched(void) { return 0; }
> +static inline int _cond_resched(void) { rcu_all_qs(); return 0; }
>  #endif
>  
>  #define cond_resched() ({			\
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index c56fb57f2991..b442f8918eb9 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
>  		preempt_schedule_common();
>  		return 1;
>  	}
> +	rcu_all_qs();
>  	return 0;
>  }
>  EXPORT_SYMBOL(_cond_resched);
> -- 
> 2.5.2
> 

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Paul E. McKenney]

On Mon, Jan 16, 2017 at 12:08:43AM -0800, Josh Triplett wrote:
> On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > There is some confusion as to which of cond_resched() or
> > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > This commit therefore eliminates the decision by adding RCU
> > quiescent states to cond_resched().
> > 
> > Warning: This is a prototype.  For example, it does not correctly
> > handle Tasks RCU.  Which is OK for the moment, given that no one
> > actually uses Tasks RCU yet.
> > 
> > Reported-by: Michal Hocko <mhocko@kernel.org>
> > Not-yet-signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> 
> Do you intend to merge this patch, or have you just posted it for
> review?
> 
> You may want to remove it from this 20-patch series and post it as a
> separate RFC PATCH, to ensure that it doesn't get merged while still a
> prototype.

It does look like I should delay this one until I get my story straight.

Good catch!

							Thanx, Paul

> >  include/linux/sched.h | 3 ++-
> >  kernel/sched/core.c   | 1 +
> >  2 files changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index 4d1905245c7a..1531c48f56e2 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -3352,10 +3352,11 @@ static inline int signal_pending_state(long state, struct task_struct *p)
> >   * cond_resched_lock() will drop the spinlock before scheduling,
> >   * cond_resched_softirq() will enable bhs before scheduling.
> >   */
> > +void rcu_all_qs(void);
> >  #ifndef CONFIG_PREEMPT
> >  extern int _cond_resched(void);
> >  #else
> > -static inline int _cond_resched(void) { return 0; }
> > +static inline int _cond_resched(void) { rcu_all_qs(); return 0; }
> >  #endif
> >  
> >  #define cond_resched() ({			\
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index c56fb57f2991..b442f8918eb9 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> >  		preempt_schedule_common();
> >  		return 1;
> >  	}
> > +	rcu_all_qs();
> >  	return 0;
> >  }
> >  EXPORT_SYMBOL(_cond_resched);
> > -- 
> > 2.5.2
> > 
> 

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Peter Zijlstra]

On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> There is some confusion as to which of cond_resched() or
> cond_resched_rcu_qs() should be added to long in-kernel loops.
> This commit therefore eliminates the decision by adding RCU
> quiescent states to cond_resched().

Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
invalid under preemptible RCU. Is it already?

> Warning: This is a prototype.  For example, it does not correctly
> handle Tasks RCU.  Which is OK for the moment, given that no one
> actually uses Tasks RCU yet.

> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
>  		preempt_schedule_common();
>  		return 1;
>  	}
> +	rcu_all_qs();
>  	return 0;
>  }

Still not a real fan of this, it does make cond_resched() touch a bunch
more cachelines, also, I suppose that if we're going to do this, we
should make __cond_resched_lock() and __cond_resched_softirq() act
similarly.

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Paul E. McKenney]

On Mon, Jan 16, 2017 at 06:11:30PM +0100, Peter Zijlstra wrote:
> On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > There is some confusion as to which of cond_resched() or
> > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > This commit therefore eliminates the decision by adding RCU
> > quiescent states to cond_resched().
> 
> Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
> invalid under preemptible RCU. Is it already?

In theory, yes.  In practice, I just tested it with preemption and
lockdep enabled, and it didn't complain.  If further testing finds
complaints, we can either fix those uses (preferred) or revert
this patch.

> > Warning: This is a prototype.  For example, it does not correctly
> > handle Tasks RCU.  Which is OK for the moment, given that no one
> > actually uses Tasks RCU yet.
> 
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> >  		preempt_schedule_common();
> >  		return 1;
> >  	}
> > +	rcu_all_qs();
> >  	return 0;
> >  }
> 
> Still not a real fan of this, it does make cond_resched() touch a bunch
> more cachelines, also, I suppose that if we're going to do this, we
> should make __cond_resched_lock() and __cond_resched_softirq() act
> similarly.

Michal (now CCed) argues that having to distinguish between cond_resched()
and cond_resched_rcu_qs() is overly burdensome.  Michal?

Any thoughts on how we might remove this burden without the additional
cache misses?  I will take another look as well to see what could make
it lower cost.  There are probably ways...  Would it make sense to
have RCU maintain a need-rcu_all_qs() flage in the same cacheline as
the __preempt_count?  Perhaps throttling the writes to this flag from
the RCU grace-period kthreads to once per 100 milliseconds or so?

							Thanx, Paul

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Michal Hocko]

On Mon 16-01-17 16:54:03, Paul E. McKenney wrote:
> On Mon, Jan 16, 2017 at 06:11:30PM +0100, Peter Zijlstra wrote:
> > On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > > There is some confusion as to which of cond_resched() or
> > > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > > This commit therefore eliminates the decision by adding RCU
> > > quiescent states to cond_resched().
> > 
> > Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
> > invalid under preemptible RCU. Is it already?
> 
> In theory, yes.  In practice, I just tested it with preemption and
> lockdep enabled, and it didn't complain.  If further testing finds
> complaints, we can either fix those uses (preferred) or revert
> this patch.
> 
> > > Warning: This is a prototype.  For example, it does not correctly
> > > handle Tasks RCU.  Which is OK for the moment, given that no one
> > > actually uses Tasks RCU yet.
> > 
> > > --- a/kernel/sched/core.c
> > > +++ b/kernel/sched/core.c
> > > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> > >  		preempt_schedule_common();
> > >  		return 1;
> > >  	}
> > > +	rcu_all_qs();
> > >  	return 0;
> > >  }
> > 
> > Still not a real fan of this, it does make cond_resched() touch a bunch
> > more cachelines, also, I suppose that if we're going to do this, we
> > should make __cond_resched_lock() and __cond_resched_softirq() act
> > similarly.
> 
> Michal (now CCed) argues that having to distinguish between cond_resched()
> and cond_resched_rcu_qs() is overly burdensome.  Michal?

Yes, it is really not clear which one is meant to be in which context. I
really do not see which cond_resched should be turned intto
cond_resched_rcu_qs.

> Any thoughts on how we might remove this burden without the additional
> cache misses?  I will take another look as well to see what could make
> it lower cost.  There are probably ways...  Would it make sense to
> have RCU maintain a need-rcu_all_qs() flage in the same cacheline as
> the __preempt_count?  Perhaps throttling the writes to this flag from
> the RCU grace-period kthreads to once per 100 milliseconds or so?

Can the stall detector simply request rescheduling when it gets
dangerously close to the timeout?

-- 
Michal Hocko
SUSE Labs

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Paul E. McKenney]

On Tue, Jan 17, 2017 at 11:51:41AM +0100, Michal Hocko wrote:
> On Mon 16-01-17 16:54:03, Paul E. McKenney wrote:
> > On Mon, Jan 16, 2017 at 06:11:30PM +0100, Peter Zijlstra wrote:
> > > On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > > > There is some confusion as to which of cond_resched() or
> > > > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > > > This commit therefore eliminates the decision by adding RCU
> > > > quiescent states to cond_resched().
> > > 
> > > Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
> > > invalid under preemptible RCU. Is it already?
> > 
> > In theory, yes.  In practice, I just tested it with preemption and
> > lockdep enabled, and it didn't complain.  If further testing finds
> > complaints, we can either fix those uses (preferred) or revert
> > this patch.
> > 
> > > > Warning: This is a prototype.  For example, it does not correctly
> > > > handle Tasks RCU.  Which is OK for the moment, given that no one
> > > > actually uses Tasks RCU yet.
> > > 
> > > > --- a/kernel/sched/core.c
> > > > +++ b/kernel/sched/core.c
> > > > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> > > >  		preempt_schedule_common();
> > > >  		return 1;
> > > >  	}
> > > > +	rcu_all_qs();
> > > >  	return 0;
> > > >  }
> > > 
> > > Still not a real fan of this, it does make cond_resched() touch a bunch
> > > more cachelines, also, I suppose that if we're going to do this, we
> > > should make __cond_resched_lock() and __cond_resched_softirq() act
> > > similarly.
> > 
> > Michal (now CCed) argues that having to distinguish between cond_resched()
> > and cond_resched_rcu_qs() is overly burdensome.  Michal?
> 
> Yes, it is really not clear which one is meant to be in which context. I
> really do not see which cond_resched should be turned intto
> cond_resched_rcu_qs.
> 
> > Any thoughts on how we might remove this burden without the additional
> > cache misses?  I will take another look as well to see what could make
> > it lower cost.  There are probably ways...  Would it make sense to
> > have RCU maintain a need-rcu_all_qs() flage in the same cacheline as
> > the __preempt_count?  Perhaps throttling the writes to this flag from
> > the RCU grace-period kthreads to once per 100 milliseconds or so?
> 
> Can the stall detector simply request rescheduling when it gets
> dangerously close to the timeout?

It is quite possible that half of the stall timeout would be a better
choice than my 100 milliseconds, but either way, there would be need
for a flag or some such.

							Thanx, Paul

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Michal Hocko]

On Tue 17-01-17 04:05:13, Paul E. McKenney wrote:
> On Tue, Jan 17, 2017 at 11:51:41AM +0100, Michal Hocko wrote:
> > On Mon 16-01-17 16:54:03, Paul E. McKenney wrote:
> > > On Mon, Jan 16, 2017 at 06:11:30PM +0100, Peter Zijlstra wrote:
> > > > On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > > > > There is some confusion as to which of cond_resched() or
> > > > > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > > > > This commit therefore eliminates the decision by adding RCU
> > > > > quiescent states to cond_resched().
> > > > 
> > > > Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
> > > > invalid under preemptible RCU. Is it already?
> > > 
> > > In theory, yes.  In practice, I just tested it with preemption and
> > > lockdep enabled, and it didn't complain.  If further testing finds
> > > complaints, we can either fix those uses (preferred) or revert
> > > this patch.
> > > 
> > > > > Warning: This is a prototype.  For example, it does not correctly
> > > > > handle Tasks RCU.  Which is OK for the moment, given that no one
> > > > > actually uses Tasks RCU yet.
> > > > 
> > > > > --- a/kernel/sched/core.c
> > > > > +++ b/kernel/sched/core.c
> > > > > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> > > > >  		preempt_schedule_common();
> > > > >  		return 1;
> > > > >  	}
> > > > > +	rcu_all_qs();
> > > > >  	return 0;
> > > > >  }
> > > > 
> > > > Still not a real fan of this, it does make cond_resched() touch a bunch
> > > > more cachelines, also, I suppose that if we're going to do this, we
> > > > should make __cond_resched_lock() and __cond_resched_softirq() act
> > > > similarly.
> > > 
> > > Michal (now CCed) argues that having to distinguish between cond_resched()
> > > and cond_resched_rcu_qs() is overly burdensome.  Michal?
> > 
> > Yes, it is really not clear which one is meant to be in which context. I
> > really do not see which cond_resched should be turned intto
> > cond_resched_rcu_qs.
> > 
> > > Any thoughts on how we might remove this burden without the additional
> > > cache misses?  I will take another look as well to see what could make
> > > it lower cost.  There are probably ways...  Would it make sense to
> > > have RCU maintain a need-rcu_all_qs() flage in the same cacheline as
> > > the __preempt_count?  Perhaps throttling the writes to this flag from
> > > the RCU grace-period kthreads to once per 100 milliseconds or so?
> > 
> > Can the stall detector simply request rescheduling when it gets
> > dangerously close to the timeout?
> 
> It is quite possible that half of the stall timeout would be a better
> choice than my 100 milliseconds, but either way, there would be need
> for a flag or some such.

E.g. set_tsk_need_resched() on the task currently running on a cpu which
is preventing the rcu grace period for too long?

That would only require change to the stall detector and the cond_resched
could be left alone completely.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH tip/core/rcu 11/20] sched,rcu: Make cond_resched() provide RCU quiescent state

[Paul E. McKenney]

On Tue, Jan 17, 2017 at 01:11:46PM +0100, Michal Hocko wrote:
> On Tue 17-01-17 04:05:13, Paul E. McKenney wrote:
> > On Tue, Jan 17, 2017 at 11:51:41AM +0100, Michal Hocko wrote:
> > > On Mon 16-01-17 16:54:03, Paul E. McKenney wrote:
> > > > On Mon, Jan 16, 2017 at 06:11:30PM +0100, Peter Zijlstra wrote:
> > > > > On Sat, Jan 14, 2017 at 01:13:12AM -0800, Paul E. McKenney wrote:
> > > > > > There is some confusion as to which of cond_resched() or
> > > > > > cond_resched_rcu_qs() should be added to long in-kernel loops.
> > > > > > This commit therefore eliminates the decision by adding RCU
> > > > > > quiescent states to cond_resched().
> > > > > 
> > > > > Which would make: rcu_read_lock(); cond_resched(); rcu_read_unlock();
> > > > > invalid under preemptible RCU. Is it already?
> > > > 
> > > > In theory, yes.  In practice, I just tested it with preemption and
> > > > lockdep enabled, and it didn't complain.  If further testing finds
> > > > complaints, we can either fix those uses (preferred) or revert
> > > > this patch.
> > > > 
> > > > > > Warning: This is a prototype.  For example, it does not correctly
> > > > > > handle Tasks RCU.  Which is OK for the moment, given that no one
> > > > > > actually uses Tasks RCU yet.
> > > > > 
> > > > > > --- a/kernel/sched/core.c
> > > > > > +++ b/kernel/sched/core.c
> > > > > > @@ -4907,6 +4907,7 @@ int __sched _cond_resched(void)
> > > > > >  		preempt_schedule_common();
> > > > > >  		return 1;
> > > > > >  	}
> > > > > > +	rcu_all_qs();
> > > > > >  	return 0;
> > > > > >  }
> > > > > 
> > > > > Still not a real fan of this, it does make cond_resched() touch a bunch
> > > > > more cachelines, also, I suppose that if we're going to do this, we
> > > > > should make __cond_resched_lock() and __cond_resched_softirq() act
> > > > > similarly.
> > > > 
> > > > Michal (now CCed) argues that having to distinguish between cond_resched()
> > > > and cond_resched_rcu_qs() is overly burdensome.  Michal?
> > > 
> > > Yes, it is really not clear which one is meant to be in which context. I
> > > really do not see which cond_resched should be turned intto
> > > cond_resched_rcu_qs.
> > > 
> > > > Any thoughts on how we might remove this burden without the additional
> > > > cache misses?  I will take another look as well to see what could make
> > > > it lower cost.  There are probably ways...  Would it make sense to
> > > > have RCU maintain a need-rcu_all_qs() flage in the same cacheline as
> > > > the __preempt_count?  Perhaps throttling the writes to this flag from
> > > > the RCU grace-period kthreads to once per 100 milliseconds or so?
> > > 
> > > Can the stall detector simply request rescheduling when it gets
> > > dangerously close to the timeout?
> > 
> > It is quite possible that half of the stall timeout would be a better
> > choice than my 100 milliseconds, but either way, there would be need
> > for a flag or some such.
> 
> E.g. set_tsk_need_resched() on the task currently running on a cpu which
> is preventing the rcu grace period for too long?
> 
> That would only require change to the stall detector and the cond_resched
> could be left alone completely.

Thank you!!!

The other complication is that under CONFIG_PREEMPT=y, _cond_resched()
is an empty function.  That would be one reason why use of cond_resched()
wasn't always giving RCU the quiescent states that it needs.  And that
is a problem with this patch, which I therefore need to defer to 4.12.

That aside, the reason I am reluctant to use the need-resched approach
except as an emergency measure is that the way I have to set that bit
remotely involves IPIs.

But don't get me wrong, it is extremely useful as an emergency meaure.
I am just trying to get cond_resched() to help on a non-emergency basis.

							Thanx, Paul

[PATCH tip/core/rcu 12/20] rcu: Re-enable TASKS_RCU for User Mode Linux

[Paul E. McKenney]

Now that User Mode Linux supports arch_irqs_disabled_flags(), this
commit re-enables TASKS_RCU for User Mode Linux.

Reported-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 init/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/init/Kconfig b/init/Kconfig
index 96e6d56acd50..d0caacb4bc14 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -529,7 +529,6 @@ config SRCU
 config TASKS_RCU
 	bool
 	default n
-	depends on !UML
 	select SRCU
 	help
 	  This option enables a task-based RCU implementation that uses
-- 
2.5.2

[PATCH tip/core/rcu 13/20] rcu: Don't wake rcuc/X kthreads on NOCB CPUs

[Paul E. McKenney]

Chris Friesen notice that rcuc/X kthreads were consuming CPU even on
NOCB CPUs.  This makes no sense because the only purpose or these
kthreads is to invoke normal (non-offloaded) callbacks, of which there
will never be any on NOCB CPUs.  This problem was due to a bug in
cpu_has_callbacks_ready_to_invoke(), which should have been checking
->nxttail[RCU_NEXT_TAIL] for NULL, but which was instead (incorrectly)
checking ->nxttail[RCU_DONE_TAIL].  Because ->nxttail[RCU_DONE_TAIL] is
never NULL, the only effect is to cause the rcuc/X kthread to execute
when it should not do so.

This commit therefore checks ->nxttail[RCU_NEXT_TAIL], which is NULL
for NOCB CPUs.

Reported-by: Chris Friesen <chris.friesen@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index ed5a17aca281..cac80e08b353 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -611,7 +611,7 @@ static int
 cpu_has_callbacks_ready_to_invoke(struct rcu_data *rdp)
 {
 	return &rdp->nxtlist != rdp->nxttail[RCU_DONE_TAIL] &&
-	       rdp->nxttail[RCU_DONE_TAIL] != NULL;
+	       rdp->nxttail[RCU_NEXT_TAIL] != NULL;
 }
 
 /*
-- 
2.5.2

[PATCH tip/core/rcu 14/20] rcu: Add comment headers to expedited-grace-period counter functions

[Paul E. McKenney]

These functions (rcu_exp_gp_seq_start(), rcu_exp_gp_seq_end(),
rcu_exp_gp_seq_snap(), and rcu_exp_gp_seq_done() seemed too obvious
to comment when written, but not so much when being documented.
This commit therefore adds header comments to each of them.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree_exp.h | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index e59e1849b89a..303df97bbfc5 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -20,16 +20,26 @@
  * Authors: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
  */
 
-/* Wrapper functions for expedited grace periods.  */
+/*
+ * Record the start of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_start(struct rcu_state *rsp)
 {
 	rcu_seq_start(&rsp->expedited_sequence);
 }
+
+/*
+ * Record the end of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_end(struct rcu_state *rsp)
 {
 	rcu_seq_end(&rsp->expedited_sequence);
 	smp_mb(); /* Ensure that consecutive grace periods serialize. */
 }
+
+/*
+ * Take a snapshot of the expedited-grace-period counter.
+ */
 static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 {
 	unsigned long s;
@@ -39,6 +49,12 @@ static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 	trace_rcu_exp_grace_period(rsp->name, s, TPS("snap"));
 	return s;
 }
+
+/*
+ * Given a counter snapshot from rcu_exp_gp_seq_snap(), return true
+ * if a full expedited grace period has elapsed since that snapshot
+ * was taken.
+ */
 static bool rcu_exp_gp_seq_done(struct rcu_state *rsp, unsigned long s)
 {
 	return rcu_seq_done(&rsp->expedited_sequence, s);
-- 
2.5.2

[PATCH tip/core/rcu 15/20] rcu: Make rcu_cpu_starting() use its "cpu" argument

[Paul E. McKenney]

The rcu_cpu_starting() function uses this_cpu_ptr() to locate the
incoming CPU's rcu_data structure.  This works for the boot CPU and for
all CPUs onlined after rcu_init() executes (during very early boot).
Currently, this is the full set of CPUs, so all is well.  But if
anyone ever parallelizes boot before rcu_init() time, it will fail.
This commit therefore substitutes the rcu_cpu_starting() function's
this_cpu_pointer() for per_cpu_ptr(), future-proofing the code and
(arguably) improving readability.

This commit inadvertently fixes a latent bug: If there ever had been
more than just the boot CPU online at rcu_init() time, the old code
would not initialize the non-boot CPUs, but rather would repeatedly
initialize the boot CPU.

Reported-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cac80e08b353..b1e803fbb46d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3897,7 +3897,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	struct rcu_state *rsp;
 
 	for_each_rcu_flavor(rsp) {
-		rdp = this_cpu_ptr(rsp->rda);
+		rdp = per_cpu_ptr(rsp->rda, cpu);
 		rnp = rdp->mynode;
 		mask = rdp->grpmask;
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-- 
2.5.2

[PATCH tip/core/rcu 16/20] rcu: Enable RCU tracepoints by default to aid in debugging

[Paul E. McKenney]

From: Matt Fleming <matt@codeblueprint.co.uk>

While debugging a performance issue I needed to understand why
RCU sofitrqs were firing so frequently.

Unfortunately, the RCU callback tracepoints are hidden behind
CONFIG_RCU_TRACE which defaults to off in the upstream kernel and is
likely to also be disabled in enterprise distribution configs.

Enable it by default for CONFIG_TREE_RCU. However, we must keep it
disabled for tiny RCU, because it would otherwise pull in a large
amount of code that would make tiny RCU less than tiny.

I ran some file system metadata intensive workloads (git checkout,
FS-Mark) on a variety of machines with this patch and saw no
detectable change in performance.

Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 lib/Kconfig.debug | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index b06848a104e6..5655bd95919a 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1450,6 +1450,7 @@ config RCU_CPU_STALL_TIMEOUT
 config RCU_TRACE
 	bool "Enable tracing for RCU"
 	depends on DEBUG_KERNEL
+	default y if TREE_RCU
 	select TRACE_CLOCK
 	help
 	  This option provides tracing in RCU which presents stats
-- 
2.5.2

[PATCH tip/core/rcu 17/20] rcu: Fix comment in rcu_organize_nocb_kthreads()

[Paul E. McKenney]

It used to be that the rcuo callback-offload kthreads were spawned
in rcu_organize_nocb_kthreads(), and the comment before the "for"
loop says as much.  However, this spawning has long since moved to
the CPU-hotplug code, so this commit fixes this comment.

Reported-by: Michalis Kokologiannakis <mixaskok@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree_plugin.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index e92d67a3fad2..9e295c3516dc 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2377,8 +2377,9 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp)
 	}
 
 	/*
-	 * Each pass through this loop sets up one rcu_data structure and
-	 * spawns one rcu_nocb_kthread().
+	 * Each pass through this loop sets up one rcu_data structure.
+	 * Should the corresponding CPU come online in the future, then
+	 * we will spawn the needed set of rcu_nocb_kthread() kthreads.
 	 */
 	for_each_cpu(cpu, rcu_nocb_mask) {
 		rdp = per_cpu_ptr(rsp->rda, cpu);
-- 
2.5.2

[PATCH tip/core/rcu 18/20] llist: Clarify comments about when locking is needed

[Paul E. McKenney]

From: Joel Fernandes <joelaf@google.com>

llist.h comments are confusing about when locking is needed versus when it
isn't. Clarify these comments by being more descriptive about why locking is
needed for llist_del_first.

Cc: Ingo Molnar <mingo@kernel.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Huang Ying <ying.huang@intel.com>
Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/llist.h | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)

diff --git a/include/linux/llist.h b/include/linux/llist.h
index fd4ca0b4fe0f..171baa90f6f6 100644
--- a/include/linux/llist.h
+++ b/include/linux/llist.h
@@ -3,28 +3,33 @@
 /*
  * Lock-less NULL terminated single linked list
  *
- * If there are multiple producers and multiple consumers, llist_add
- * can be used in producers and llist_del_all can be used in
- * consumers.  They can work simultaneously without lock.  But
- * llist_del_first can not be used here.  Because llist_del_first
- * depends on list->first->next does not changed if list->first is not
- * changed during its operation, but llist_del_first, llist_add,
- * llist_add (or llist_del_all, llist_add, llist_add) sequence in
- * another consumer may violate that.
- *
- * If there are multiple producers and one consumer, llist_add can be
- * used in producers and llist_del_all or llist_del_first can be used
- * in the consumer.
- *
- * This can be summarized as follow:
+ * Cases where locking is not needed:
+ * If there are multiple producers and multiple consumers, llist_add can be
+ * used in producers and llist_del_all can be used in consumers simultaneously
+ * without locking. Also a single consumer can use llist_del_first while
+ * multiple producers simultaneously use llist_add, without any locking.
+ *
+ * Cases where locking is needed:
+ * If we have multiple consumers with llist_del_first used in one consumer, and
+ * llist_del_first or llist_del_all used in other consumers, then a lock is
+ * needed.  This is because llist_del_first depends on list->first->next not
+ * changing, but without lock protection, there's no way to be sure about that
+ * if a preemption happens in the middle of the delete operation and on being
+ * preempted back, the list->first is the same as before causing the cmpxchg in
+ * llist_del_first to succeed. For example, while a llist_del_first operation
+ * is in progress in one consumer, then a llist_del_first, llist_add,
+ * llist_add (or llist_del_all, llist_add, llist_add) sequence in another
+ * consumer may cause violations.
+ *
+ * This can be summarized as follows:
  *
  *           |   add    | del_first |  del_all
  * add       |    -     |     -     |     -
  * del_first |          |     L     |     L
  * del_all   |          |           |     -
  *
- * Where "-" stands for no lock is needed, while "L" stands for lock
- * is needed.
+ * Where, a particular row's operation can happen concurrently with a column's
+ * operation, with "-" being no lock needed, while "L" being lock is needed.
  *
  * The list entries deleted via llist_del_all can be traversed with
  * traversing function such as llist_for_each etc.  But the list
-- 
2.5.2

[PATCH tip/core/rcu 19/20] rcu: Eliminate unused expedited_normal counter

[Paul E. McKenney]

Expedited grace periods no longer fall back to normal grace periods
in response to lock contention, given that expedited grace periods
now use the rcu_node tree so as to avoid contention.  This commit
therfore removes the expedited_normal counter.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 Documentation/RCU/trace.txt | 5 +----
 kernel/rcu/tree.h           | 1 -
 kernel/rcu/tree_trace.c     | 3 +--
 3 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/Documentation/RCU/trace.txt b/Documentation/RCU/trace.txt
index 00a3a38b375a..6549012033f9 100644
--- a/Documentation/RCU/trace.txt
+++ b/Documentation/RCU/trace.txt
@@ -237,7 +237,7 @@ o	"ktl" is the low-order 16 bits (in hexadecimal) of the count of
 
 The output of "cat rcu/rcu_preempt/rcuexp" looks as follows:
 
-s=21872 wd1=0 wd2=0 wd3=5 n=0 enq=0 sc=21872
+s=21872 wd1=0 wd2=0 wd3=5 enq=0 sc=21872
 
 These fields are as follows:
 
@@ -249,9 +249,6 @@ o	"wd1", "wd2", and "wd3" are the number of times that an attempt
 	completed an expedited grace period that satisfies the attempted
 	request.  "Our work is done."
 
-o	"n" is number of times that a concurrent CPU-hotplug operation
-	forced a fallback to a normal grace period.
-
 o	"enq" is the number of quiescent states still outstanding.
 
 o	"sc" is the number of times that the attempt to start a
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..8f750dffb0dd 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -521,7 +521,6 @@ struct rcu_state {
 	struct mutex exp_mutex;			/* Serialize expedited GP. */
 	struct mutex exp_wake_mutex;		/* Serialize wakeup. */
 	unsigned long expedited_sequence;	/* Take a ticket. */
-	atomic_long_t expedited_normal;		/* # fallbacks to normal. */
 	atomic_t expedited_need_qs;		/* # CPUs left to check in. */
 	struct swait_queue_head expedited_wq;	/* Wait for check-ins. */
 	int ncpus_snap;				/* # CPUs seen last time. */
diff --git a/kernel/rcu/tree_trace.c b/kernel/rcu/tree_trace.c
index b1f28972872c..2e932cd1da31 100644
--- a/kernel/rcu/tree_trace.c
+++ b/kernel/rcu/tree_trace.c
@@ -194,9 +194,8 @@ static int show_rcuexp(struct seq_file *m, void *v)
 		s2 += atomic_long_read(&rdp->exp_workdone2);
 		s3 += atomic_long_read(&rdp->exp_workdone3);
 	}
-	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu n=%lu enq=%d sc=%lu\n",
+	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu enq=%d sc=%lu\n",
 		   rsp->expedited_sequence, s0, s1, s2, s3,
-		   atomic_long_read(&rsp->expedited_normal),
 		   atomic_read(&rsp->expedited_need_qs),
 		   rsp->expedited_sequence / 2);
 	return 0;
-- 
2.5.2

[PATCH tip/core/rcu 20/20] rcu: Add lockdep checks to synchronous expedited primitives

[Paul E. McKenney]

The non-expedited synchronize_*rcu() primitives have lockdep checks, but
their expedited counterparts lack these checks.  This commit therefore
adds these checks to the expedited synchronize_*rcu() primitives.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree_exp.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 303df97bbfc5..f3e214898e3a 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -639,6 +639,11 @@ void synchronize_sched_expedited(void)
 {
 	struct rcu_state *rsp = &rcu_sched_state;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_sched_expedited() in RCU read-side critical section");
+
 	/* If only one CPU, this is automatically a grace period. */
 	if (rcu_blocking_is_gp())
 		return;
@@ -708,6 +713,11 @@ void synchronize_rcu_expedited(void)
 {
 	struct rcu_state *rsp = rcu_state_p;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_rcu_expedited() in RCU read-side critical section");
+
 	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)
 		return;
 	_synchronize_rcu_expedited(rsp, sync_rcu_exp_handler);
-- 
2.5.2

Re: [PATCH tip/core/rcu 0/20] Miscellaneous fixes for 4.11

[Josh Triplett]

On Sat, Jan 14, 2017 at 01:12:55AM -0800, Paul E. McKenney wrote:
> Hello!
> 
> This series provides miscellaneous fixes:
> 
> 1.	Make RCU_EXPEDITE_BOOT be the default in order to speed up
> 	boot, courtesy of Sebastian Andrzej Siewior.
> 
> 2.	Make RCU suspicious-access lockdep splats use pr_err().
> 
> 3.	Disable sys_membarrier when nohz_full is enabled, courtesy
> 	of Mathieu Desnoyers.
> 
> 4.	Only dump stalled-tasks stacks if there was a real stall, courtesy
> 	of Byungchul Park.
> 
> 5.	Remove unneeded rcu_process_callbacks() declarations.
> 
> 6.	Remove unused but set "mask" variable, courtesy of Tobias Klauser.
> 
> 7.	Remove short-term CPU kicking.
> 
> 8.	Add long-term CPU kicking.
> 
> 9.	Once again use NMI-based stack traces in stall warnings.
> 
> 10.	Add functions to test for trivial grace periods.
> 
> 11.	Make cond_resched() provide RCU quiescent state, which will
> 	hopefully lead to the removal of cond_resched_rcu_qs().
> 
> 12.	Re-enable TASKS_RCU for User Mode Linux.
> 
> 13.	Don't wake rcuc/X kthreads on NOCB CPUs.
> 
> 14.	Add comment headers to expedited-grace-period counter functions.
> 
> 15.	Make rcu_cpu_starting() use its "cpu" argument.
> 
> 16.	Enable RCU tracepoints by default to aid in debugging, courtesy
> 	of Matt Fleming.
> 
> 17.	Fix comment in rcu_organize_nocb_kthreads(), which no longer
> 	spawns kthreads.
> 
> 18.	Clarify comments about when llist locking is needed, courtesy
> 	of Joel Fernandes.
> 
> 19.	Eliminate unused expedited_normal counter.
> 
> 20.	Add lockdep checks to synchronous expedited primitives.

I replied to patches 2, 8, 10, and 11 with comments.  For the rest:
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Re: [PATCH tip/core/rcu 0/20] Miscellaneous fixes for 4.11

[Paul E. McKenney]

On Mon, Jan 16, 2017 at 12:09:20AM -0800, Josh Triplett wrote:
> On Sat, Jan 14, 2017 at 01:12:55AM -0800, Paul E. McKenney wrote:
> > Hello!
> > 
> > This series provides miscellaneous fixes:
> > 
> > 1.	Make RCU_EXPEDITE_BOOT be the default in order to speed up
> > 	boot, courtesy of Sebastian Andrzej Siewior.
> > 
> > 2.	Make RCU suspicious-access lockdep splats use pr_err().
> > 
> > 3.	Disable sys_membarrier when nohz_full is enabled, courtesy
> > 	of Mathieu Desnoyers.
> > 
> > 4.	Only dump stalled-tasks stacks if there was a real stall, courtesy
> > 	of Byungchul Park.
> > 
> > 5.	Remove unneeded rcu_process_callbacks() declarations.
> > 
> > 6.	Remove unused but set "mask" variable, courtesy of Tobias Klauser.
> > 
> > 7.	Remove short-term CPU kicking.
> > 
> > 8.	Add long-term CPU kicking.
> > 
> > 9.	Once again use NMI-based stack traces in stall warnings.
> > 
> > 10.	Add functions to test for trivial grace periods.
> > 
> > 11.	Make cond_resched() provide RCU quiescent state, which will
> > 	hopefully lead to the removal of cond_resched_rcu_qs().
> > 
> > 12.	Re-enable TASKS_RCU for User Mode Linux.
> > 
> > 13.	Don't wake rcuc/X kthreads on NOCB CPUs.
> > 
> > 14.	Add comment headers to expedited-grace-period counter functions.
> > 
> > 15.	Make rcu_cpu_starting() use its "cpu" argument.
> > 
> > 16.	Enable RCU tracepoints by default to aid in debugging, courtesy
> > 	of Matt Fleming.
> > 
> > 17.	Fix comment in rcu_organize_nocb_kthreads(), which no longer
> > 	spawns kthreads.
> > 
> > 18.	Clarify comments about when llist locking is needed, courtesy
> > 	of Joel Fernandes.
> > 
> > 19.	Eliminate unused expedited_normal counter.
> > 
> > 20.	Add lockdep checks to synchronous expedited primitives.
> 
> I replied to patches 2, 8, 10, and 11 with comments.  For the rest:
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Applied, thank you!

							Thanx, Paul

Re: [PATCH tip/core/rcu 0/20] Miscellaneous fixes for 4.11

[Paul E. McKenney]

On Sat, Jan 14, 2017 at 01:12:55AM -0800, Paul E. McKenney wrote:
Hello!

This series provides miscellaneous fixes:

1.	Make RCU_EXPEDITE_BOOT be the default in order to speed up
	boot, courtesy of Sebastian Andrzej Siewior.

2.	Make RCU suspicious-access lockdep splats use pr_err().

3.	Disable sys_membarrier when nohz_full is enabled, courtesy
	of Mathieu Desnoyers.

4.	Only dump stalled-tasks stacks if there was a real stall, courtesy
	of Byungchul Park.

5.	Remove unneeded rcu_process_callbacks() declarations.

6.	Remove unused but set "mask" variable, courtesy of Tobias Klauser.

7.	Add long-term CPU kicking.

8.	Remove short-term CPU kicking.

9.	Once again use NMI-based stack traces in stall warnings.

10.	Re-enable TASKS_RCU for User Mode Linux.

11.	Don't wake rcuc/X kthreads on NOCB CPUs.

12.	Add comment headers to expedited-grace-period counter functions.

13.	Make rcu_cpu_starting() use its "cpu" argument.

14.	Enable RCU tracepoints by default to aid in debugging, courtesy
	of Matt Fleming.

15.	Fix comment in rcu_organize_nocb_kthreads(), which no longer
	spawns kthreads.

16.	Clarify comments about when llist locking is needed, courtesy
	of Joel Fernandes.

17.	Eliminate unused expedited_normal counter.

18.	Add lockdep checks to synchronous expedited primitives.

Changes since v1:

o	Reverse the order of patches 7 and 8.

o	Defer the patch adding tests for trivial grace periods.

o	Defer the patch making cond_resched() provide RCU quiescent
	states.

o	Apply other feedback from Josh Triplett, Peter Zijlstra, and
	Michal Hocko.

							Thanx, Paul

------------------------------------------------------------------------

 Documentation/RCU/trace.txt |    5 +----
 include/linux/llist.h       |   37 +++++++++++++++++++++----------------
 init/Kconfig                |   14 --------------
 kernel/locking/lockdep.c    |   12 ++++++------
 kernel/membarrier.c         |    4 ++++
 kernel/rcu/tiny.c           |    2 --
 kernel/rcu/tree.c           |   35 ++++++++++++++++++-----------------
 kernel/rcu/tree.h           |    1 -
 kernel/rcu/tree_exp.h       |   28 +++++++++++++++++++++++++++-
 kernel/rcu/tree_plugin.h    |    5 +++--
 kernel/rcu/tree_trace.c     |    3 +--
 kernel/rcu/update.c         |    6 ++----
 lib/Kconfig.debug           |    1 +
 13 files changed, 84 insertions(+), 69 deletions(-)

[PATCH v2 tip/core/rcu 01/18] rcu: update: Make RCU_EXPEDITE_BOOT be the default

[Paul E. McKenney]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

RCU_EXPEDITE_BOOT should speed up the boot process by enforcing
synchronize_rcu_expedited() instead of synchronize_rcu() during the boot
process. There should be no reason why one does not want this and there
is no need worry about real time latency at this point.
Therefore make it default.

Note that users wishing to avoid expediting entirely, for example when
bringing up new hardware possibly having flaky IPIs, can use the
rcu_normal boot parameter to override boot-time expediting.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ paulmck: Reworded commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 init/Kconfig        | 13 -------------
 kernel/rcu/update.c |  6 ++----
 2 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 223b734abccd..96e6d56acd50 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -781,19 +781,6 @@ config RCU_NOCB_CPU_ALL
 
 endchoice
 
-config RCU_EXPEDITE_BOOT
-	bool
-	default n
-	help
-	  This option enables expedited grace periods at boot time,
-	  as if rcu_expedite_gp() had been invoked early in boot.
-	  The corresponding rcu_unexpedite_gp() is invoked from
-	  rcu_end_inkernel_boot(), which is intended to be invoked
-	  at the end of the kernel-only boot sequence, just before
-	  init is exec'ed.
-
-	  Accept the default if unsure.
-
 endmenu # "RCU Subsystem"
 
 config BUILD_BIN2C
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 4f6db7e6a117..9e03db9ea9c0 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -132,8 +132,7 @@ bool rcu_gp_is_normal(void)
 }
 EXPORT_SYMBOL_GPL(rcu_gp_is_normal);
 
-static atomic_t rcu_expedited_nesting =
-	ATOMIC_INIT(IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT) ? 1 : 0);
+static atomic_t rcu_expedited_nesting = ATOMIC_INIT(1);
 
 /*
  * Should normal grace-period primitives be expedited?  Intended for
@@ -182,8 +181,7 @@ EXPORT_SYMBOL_GPL(rcu_unexpedite_gp);
  */
 void rcu_end_inkernel_boot(void)
 {
-	if (IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT))
-		rcu_unexpedite_gp();
+	rcu_unexpedite_gp();
 	if (rcu_normal_after_boot)
 		WRITE_ONCE(rcu_normal, 1);
 }
-- 
2.5.2

[PATCH v2 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

This commit switches RCU suspicious-access splats use pr_err()
instead of the current INFO printk()s.  This change makes it easier
to automatically classify splats.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/locking/lockdep.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index 7c38f8f3d97b..a74c0630172a 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
 	printk("\n");
-	printk("===============================\n");
-	printk("[ INFO: suspicious RCU usage. ]\n");
+	pr_err("===============================\n");
+	pr_err("[  suspicious RCU usage.      ]\n");
 	print_kernel_ident();
-	printk("-------------------------------\n");
-	printk("%s:%d %s!\n", file, line, s);
-	printk("\nother info that might help us debug this:\n\n");
-	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
+	pr_err("-------------------------------\n");
+	pr_err("%s:%d %s!\n", file, line, s);
+	pr_err("\nother info that might help us debug this:\n\n");
+	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
 	       !rcu_lockdep_current_cpu_online()
 			? "RCU used illegally from offline CPU!\n"
 			: !rcu_is_watching()
-- 
2.5.2

Re: [PATCH v2 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Josh Triplett]

On Tue, Jan 17, 2017 at 06:53:41PM -0800, Paul E. McKenney wrote:
> This commit switches RCU suspicious-access splats use pr_err()
> instead of the current INFO printk()s.  This change makes it easier
> to automatically classify splats.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/locking/lockdep.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> index 7c38f8f3d97b..a74c0630172a 100644
> --- a/kernel/locking/lockdep.c
> +++ b/kernel/locking/lockdep.c
> @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
>  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
>  	/* Note: the following can be executed concurrently, so be careful. */
>  	printk("\n");
> -	printk("===============================\n");
> -	printk("[ INFO: suspicious RCU usage. ]\n");
> +	pr_err("===============================\n");
> +	pr_err("[  suspicious RCU usage.      ]\n");

While re-adding the square bracket makes it symmetric, this change still
seems odd, and unrelated to the switch to pr_err.  You could change it
to "ERR:" if you want, if "INFO:" feels inaccurate to you.

>  	print_kernel_ident();
> -	printk("-------------------------------\n");
> -	printk("%s:%d %s!\n", file, line, s);
> -	printk("\nother info that might help us debug this:\n\n");
> -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> +	pr_err("-------------------------------\n");
> +	pr_err("%s:%d %s!\n", file, line, s);
> +	pr_err("\nother info that might help us debug this:\n\n");
> +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
>  	       !rcu_lockdep_current_cpu_online()
>  			? "RCU used illegally from offline CPU!\n"
>  			: !rcu_is_watching()
> -- 
> 2.5.2
> 

Re: [PATCH v2 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

On Sat, Jan 21, 2017 at 12:40:23PM -0800, Josh Triplett wrote:
> On Tue, Jan 17, 2017 at 06:53:41PM -0800, Paul E. McKenney wrote:
> > This commit switches RCU suspicious-access splats use pr_err()
> > instead of the current INFO printk()s.  This change makes it easier
> > to automatically classify splats.
> > 
> > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/locking/lockdep.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> > index 7c38f8f3d97b..a74c0630172a 100644
> > --- a/kernel/locking/lockdep.c
> > +++ b/kernel/locking/lockdep.c
> > @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
> >  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
> >  	/* Note: the following can be executed concurrently, so be careful. */
> >  	printk("\n");
> > -	printk("===============================\n");
> > -	printk("[ INFO: suspicious RCU usage. ]\n");
> > +	pr_err("===============================\n");
> > +	pr_err("[  suspicious RCU usage.      ]\n");
> 
> While re-adding the square bracket makes it symmetric, this change still
> seems odd, and unrelated to the switch to pr_err.  You could change it
> to "ERR:" if you want, if "INFO:" feels inaccurate to you.

So this would be OK?

	pr_err("[ ERR: suspicious RCU usage.  ]\n");

(Changed to this as a best guess, but please let me know.)

							Thanx, Paul

> >  	print_kernel_ident();
> > -	printk("-------------------------------\n");
> > -	printk("%s:%d %s!\n", file, line, s);
> > -	printk("\nother info that might help us debug this:\n\n");
> > -	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> > +	pr_err("-------------------------------\n");
> > +	pr_err("%s:%d %s!\n", file, line, s);
> > +	pr_err("\nother info that might help us debug this:\n\n");
> > +	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
> >  	       !rcu_lockdep_current_cpu_online()
> >  			? "RCU used illegally from offline CPU!\n"
> >  			: !rcu_is_watching()
> > -- 
> > 2.5.2
> > 
> 

Re: [PATCH v2 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Josh Triplett]

On Mon, Jan 23, 2017 at 11:30:16AM -0800, Paul E. McKenney wrote:
> On Sat, Jan 21, 2017 at 12:40:23PM -0800, Josh Triplett wrote:
> > On Tue, Jan 17, 2017 at 06:53:41PM -0800, Paul E. McKenney wrote:
> > > This commit switches RCU suspicious-access splats use pr_err()
> > > instead of the current INFO printk()s.  This change makes it easier
> > > to automatically classify splats.
> > > 
> > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > ---
> > >  kernel/locking/lockdep.c | 12 ++++++------
> > >  1 file changed, 6 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> > > index 7c38f8f3d97b..a74c0630172a 100644
> > > --- a/kernel/locking/lockdep.c
> > > +++ b/kernel/locking/lockdep.c
> > > @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
> > >  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
> > >  	/* Note: the following can be executed concurrently, so be careful. */
> > >  	printk("\n");
> > > -	printk("===============================\n");
> > > -	printk("[ INFO: suspicious RCU usage. ]\n");
> > > +	pr_err("===============================\n");
> > > +	pr_err("[  suspicious RCU usage.      ]\n");
> > 
> > While re-adding the square bracket makes it symmetric, this change still
> > seems odd, and unrelated to the switch to pr_err.  You could change it
> > to "ERR:" if you want, if "INFO:" feels inaccurate to you.
> 
> So this would be OK?
> 
> 	pr_err("[ ERR: suspicious RCU usage.  ]\n");
> 
> (Changed to this as a best guess, but please let me know.)

Yeah, that seems fine.  Sorry to nitpick this; it just otherwise seemed
entirely unrelated to the rest of the change.

Re: [PATCH v2 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

On Mon, Jan 23, 2017 at 02:33:19PM -0800, Josh Triplett wrote:
> On Mon, Jan 23, 2017 at 11:30:16AM -0800, Paul E. McKenney wrote:
> > On Sat, Jan 21, 2017 at 12:40:23PM -0800, Josh Triplett wrote:
> > > On Tue, Jan 17, 2017 at 06:53:41PM -0800, Paul E. McKenney wrote:
> > > > This commit switches RCU suspicious-access splats use pr_err()
> > > > instead of the current INFO printk()s.  This change makes it easier
> > > > to automatically classify splats.
> > > > 
> > > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > ---
> > > >  kernel/locking/lockdep.c | 12 ++++++------
> > > >  1 file changed, 6 insertions(+), 6 deletions(-)
> > > > 
> > > > diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
> > > > index 7c38f8f3d97b..a74c0630172a 100644
> > > > --- a/kernel/locking/lockdep.c
> > > > +++ b/kernel/locking/lockdep.c
> > > > @@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
> > > >  #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
> > > >  	/* Note: the following can be executed concurrently, so be careful. */
> > > >  	printk("\n");
> > > > -	printk("===============================\n");
> > > > -	printk("[ INFO: suspicious RCU usage. ]\n");
> > > > +	pr_err("===============================\n");
> > > > +	pr_err("[  suspicious RCU usage.      ]\n");
> > > 
> > > While re-adding the square bracket makes it symmetric, this change still
> > > seems odd, and unrelated to the switch to pr_err.  You could change it
> > > to "ERR:" if you want, if "INFO:" feels inaccurate to you.
> > 
> > So this would be OK?
> > 
> > 	pr_err("[ ERR: suspicious RCU usage.  ]\n");
> > 
> > (Changed to this as a best guess, but please let me know.)
> 
> Yeah, that seems fine.  Sorry to nitpick this; it just otherwise seemed
> entirely unrelated to the rest of the change.

Might as well get it right...  ;-)

							Thanx, Paul

[PATCH v2 tip/core/rcu 03/18] Fix: Disable sys_membarrier when nohz_full is enabled

[Paul E. McKenney]

From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Userspace applications should be allowed to expect the membarrier system
call with MEMBARRIER_CMD_SHARED command to issue memory barriers on
nohz_full CPUs, but synchronize_sched() does not take those into
account.

Given that we do not want unrelated processes to be able to affect
real-time sensitive nohz_full CPUs, simply return ENOSYS when membarrier
is invoked on a kernel with enabled nohz_full CPUs.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Josh Triplett <josh@joshtriplett.org>
CC: Steven Rostedt <rostedt@goodmis.org>
CC: <stable@vger.kernel.org>	[3.10+]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Chris Metcalf <cmetcalf@mellanox.com>
Cc: Rik van Riel <riel@redhat.com>
Acked-by: Lai Jiangshan <jiangshanlai@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/membarrier.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/membarrier.c b/kernel/membarrier.c
index 536c727a56e9..9f9284f37f8d 100644
--- a/kernel/membarrier.c
+++ b/kernel/membarrier.c
@@ -16,6 +16,7 @@
 
 #include <linux/syscalls.h>
 #include <linux/membarrier.h>
+#include <linux/tick.h>
 
 /*
  * Bitmask made from a "or" of all commands within enum membarrier_cmd,
@@ -51,6 +52,9 @@
  */
 SYSCALL_DEFINE2(membarrier, int, cmd, int, flags)
 {
+	/* MEMBARRIER_CMD_SHARED is not compatible with nohz_full. */
+	if (tick_nohz_full_enabled())
+		return -ENOSYS;
 	if (unlikely(flags))
 		return -EINVAL;
 	switch (cmd) {
-- 
2.5.2

[PATCH v2 tip/core/rcu 04/18] rcu: Only dump stalled-tasks stacks if there was a real stall

[Paul E. McKenney]

From: Byungchul Park <byungchul.park@lge.com>

The print_other_cpu_stall() function currently unconditionally invokes
rcu_print_detail_task_stall().  This is OK because if there was a stall
sufficient to cause print_other_cpu_stall() to be invoked, that stall
is very likely to persist through the entire print_other_cpu_stall()
execution.  However, if the stall did not persist, the variable ndetected
will be zero, and that variable is already tested in an "if" statement.
Therefore, this commit moves the call to rcu_print_detail_task_stall()
under that pre-existing "if" to improve readability, with a very rare
reduction in overhead.

Signed-off-by: Byungchul Park <byungchul.park@lge.com>
[ paulmck: Reworked commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cb4e2056ccf3..6232d2f9a84e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1379,6 +1379,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 	       (long)rsp->gpnum, (long)rsp->completed, totqlen);
 	if (ndetected) {
 		rcu_dump_cpu_stacks(rsp);
+
+		/* Complain about tasks blocking the grace period. */
+		rcu_print_detail_task_stall(rsp);
 	} else {
 		if (READ_ONCE(rsp->gpnum) != gpnum ||
 		    READ_ONCE(rsp->completed) == gpnum) {
@@ -1395,9 +1398,6 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 		}
 	}
 
-	/* Complain about tasks blocking the grace period. */
-	rcu_print_detail_task_stall(rsp);
-
 	rcu_check_gp_kthread_starvation(rsp);
 
 	panic_on_rcu_stall();
-- 
2.5.2

[PATCH v2 tip/core/rcu 05/18] rcu: Remove unneeded rcu_process_callbacks() declarations

[Paul E. McKenney]

The declarations of __rcu_process_callbacks() and rcu_process_callbacks()
are not needed, as the definition of both of these functions appear before
any uses.  This commit therefore removes both declarations.

Reported-by: "Ahmed, Iftekhar" <ahmedi@oregonstate.edu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tiny.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index b23a4d076f3d..fa6a48d3917b 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -41,8 +41,6 @@
 
 /* Forward declarations for tiny_plugin.h. */
 struct rcu_ctrlblk;
-static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp);
-static void rcu_process_callbacks(struct softirq_action *unused);
 static void __call_rcu(struct rcu_head *head,
 		       rcu_callback_t func,
 		       struct rcu_ctrlblk *rcp);
-- 
2.5.2

[PATCH v2 tip/core/rcu 06/18] rcu: Remove unused but set variable

[Paul E. McKenney]

From: Tobias Klauser <tklauser@distanz.ch>

Since commit 7ec99de36f40 ("rcu: Provide exact CPU-online tracking for
RCU"), the variable mask in rcu_init_percpu_data is set but no longer
used. Remove it to fix the following warning when building with 'W=1':

  kernel/rcu/tree.c: In function ‘rcu_init_percpu_data’:
  kernel/rcu/tree.c:3765:16: warning: variable ‘mask’ set but not used [-Wunused-but-set-variable]

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6232d2f9a84e..83bf054e194e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3765,7 +3765,6 @@ static void
 rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
-	unsigned long mask;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rcu_get_root(rsp);
 
@@ -3788,7 +3787,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 	 * of the next grace period.
 	 */
 	rnp = rdp->mynode;
-	mask = rdp->grpmask;
 	raw_spin_lock_rcu_node(rnp);		/* irqs already disabled. */
 	if (!rdp->beenonline)
 		WRITE_ONCE(rsp->ncpus, READ_ONCE(rsp->ncpus) + 1);
-- 
2.5.2

[PATCH v2 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Paul E. McKenney]

Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
frequent calls to resched_cpu(), which means that the only time
resched_cpu() is invoked is after an RCU CPU stall warning.  Although
this is good from an avoid-IPIs perspective, we should try to break
things loose -before- splatting.  This commit therefore starts invoking
resched_cpu() for each holdout at each force-quiescent-state interval
that is more than halfway through the stall-warning interval.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 83bf054e194e..0e61b62e3f4a 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
 	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
 		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
+	/*
+	 * If more than halfway to RCU CPU stall-warning time, do
+	 * a resched_cpu() to try to loosen things up a bit.
+	 */
+	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
+		resched_cpu(rdp->cpu);
 
 	return 0;
 }
-- 
2.5.2

Re: [PATCH v2 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Josh Triplett]

On Tue, Jan 17, 2017 at 06:53:46PM -0800, Paul E. McKenney wrote:
> Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> frequent calls to resched_cpu(), which means that the only time
> resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> this is good from an avoid-IPIs perspective, we should try to break
> things loose -before- splatting.  This commit therefore starts invoking
> resched_cpu() for each holdout at each force-quiescent-state interval
> that is more than halfway through the stall-warning interval.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

> ---
>  kernel/rcu/tree.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 83bf054e194e..0e61b62e3f4a 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
>  			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
>  	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
>  		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
> +	/*
> +	 * If more than halfway to RCU CPU stall-warning time, do
> +	 * a resched_cpu() to try to loosen things up a bit.
> +	 */
> +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> +		resched_cpu(rdp->cpu);
>  
>  	return 0;
>  }
> -- 
> 2.5.2
> 

Re: [PATCH v2 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Josh Triplett]

On Tue, Jan 17, 2017 at 06:53:46PM -0800, Paul E. McKenney wrote:
> Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> frequent calls to resched_cpu(), which means that the only time
> resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> this is good from an avoid-IPIs perspective, we should try to break
> things loose -before- splatting.  This commit therefore starts invoking
> resched_cpu() for each holdout at each force-quiescent-state interval
> that is more than halfway through the stall-warning interval.

Just realized an issue with this commit message: you're referring to
what now appears as patch 8 in the past tense as something already done,
and with a commit ID that probably doesn't work anymore.  You need to
rephrase this to describe how it leads to a *subsequent* change rather
than fixing something already changed.

With the commit message fixed:
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>



> ---
>  kernel/rcu/tree.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 83bf054e194e..0e61b62e3f4a 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
>  			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
>  	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
>  		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
> +	/*
> +	 * If more than halfway to RCU CPU stall-warning time, do
> +	 * a resched_cpu() to try to loosen things up a bit.
> +	 */
> +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> +		resched_cpu(rdp->cpu);
>  
>  	return 0;
>  }
> -- 
> 2.5.2
> 

Re: [PATCH v2 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Paul E. McKenney]

On Sat, Jan 21, 2017 at 12:42:55PM -0800, Josh Triplett wrote:
> On Tue, Jan 17, 2017 at 06:53:46PM -0800, Paul E. McKenney wrote:
> > Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> > frequent calls to resched_cpu(), which means that the only time
> > resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> > this is good from an avoid-IPIs perspective, we should try to break
> > things loose -before- splatting.  This commit therefore starts invoking
> > resched_cpu() for each holdout at each force-quiescent-state interval
> > that is more than halfway through the stall-warning interval.
> 
> Just realized an issue with this commit message: you're referring to
> what now appears as patch 8 in the past tense as something already done,
> and with a commit ID that probably doesn't work anymore.  You need to
> rephrase this to describe how it leads to a *subsequent* change rather
> than fixing something already changed.
> 
> With the commit message fixed:
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Good point!  It now reads as follows:

------------------------------------------------------------------------
This commit prepares for the removal of short-term CPU kicking (in a
subsequent commit).  It does so by starting to invoke resched_cpu()
for each holdout at each force-quiescent-state interval that is more
than halfway through the stall-warning interval.
------------------------------------------------------------------------

Does that work?

							Thanx, Paul

> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> 
> 
> > ---
> >  kernel/rcu/tree.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 83bf054e194e..0e61b62e3f4a 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
> >  			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
> >  	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
> >  		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
> > +	/*
> > +	 * If more than halfway to RCU CPU stall-warning time, do
> > +	 * a resched_cpu() to try to loosen things up a bit.
> > +	 */
> > +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> > +		resched_cpu(rdp->cpu);
> >  
> >  	return 0;
> >  }
> > -- 
> > 2.5.2
> > 
> 

Re: [PATCH v2 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Josh Triplett]

On Mon, Jan 23, 2017 at 11:34:45AM -0800, Paul E. McKenney wrote:
> On Sat, Jan 21, 2017 at 12:42:55PM -0800, Josh Triplett wrote:
> > On Tue, Jan 17, 2017 at 06:53:46PM -0800, Paul E. McKenney wrote:
> > > Commit d2db185bfee8 ("rcu: Remove short-term CPU kicking") removed
> > > frequent calls to resched_cpu(), which means that the only time
> > > resched_cpu() is invoked is after an RCU CPU stall warning.  Although
> > > this is good from an avoid-IPIs perspective, we should try to break
> > > things loose -before- splatting.  This commit therefore starts invoking
> > > resched_cpu() for each holdout at each force-quiescent-state interval
> > > that is more than halfway through the stall-warning interval.
> > 
> > Just realized an issue with this commit message: you're referring to
> > what now appears as patch 8 in the past tense as something already done,
> > and with a commit ID that probably doesn't work anymore.  You need to
> > rephrase this to describe how it leads to a *subsequent* change rather
> > than fixing something already changed.
> > 
> > With the commit message fixed:
> > Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> 
> Good point!  It now reads as follows:
> 
> ------------------------------------------------------------------------
> This commit prepares for the removal of short-term CPU kicking (in a
> subsequent commit).  It does so by starting to invoke resched_cpu()
> for each holdout at each force-quiescent-state interval that is more
> than halfway through the stall-warning interval.
> ------------------------------------------------------------------------
> 
> Does that work?

Looks good to me.

> 							Thanx, Paul
> 
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > 
> > 
> > 
> > > ---
> > >  kernel/rcu/tree.c | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > > 
> > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > index 83bf054e194e..0e61b62e3f4a 100644
> > > --- a/kernel/rcu/tree.c
> > > +++ b/kernel/rcu/tree.c
> > > @@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
> > >  			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
> > >  	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
> > >  		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
> > > +	/*
> > > +	 * If more than halfway to RCU CPU stall-warning time, do
> > > +	 * a resched_cpu() to try to loosen things up a bit.
> > > +	 */
> > > +	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
> > > +		resched_cpu(rdp->cpu);
> > >  
> > >  	return 0;
> > >  }
> > > -- 
> > > 2.5.2
> > > 
> > 
> 

[PATCH v2 tip/core/rcu 08/18] rcu: Remove short-term CPU kicking

[Paul E. McKenney]

Commit 4914950aaa12d ("rcu: Stop treating in-kernel CPU-bound workloads
as errors") added a (relatively) short-timeout call to resched_cpu().
This was inspired by as issue that was fixed by b7e7ade34e61 ("sched/core:
Fix remote wakeups").  But given that this issue was fixed, it is time
for the current commit to remove this call to resched_cpu().

Reported-by: Byungchul Park <byungchul.park@lge.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 0e61b62e3f4a..5a4aaad75e76 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1220,11 +1220,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
 	}
 
-	/* And if it has been a really long time, kick the CPU as well. */
-	if (ULONG_CMP_GE(jiffies,
-			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
-	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
-		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
 	/*
 	 * If more than halfway to RCU CPU stall-warning time, do
 	 * a resched_cpu() to try to loosen things up a bit.
-- 
2.5.2

Re: [PATCH v2 tip/core/rcu 08/18] rcu: Remove short-term CPU kicking

[Josh Triplett]

On Tue, Jan 17, 2017 at 06:53:47PM -0800, Paul E. McKenney wrote:
> Commit 4914950aaa12d ("rcu: Stop treating in-kernel CPU-bound workloads

Does this commit ID still match what you have in your tree, post-rebase?

> as errors") added a (relatively) short-timeout call to resched_cpu().
> This was inspired by as issue that was fixed by b7e7ade34e61 ("sched/core:
> Fix remote wakeups").  But given that this issue was fixed, it is time
> for the current commit to remove this call to resched_cpu().
> 
> Reported-by: Byungchul Park <byungchul.park@lge.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> ---
>  kernel/rcu/tree.c | 5 -----
>  1 file changed, 5 deletions(-)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 0e61b62e3f4a..5a4aaad75e76 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1220,11 +1220,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
>  		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
>  	}
>  
> -	/* And if it has been a really long time, kick the CPU as well. */
> -	if (ULONG_CMP_GE(jiffies,
> -			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
> -	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
> -		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
>  	/*
>  	 * If more than halfway to RCU CPU stall-warning time, do
>  	 * a resched_cpu() to try to loosen things up a bit.
> -- 
> 2.5.2
> 

Re: [PATCH v2 tip/core/rcu 08/18] rcu: Remove short-term CPU kicking

[Paul E. McKenney]

On Sat, Jan 21, 2017 at 12:43:44PM -0800, Josh Triplett wrote:
> On Tue, Jan 17, 2017 at 06:53:47PM -0800, Paul E. McKenney wrote:
> > Commit 4914950aaa12d ("rcu: Stop treating in-kernel CPU-bound workloads
> 
> Does this commit ID still match what you have in your tree, post-rebase?

In light of your earlier email, good question!  But this one is still
OK because it is already in mainline.

							Thanx, Paul

> > as errors") added a (relatively) short-timeout call to resched_cpu().
> > This was inspired by as issue that was fixed by b7e7ade34e61 ("sched/core:
> > Fix remote wakeups").  But given that this issue was fixed, it is time
> > for the current commit to remove this call to resched_cpu().
> > 
> > Reported-by: Byungchul Park <byungchul.park@lge.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> > ---
> >  kernel/rcu/tree.c | 5 -----
> >  1 file changed, 5 deletions(-)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 0e61b62e3f4a..5a4aaad75e76 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -1220,11 +1220,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
> >  		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
> >  	}
> >  
> > -	/* And if it has been a really long time, kick the CPU as well. */
> > -	if (ULONG_CMP_GE(jiffies,
> > -			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
> > -	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
> > -		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
> >  	/*
> >  	 * If more than halfway to RCU CPU stall-warning time, do
> >  	 * a resched_cpu() to try to loosen things up a bit.
> > -- 
> > 2.5.2
> > 
> 

[PATCH v2 tip/core/rcu 09/18] rcu: Once again use NMI-based stack traces in stall warnings

[Paul E. McKenney]

This commit is for all intents and purposes a revert of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks").  The reason to suppose
that this can now safely be reverted is the presence of 42a0bb3f7138
("printk/nmi: generic solution for safe printk in NMI"), which is said
to have made NMI-based stack dumps safe.

However, this reversion keeps one nice property of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks"), namely that
only those CPUs blocking the grace period are dumped.  The new
trigger_single_cpu_backtrace() is used to make this happen, as
suggested by Josh Poimboeuf.

Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 5a4aaad75e76..d7b63b88434b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1278,7 +1278,10 @@ static void rcu_check_gp_kthread_starvation(struct rcu_state *rsp)
 }
 
 /*
- * Dump stacks of all tasks running on stalled CPUs.
+ * Dump stacks of all tasks running on stalled CPUs.  First try using
+ * NMIs, but fall back to manual remote stack tracing on architectures
+ * that don't support NMI-based stack dumps.  The NMI-triggered stack
+ * traces are more accurate because they are printed by the target CPU.
  */
 static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 {
@@ -1288,11 +1291,10 @@ static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 
 	rcu_for_each_leaf_node(rsp, rnp) {
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-		if (rnp->qsmask != 0) {
-			for_each_leaf_node_possible_cpu(rnp, cpu)
-				if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+		for_each_leaf_node_possible_cpu(rnp, cpu)
+			if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+				if (!trigger_single_cpu_backtrace(cpu))
 					dump_cpu_task(cpu);
-		}
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	}
 }
-- 
2.5.2

[PATCH v2 tip/core/rcu 10/18] rcu: Re-enable TASKS_RCU for User Mode Linux

[Paul E. McKenney]

Now that User Mode Linux supports arch_irqs_disabled_flags(), this
commit re-enables TASKS_RCU for User Mode Linux.

Reported-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 init/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/init/Kconfig b/init/Kconfig
index 96e6d56acd50..d0caacb4bc14 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -529,7 +529,6 @@ config SRCU
 config TASKS_RCU
 	bool
 	default n
-	depends on !UML
 	select SRCU
 	help
 	  This option enables a task-based RCU implementation that uses
-- 
2.5.2

[PATCH v2 tip/core/rcu 11/18] rcu: Don't wake rcuc/X kthreads on NOCB CPUs

[Paul E. McKenney]

Chris Friesen notice that rcuc/X kthreads were consuming CPU even on
NOCB CPUs.  This makes no sense because the only purpose or these
kthreads is to invoke normal (non-offloaded) callbacks, of which there
will never be any on NOCB CPUs.  This problem was due to a bug in
cpu_has_callbacks_ready_to_invoke(), which should have been checking
->nxttail[RCU_NEXT_TAIL] for NULL, but which was instead (incorrectly)
checking ->nxttail[RCU_DONE_TAIL].  Because ->nxttail[RCU_DONE_TAIL] is
never NULL, the only effect is to cause the rcuc/X kthread to execute
when it should not do so.

This commit therefore checks ->nxttail[RCU_NEXT_TAIL], which is NULL
for NOCB CPUs.

Reported-by: Chris Friesen <chris.friesen@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d7b63b88434b..be2301238a23 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -611,7 +611,7 @@ static int
 cpu_has_callbacks_ready_to_invoke(struct rcu_data *rdp)
 {
 	return &rdp->nxtlist != rdp->nxttail[RCU_DONE_TAIL] &&
-	       rdp->nxttail[RCU_DONE_TAIL] != NULL;
+	       rdp->nxttail[RCU_NEXT_TAIL] != NULL;
 }
 
 /*
-- 
2.5.2

[PATCH v2 tip/core/rcu 12/18] rcu: Add comment headers to expedited-grace-period counter functions

[Paul E. McKenney]

These functions (rcu_exp_gp_seq_start(), rcu_exp_gp_seq_end(),
rcu_exp_gp_seq_snap(), and rcu_exp_gp_seq_done() seemed too obvious
to comment when written, but not so much when being documented.
This commit therefore adds header comments to each of them.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_exp.h | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index e59e1849b89a..303df97bbfc5 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -20,16 +20,26 @@
  * Authors: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
  */
 
-/* Wrapper functions for expedited grace periods.  */
+/*
+ * Record the start of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_start(struct rcu_state *rsp)
 {
 	rcu_seq_start(&rsp->expedited_sequence);
 }
+
+/*
+ * Record the end of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_end(struct rcu_state *rsp)
 {
 	rcu_seq_end(&rsp->expedited_sequence);
 	smp_mb(); /* Ensure that consecutive grace periods serialize. */
 }
+
+/*
+ * Take a snapshot of the expedited-grace-period counter.
+ */
 static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 {
 	unsigned long s;
@@ -39,6 +49,12 @@ static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 	trace_rcu_exp_grace_period(rsp->name, s, TPS("snap"));
 	return s;
 }
+
+/*
+ * Given a counter snapshot from rcu_exp_gp_seq_snap(), return true
+ * if a full expedited grace period has elapsed since that snapshot
+ * was taken.
+ */
 static bool rcu_exp_gp_seq_done(struct rcu_state *rsp, unsigned long s)
 {
 	return rcu_seq_done(&rsp->expedited_sequence, s);
-- 
2.5.2

[PATCH v2 tip/core/rcu 13/18] rcu: Make rcu_cpu_starting() use its "cpu" argument

[Paul E. McKenney]

The rcu_cpu_starting() function uses this_cpu_ptr() to locate the
incoming CPU's rcu_data structure.  This works for the boot CPU and for
all CPUs onlined after rcu_init() executes (during very early boot).
Currently, this is the full set of CPUs, so all is well.  But if
anyone ever parallelizes boot before rcu_init() time, it will fail.
This commit therefore substitutes the rcu_cpu_starting() function's
this_cpu_pointer() for per_cpu_ptr(), future-proofing the code and
(arguably) improving readability.

This commit inadvertently fixes a latent bug: If there ever had been
more than just the boot CPU online at rcu_init() time, the old code
would not initialize the non-boot CPUs, but rather would repeatedly
initialize the boot CPU.

Reported-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index be2301238a23..a4b4762442bb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3873,7 +3873,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	struct rcu_state *rsp;
 
 	for_each_rcu_flavor(rsp) {
-		rdp = this_cpu_ptr(rsp->rda);
+		rdp = per_cpu_ptr(rsp->rda, cpu);
 		rnp = rdp->mynode;
 		mask = rdp->grpmask;
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-- 
2.5.2

[PATCH v2 tip/core/rcu 14/18] rcu: Enable RCU tracepoints by default to aid in debugging

[Paul E. McKenney]

From: Matt Fleming <matt@codeblueprint.co.uk>

While debugging a performance issue I needed to understand why
RCU sofitrqs were firing so frequently.

Unfortunately, the RCU callback tracepoints are hidden behind
CONFIG_RCU_TRACE which defaults to off in the upstream kernel and is
likely to also be disabled in enterprise distribution configs.

Enable it by default for CONFIG_TREE_RCU. However, we must keep it
disabled for tiny RCU, because it would otherwise pull in a large
amount of code that would make tiny RCU less than tiny.

I ran some file system metadata intensive workloads (git checkout,
FS-Mark) on a variety of machines with this patch and saw no
detectable change in performance.

Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 lib/Kconfig.debug | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index b06848a104e6..5655bd95919a 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1450,6 +1450,7 @@ config RCU_CPU_STALL_TIMEOUT
 config RCU_TRACE
 	bool "Enable tracing for RCU"
 	depends on DEBUG_KERNEL
+	default y if TREE_RCU
 	select TRACE_CLOCK
 	help
 	  This option provides tracing in RCU which presents stats
-- 
2.5.2

[PATCH v2 tip/core/rcu 15/18] rcu: Fix comment in rcu_organize_nocb_kthreads()

[Paul E. McKenney]

It used to be that the rcuo callback-offload kthreads were spawned
in rcu_organize_nocb_kthreads(), and the comment before the "for"
loop says as much.  However, this spawning has long since moved to
the CPU-hotplug code, so this commit fixes this comment.

Reported-by: Michalis Kokologiannakis <mixaskok@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_plugin.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 56583e764ebf..2f5541f6f031 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2366,8 +2366,9 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp)
 	}
 
 	/*
-	 * Each pass through this loop sets up one rcu_data structure and
-	 * spawns one rcu_nocb_kthread().
+	 * Each pass through this loop sets up one rcu_data structure.
+	 * Should the corresponding CPU come online in the future, then
+	 * we will spawn the needed set of rcu_nocb_kthread() kthreads.
 	 */
 	for_each_cpu(cpu, rcu_nocb_mask) {
 		rdp = per_cpu_ptr(rsp->rda, cpu);
-- 
2.5.2

[PATCH v2 tip/core/rcu 16/18] llist: Clarify comments about when locking is needed

[Paul E. McKenney]

From: Joel Fernandes <joelaf@google.com>

llist.h comments are confusing about when locking is needed versus when it
isn't. Clarify these comments by being more descriptive about why locking is
needed for llist_del_first.

Cc: Ingo Molnar <mingo@kernel.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Huang Ying <ying.huang@intel.com>
Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 include/linux/llist.h | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)

diff --git a/include/linux/llist.h b/include/linux/llist.h
index fd4ca0b4fe0f..171baa90f6f6 100644
--- a/include/linux/llist.h
+++ b/include/linux/llist.h
@@ -3,28 +3,33 @@
 /*
  * Lock-less NULL terminated single linked list
  *
- * If there are multiple producers and multiple consumers, llist_add
- * can be used in producers and llist_del_all can be used in
- * consumers.  They can work simultaneously without lock.  But
- * llist_del_first can not be used here.  Because llist_del_first
- * depends on list->first->next does not changed if list->first is not
- * changed during its operation, but llist_del_first, llist_add,
- * llist_add (or llist_del_all, llist_add, llist_add) sequence in
- * another consumer may violate that.
- *
- * If there are multiple producers and one consumer, llist_add can be
- * used in producers and llist_del_all or llist_del_first can be used
- * in the consumer.
- *
- * This can be summarized as follow:
+ * Cases where locking is not needed:
+ * If there are multiple producers and multiple consumers, llist_add can be
+ * used in producers and llist_del_all can be used in consumers simultaneously
+ * without locking. Also a single consumer can use llist_del_first while
+ * multiple producers simultaneously use llist_add, without any locking.
+ *
+ * Cases where locking is needed:
+ * If we have multiple consumers with llist_del_first used in one consumer, and
+ * llist_del_first or llist_del_all used in other consumers, then a lock is
+ * needed.  This is because llist_del_first depends on list->first->next not
+ * changing, but without lock protection, there's no way to be sure about that
+ * if a preemption happens in the middle of the delete operation and on being
+ * preempted back, the list->first is the same as before causing the cmpxchg in
+ * llist_del_first to succeed. For example, while a llist_del_first operation
+ * is in progress in one consumer, then a llist_del_first, llist_add,
+ * llist_add (or llist_del_all, llist_add, llist_add) sequence in another
+ * consumer may cause violations.
+ *
+ * This can be summarized as follows:
  *
  *           |   add    | del_first |  del_all
  * add       |    -     |     -     |     -
  * del_first |          |     L     |     L
  * del_all   |          |           |     -
  *
- * Where "-" stands for no lock is needed, while "L" stands for lock
- * is needed.
+ * Where, a particular row's operation can happen concurrently with a column's
+ * operation, with "-" being no lock needed, while "L" being lock is needed.
  *
  * The list entries deleted via llist_del_all can be traversed with
  * traversing function such as llist_for_each etc.  But the list
-- 
2.5.2

[PATCH v2 tip/core/rcu 17/18] rcu: Eliminate unused expedited_normal counter

[Paul E. McKenney]

Expedited grace periods no longer fall back to normal grace periods
in response to lock contention, given that expedited grace periods
now use the rcu_node tree so as to avoid contention.  This commit
therfore removes the expedited_normal counter.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 Documentation/RCU/trace.txt | 5 +----
 kernel/rcu/tree.h           | 1 -
 kernel/rcu/tree_trace.c     | 3 +--
 3 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/Documentation/RCU/trace.txt b/Documentation/RCU/trace.txt
index 00a3a38b375a..6549012033f9 100644
--- a/Documentation/RCU/trace.txt
+++ b/Documentation/RCU/trace.txt
@@ -237,7 +237,7 @@ o	"ktl" is the low-order 16 bits (in hexadecimal) of the count of
 
 The output of "cat rcu/rcu_preempt/rcuexp" looks as follows:
 
-s=21872 wd1=0 wd2=0 wd3=5 n=0 enq=0 sc=21872
+s=21872 wd1=0 wd2=0 wd3=5 enq=0 sc=21872
 
 These fields are as follows:
 
@@ -249,9 +249,6 @@ o	"wd1", "wd2", and "wd3" are the number of times that an attempt
 	completed an expedited grace period that satisfies the attempted
 	request.  "Our work is done."
 
-o	"n" is number of times that a concurrent CPU-hotplug operation
-	forced a fallback to a normal grace period.
-
 o	"enq" is the number of quiescent states still outstanding.
 
 o	"sc" is the number of times that the attempt to start a
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..8f750dffb0dd 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -521,7 +521,6 @@ struct rcu_state {
 	struct mutex exp_mutex;			/* Serialize expedited GP. */
 	struct mutex exp_wake_mutex;		/* Serialize wakeup. */
 	unsigned long expedited_sequence;	/* Take a ticket. */
-	atomic_long_t expedited_normal;		/* # fallbacks to normal. */
 	atomic_t expedited_need_qs;		/* # CPUs left to check in. */
 	struct swait_queue_head expedited_wq;	/* Wait for check-ins. */
 	int ncpus_snap;				/* # CPUs seen last time. */
diff --git a/kernel/rcu/tree_trace.c b/kernel/rcu/tree_trace.c
index b1f28972872c..2e932cd1da31 100644
--- a/kernel/rcu/tree_trace.c
+++ b/kernel/rcu/tree_trace.c
@@ -194,9 +194,8 @@ static int show_rcuexp(struct seq_file *m, void *v)
 		s2 += atomic_long_read(&rdp->exp_workdone2);
 		s3 += atomic_long_read(&rdp->exp_workdone3);
 	}
-	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu n=%lu enq=%d sc=%lu\n",
+	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu enq=%d sc=%lu\n",
 		   rsp->expedited_sequence, s0, s1, s2, s3,
-		   atomic_long_read(&rsp->expedited_normal),
 		   atomic_read(&rsp->expedited_need_qs),
 		   rsp->expedited_sequence / 2);
 	return 0;
-- 
2.5.2

[PATCH v2 tip/core/rcu 18/18] rcu: Add lockdep checks to synchronous expedited primitives

[Paul E. McKenney]

The non-expedited synchronize_*rcu() primitives have lockdep checks, but
their expedited counterparts lack these checks.  This commit therefore
adds these checks to the expedited synchronize_*rcu() primitives.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_exp.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 303df97bbfc5..f3e214898e3a 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -639,6 +639,11 @@ void synchronize_sched_expedited(void)
 {
 	struct rcu_state *rsp = &rcu_sched_state;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_sched_expedited() in RCU read-side critical section");
+
 	/* If only one CPU, this is automatically a grace period. */
 	if (rcu_blocking_is_gp())
 		return;
@@ -708,6 +713,11 @@ void synchronize_rcu_expedited(void)
 {
 	struct rcu_state *rsp = rcu_state_p;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_rcu_expedited() in RCU read-side critical section");
+
 	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)
 		return;
 	_synchronize_rcu_expedited(rsp, sync_rcu_exp_handler);
-- 
2.5.2

[PATCH v3 tip/core/rcu 0/18] Miscellaneous fixes for 4.11

[Paul E. McKenney]

Hello!

This series provides v3 of miscellaneous fixes:

1.	Make RCU_EXPEDITE_BOOT be the default in order to speed up
	boot, courtesy of Sebastian Andrzej Siewior.

2.	Make RCU suspicious-access lockdep splats use pr_err().

3.	Disable sys_membarrier when nohz_full is enabled, courtesy
	of Mathieu Desnoyers.

4.	Only dump stalled-tasks stacks if there was a real stall, courtesy
	of Byungchul Park.

5.	Remove unneeded rcu_process_callbacks() declarations.

6.	Remove unused but set "mask" variable, courtesy of Tobias Klauser.

7.	Add long-term CPU kicking.

8.	Remove short-term CPU kicking.

9.	Once again use NMI-based stack traces in stall warnings.

10.	Re-enable TASKS_RCU for User Mode Linux.

11.	Don't wake rcuc/X kthreads on NOCB CPUs.

12.	Add comment headers to expedited-grace-period counter functions.

13.	Make rcu_cpu_starting() use its "cpu" argument.

14.	Enable RCU tracepoints by default to aid in debugging, courtesy
	of Matt Fleming.

15.	Fix comment in rcu_organize_nocb_kthreads(), which no longer
	spawns kthreads.

16.	Clarify comments about when llist locking is needed, courtesy
	of Joel Fernandes.

17.	Eliminate unused expedited_normal counter.

18.	Add lockdep checks to synchronous expedited primitives.

Changes since v2:

o	Fixed lockdep message as noted by Josh Triplett.

Changes since v1:

o	Reverse the order of patches 7 and 8.

o	Defer the patch adding tests for trivial grace periods.

o	Defer the patch making cond_resched() provide RCU quiescent
	states.

o	Apply other feedback from Josh Triplett, Peter Zijlstra, and
	Michal Hocko.

							Thanx, Paul

------------------------------------------------------------------------

 Documentation/RCU/trace.txt |    5 +----
 include/linux/llist.h       |   37 +++++++++++++++++++++----------------
 init/Kconfig                |   14 --------------
 kernel/locking/lockdep.c    |   12 ++++++------
 kernel/membarrier.c         |    4 ++++
 kernel/rcu/tiny.c           |    2 --
 kernel/rcu/tree.c           |   35 ++++++++++++++++++-----------------
 kernel/rcu/tree.h           |    1 -
 kernel/rcu/tree_exp.h       |   28 +++++++++++++++++++++++++++-
 kernel/rcu/tree_plugin.h    |    5 +++--
 kernel/rcu/tree_trace.c     |    3 +--
 kernel/rcu/update.c         |    6 ++----
 lib/Kconfig.debug           |    1 +
 13 files changed, 84 insertions(+), 69 deletions(-)

[PATCH v3 tip/core/rcu 01/18] rcu: update: Make RCU_EXPEDITE_BOOT be the default

[Paul E. McKenney]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

RCU_EXPEDITE_BOOT should speed up the boot process by enforcing
synchronize_rcu_expedited() instead of synchronize_rcu() during the boot
process. There should be no reason why one does not want this and there
is no need worry about real time latency at this point.
Therefore make it default.

Note that users wishing to avoid expediting entirely, for example when
bringing up new hardware possibly having flaky IPIs, can use the
rcu_normal boot parameter to override boot-time expediting.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ paulmck: Reworded commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 init/Kconfig        | 13 -------------
 kernel/rcu/update.c |  6 ++----
 2 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 223b734abccd..96e6d56acd50 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -781,19 +781,6 @@ config RCU_NOCB_CPU_ALL
 
 endchoice
 
-config RCU_EXPEDITE_BOOT
-	bool
-	default n
-	help
-	  This option enables expedited grace periods at boot time,
-	  as if rcu_expedite_gp() had been invoked early in boot.
-	  The corresponding rcu_unexpedite_gp() is invoked from
-	  rcu_end_inkernel_boot(), which is intended to be invoked
-	  at the end of the kernel-only boot sequence, just before
-	  init is exec'ed.
-
-	  Accept the default if unsure.
-
 endmenu # "RCU Subsystem"
 
 config BUILD_BIN2C
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 4f6db7e6a117..9e03db9ea9c0 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -132,8 +132,7 @@ bool rcu_gp_is_normal(void)
 }
 EXPORT_SYMBOL_GPL(rcu_gp_is_normal);
 
-static atomic_t rcu_expedited_nesting =
-	ATOMIC_INIT(IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT) ? 1 : 0);
+static atomic_t rcu_expedited_nesting = ATOMIC_INIT(1);
 
 /*
  * Should normal grace-period primitives be expedited?  Intended for
@@ -182,8 +181,7 @@ EXPORT_SYMBOL_GPL(rcu_unexpedite_gp);
  */
 void rcu_end_inkernel_boot(void)
 {
-	if (IS_ENABLED(CONFIG_RCU_EXPEDITE_BOOT))
-		rcu_unexpedite_gp();
+	rcu_unexpedite_gp();
 	if (rcu_normal_after_boot)
 		WRITE_ONCE(rcu_normal, 1);
 }
-- 
2.5.2

[PATCH v3 tip/core/rcu 02/18] lockdep: Make RCU suspicious-access splats use pr_err

[Paul E. McKenney]

This commit switches RCU suspicious-access splats use pr_err()
instead of the current INFO printk()s.  This change makes it easier
to automatically classify splats.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/locking/lockdep.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index 7c38f8f3d97b..d9a698e8458f 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -4412,13 +4412,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
 	printk("\n");
-	printk("===============================\n");
-	printk("[ INFO: suspicious RCU usage. ]\n");
+	pr_err("===============================\n");
+	pr_err("[ ERR: suspicious RCU usage.  ]\n");
 	print_kernel_ident();
-	printk("-------------------------------\n");
-	printk("%s:%d %s!\n", file, line, s);
-	printk("\nother info that might help us debug this:\n\n");
-	printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
+	pr_err("-------------------------------\n");
+	pr_err("%s:%d %s!\n", file, line, s);
+	pr_err("\nother info that might help us debug this:\n\n");
+	pr_err("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
 	       !rcu_lockdep_current_cpu_online()
 			? "RCU used illegally from offline CPU!\n"
 			: !rcu_is_watching()
-- 
2.5.2

[PATCH v3 tip/core/rcu 03/18] Fix: Disable sys_membarrier when nohz_full is enabled

[Paul E. McKenney]

From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Userspace applications should be allowed to expect the membarrier system
call with MEMBARRIER_CMD_SHARED command to issue memory barriers on
nohz_full CPUs, but synchronize_sched() does not take those into
account.

Given that we do not want unrelated processes to be able to affect
real-time sensitive nohz_full CPUs, simply return ENOSYS when membarrier
is invoked on a kernel with enabled nohz_full CPUs.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Josh Triplett <josh@joshtriplett.org>
CC: Steven Rostedt <rostedt@goodmis.org>
CC: <stable@vger.kernel.org>	[3.10+]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Chris Metcalf <cmetcalf@mellanox.com>
Cc: Rik van Riel <riel@redhat.com>
Acked-by: Lai Jiangshan <jiangshanlai@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/membarrier.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/membarrier.c b/kernel/membarrier.c
index 536c727a56e9..9f9284f37f8d 100644
--- a/kernel/membarrier.c
+++ b/kernel/membarrier.c
@@ -16,6 +16,7 @@
 
 #include <linux/syscalls.h>
 #include <linux/membarrier.h>
+#include <linux/tick.h>
 
 /*
  * Bitmask made from a "or" of all commands within enum membarrier_cmd,
@@ -51,6 +52,9 @@
  */
 SYSCALL_DEFINE2(membarrier, int, cmd, int, flags)
 {
+	/* MEMBARRIER_CMD_SHARED is not compatible with nohz_full. */
+	if (tick_nohz_full_enabled())
+		return -ENOSYS;
 	if (unlikely(flags))
 		return -EINVAL;
 	switch (cmd) {
-- 
2.5.2

[PATCH v3 tip/core/rcu 04/18] rcu: Only dump stalled-tasks stacks if there was a real stall

[Paul E. McKenney]

From: Byungchul Park <byungchul.park@lge.com>

The print_other_cpu_stall() function currently unconditionally invokes
rcu_print_detail_task_stall().  This is OK because if there was a stall
sufficient to cause print_other_cpu_stall() to be invoked, that stall
is very likely to persist through the entire print_other_cpu_stall()
execution.  However, if the stall did not persist, the variable ndetected
will be zero, and that variable is already tested in an "if" statement.
Therefore, this commit moves the call to rcu_print_detail_task_stall()
under that pre-existing "if" to improve readability, with a very rare
reduction in overhead.

Signed-off-by: Byungchul Park <byungchul.park@lge.com>
[ paulmck: Reworked commit log. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cb4e2056ccf3..6232d2f9a84e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1379,6 +1379,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 	       (long)rsp->gpnum, (long)rsp->completed, totqlen);
 	if (ndetected) {
 		rcu_dump_cpu_stacks(rsp);
+
+		/* Complain about tasks blocking the grace period. */
+		rcu_print_detail_task_stall(rsp);
 	} else {
 		if (READ_ONCE(rsp->gpnum) != gpnum ||
 		    READ_ONCE(rsp->completed) == gpnum) {
@@ -1395,9 +1398,6 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 		}
 	}
 
-	/* Complain about tasks blocking the grace period. */
-	rcu_print_detail_task_stall(rsp);
-
 	rcu_check_gp_kthread_starvation(rsp);
 
 	panic_on_rcu_stall();
-- 
2.5.2

[PATCH v3 tip/core/rcu 05/18] rcu: Remove unneeded rcu_process_callbacks() declarations

[Paul E. McKenney]

The declarations of __rcu_process_callbacks() and rcu_process_callbacks()
are not needed, as the definition of both of these functions appear before
any uses.  This commit therefore removes both declarations.

Reported-by: "Ahmed, Iftekhar" <ahmedi@oregonstate.edu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tiny.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index b23a4d076f3d..fa6a48d3917b 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -41,8 +41,6 @@
 
 /* Forward declarations for tiny_plugin.h. */
 struct rcu_ctrlblk;
-static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp);
-static void rcu_process_callbacks(struct softirq_action *unused);
 static void __call_rcu(struct rcu_head *head,
 		       rcu_callback_t func,
 		       struct rcu_ctrlblk *rcp);
-- 
2.5.2

[PATCH v3 tip/core/rcu 06/18] rcu: Remove unused but set variable

[Paul E. McKenney]

From: Tobias Klauser <tklauser@distanz.ch>

Since commit 7ec99de36f40 ("rcu: Provide exact CPU-online tracking for
RCU"), the variable mask in rcu_init_percpu_data is set but no longer
used. Remove it to fix the following warning when building with 'W=1':

  kernel/rcu/tree.c: In function ‘rcu_init_percpu_data’:
  kernel/rcu/tree.c:3765:16: warning: variable ‘mask’ set but not used [-Wunused-but-set-variable]

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6232d2f9a84e..83bf054e194e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3765,7 +3765,6 @@ static void
 rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
-	unsigned long mask;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rcu_get_root(rsp);
 
@@ -3788,7 +3787,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 	 * of the next grace period.
 	 */
 	rnp = rdp->mynode;
-	mask = rdp->grpmask;
 	raw_spin_lock_rcu_node(rnp);		/* irqs already disabled. */
 	if (!rdp->beenonline)
 		WRITE_ONCE(rsp->ncpus, READ_ONCE(rsp->ncpus) + 1);
-- 
2.5.2

[PATCH v3 tip/core/rcu 07/18] rcu: Add long-term CPU kicking

[Paul E. McKenney]

This commit prepares for the removal of short-term CPU kicking (in a
subsequent commit).  It does so by starting to invoke resched_cpu()
for each holdout at each force-quiescent-state interval that is more
than halfway through the stall-warning interval.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 83bf054e194e..0e61b62e3f4a 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1225,6 +1225,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
 	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
 		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
+	/*
+	 * If more than halfway to RCU CPU stall-warning time, do
+	 * a resched_cpu() to try to loosen things up a bit.
+	 */
+	if (jiffies - rdp->rsp->gp_start > rcu_jiffies_till_stall_check() / 2)
+		resched_cpu(rdp->cpu);
 
 	return 0;
 }
-- 
2.5.2

[PATCH v3 tip/core/rcu 08/18] rcu: Remove short-term CPU kicking

[Paul E. McKenney]

Commit 4914950aaa12d ("rcu: Stop treating in-kernel CPU-bound workloads
as errors") added a (relatively) short-timeout call to resched_cpu().
This was inspired by as issue that was fixed by b7e7ade34e61 ("sched/core:
Fix remote wakeups").  But given that this issue was fixed, it is time
for the current commit to remove this call to resched_cpu().

Reported-by: Byungchul Park <byungchul.park@lge.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 0e61b62e3f4a..5a4aaad75e76 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1220,11 +1220,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 		rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
 	}
 
-	/* And if it has been a really long time, kick the CPU as well. */
-	if (ULONG_CMP_GE(jiffies,
-			 rdp->rsp->gp_start + 2 * jiffies_till_sched_qs) ||
-	    ULONG_CMP_GE(jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs))
-		resched_cpu(rdp->cpu);  /* Force CPU into scheduler. */
 	/*
 	 * If more than halfway to RCU CPU stall-warning time, do
 	 * a resched_cpu() to try to loosen things up a bit.
-- 
2.5.2

[PATCH v3 tip/core/rcu 09/18] rcu: Once again use NMI-based stack traces in stall warnings

[Paul E. McKenney]

This commit is for all intents and purposes a revert of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks").  The reason to suppose
that this can now safely be reverted is the presence of 42a0bb3f7138
("printk/nmi: generic solution for safe printk in NMI"), which is said
to have made NMI-based stack dumps safe.

However, this reversion keeps one nice property of bc1dce514e9b
("rcu: Don't use NMIs to dump other CPUs' stacks"), namely that
only those CPUs blocking the grace period are dumped.  The new
trigger_single_cpu_backtrace() is used to make this happen, as
suggested by Josh Poimboeuf.

Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 5a4aaad75e76..d7b63b88434b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1278,7 +1278,10 @@ static void rcu_check_gp_kthread_starvation(struct rcu_state *rsp)
 }
 
 /*
- * Dump stacks of all tasks running on stalled CPUs.
+ * Dump stacks of all tasks running on stalled CPUs.  First try using
+ * NMIs, but fall back to manual remote stack tracing on architectures
+ * that don't support NMI-based stack dumps.  The NMI-triggered stack
+ * traces are more accurate because they are printed by the target CPU.
  */
 static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 {
@@ -1288,11 +1291,10 @@ static void rcu_dump_cpu_stacks(struct rcu_state *rsp)
 
 	rcu_for_each_leaf_node(rsp, rnp) {
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-		if (rnp->qsmask != 0) {
-			for_each_leaf_node_possible_cpu(rnp, cpu)
-				if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+		for_each_leaf_node_possible_cpu(rnp, cpu)
+			if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
+				if (!trigger_single_cpu_backtrace(cpu))
 					dump_cpu_task(cpu);
-		}
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	}
 }
-- 
2.5.2

[PATCH v3 tip/core/rcu 10/18] rcu: Re-enable TASKS_RCU for User Mode Linux

[Paul E. McKenney]

Now that User Mode Linux supports arch_irqs_disabled_flags(), this
commit re-enables TASKS_RCU for User Mode Linux.

Reported-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 init/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/init/Kconfig b/init/Kconfig
index 96e6d56acd50..d0caacb4bc14 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -529,7 +529,6 @@ config SRCU
 config TASKS_RCU
 	bool
 	default n
-	depends on !UML
 	select SRCU
 	help
 	  This option enables a task-based RCU implementation that uses
-- 
2.5.2

[PATCH v3 tip/core/rcu 11/18] rcu: Don't wake rcuc/X kthreads on NOCB CPUs

[Paul E. McKenney]

Chris Friesen notice that rcuc/X kthreads were consuming CPU even on
NOCB CPUs.  This makes no sense because the only purpose or these
kthreads is to invoke normal (non-offloaded) callbacks, of which there
will never be any on NOCB CPUs.  This problem was due to a bug in
cpu_has_callbacks_ready_to_invoke(), which should have been checking
->nxttail[RCU_NEXT_TAIL] for NULL, but which was instead (incorrectly)
checking ->nxttail[RCU_DONE_TAIL].  Because ->nxttail[RCU_DONE_TAIL] is
never NULL, the only effect is to cause the rcuc/X kthread to execute
when it should not do so.

This commit therefore checks ->nxttail[RCU_NEXT_TAIL], which is NULL
for NOCB CPUs.

Reported-by: Chris Friesen <chris.friesen@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d7b63b88434b..be2301238a23 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -611,7 +611,7 @@ static int
 cpu_has_callbacks_ready_to_invoke(struct rcu_data *rdp)
 {
 	return &rdp->nxtlist != rdp->nxttail[RCU_DONE_TAIL] &&
-	       rdp->nxttail[RCU_DONE_TAIL] != NULL;
+	       rdp->nxttail[RCU_NEXT_TAIL] != NULL;
 }
 
 /*
-- 
2.5.2

[PATCH v3 tip/core/rcu 12/18] rcu: Add comment headers to expedited-grace-period counter functions

[Paul E. McKenney]

These functions (rcu_exp_gp_seq_start(), rcu_exp_gp_seq_end(),
rcu_exp_gp_seq_snap(), and rcu_exp_gp_seq_done() seemed too obvious
to comment when written, but not so much when being documented.
This commit therefore adds header comments to each of them.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_exp.h | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index e59e1849b89a..303df97bbfc5 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -20,16 +20,26 @@
  * Authors: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
  */
 
-/* Wrapper functions for expedited grace periods.  */
+/*
+ * Record the start of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_start(struct rcu_state *rsp)
 {
 	rcu_seq_start(&rsp->expedited_sequence);
 }
+
+/*
+ * Record the end of an expedited grace period.
+ */
 static void rcu_exp_gp_seq_end(struct rcu_state *rsp)
 {
 	rcu_seq_end(&rsp->expedited_sequence);
 	smp_mb(); /* Ensure that consecutive grace periods serialize. */
 }
+
+/*
+ * Take a snapshot of the expedited-grace-period counter.
+ */
 static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 {
 	unsigned long s;
@@ -39,6 +49,12 @@ static unsigned long rcu_exp_gp_seq_snap(struct rcu_state *rsp)
 	trace_rcu_exp_grace_period(rsp->name, s, TPS("snap"));
 	return s;
 }
+
+/*
+ * Given a counter snapshot from rcu_exp_gp_seq_snap(), return true
+ * if a full expedited grace period has elapsed since that snapshot
+ * was taken.
+ */
 static bool rcu_exp_gp_seq_done(struct rcu_state *rsp, unsigned long s)
 {
 	return rcu_seq_done(&rsp->expedited_sequence, s);
-- 
2.5.2

[PATCH v3 tip/core/rcu 13/18] rcu: Make rcu_cpu_starting() use its "cpu" argument

[Paul E. McKenney]

The rcu_cpu_starting() function uses this_cpu_ptr() to locate the
incoming CPU's rcu_data structure.  This works for the boot CPU and for
all CPUs onlined after rcu_init() executes (during very early boot).
Currently, this is the full set of CPUs, so all is well.  But if
anyone ever parallelizes boot before rcu_init() time, it will fail.
This commit therefore substitutes the rcu_cpu_starting() function's
this_cpu_pointer() for per_cpu_ptr(), future-proofing the code and
(arguably) improving readability.

This commit inadvertently fixes a latent bug: If there ever had been
more than just the boot CPU online at rcu_init() time, the old code
would not initialize the non-boot CPUs, but rather would repeatedly
initialize the boot CPU.

Reported-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index be2301238a23..a4b4762442bb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3873,7 +3873,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	struct rcu_state *rsp;
 
 	for_each_rcu_flavor(rsp) {
-		rdp = this_cpu_ptr(rsp->rda);
+		rdp = per_cpu_ptr(rsp->rda, cpu);
 		rnp = rdp->mynode;
 		mask = rdp->grpmask;
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
-- 
2.5.2

[PATCH v3 tip/core/rcu 14/18] rcu: Enable RCU tracepoints by default to aid in debugging

[Paul E. McKenney]

From: Matt Fleming <matt@codeblueprint.co.uk>

While debugging a performance issue I needed to understand why
RCU sofitrqs were firing so frequently.

Unfortunately, the RCU callback tracepoints are hidden behind
CONFIG_RCU_TRACE which defaults to off in the upstream kernel and is
likely to also be disabled in enterprise distribution configs.

Enable it by default for CONFIG_TREE_RCU. However, we must keep it
disabled for tiny RCU, because it would otherwise pull in a large
amount of code that would make tiny RCU less than tiny.

I ran some file system metadata intensive workloads (git checkout,
FS-Mark) on a variety of machines with this patch and saw no
detectable change in performance.

Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 lib/Kconfig.debug | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index b06848a104e6..5655bd95919a 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1450,6 +1450,7 @@ config RCU_CPU_STALL_TIMEOUT
 config RCU_TRACE
 	bool "Enable tracing for RCU"
 	depends on DEBUG_KERNEL
+	default y if TREE_RCU
 	select TRACE_CLOCK
 	help
 	  This option provides tracing in RCU which presents stats
-- 
2.5.2

[PATCH v3 tip/core/rcu 15/18] rcu: Fix comment in rcu_organize_nocb_kthreads()

[Paul E. McKenney]

It used to be that the rcuo callback-offload kthreads were spawned
in rcu_organize_nocb_kthreads(), and the comment before the "for"
loop says as much.  However, this spawning has long since moved to
the CPU-hotplug code, so this commit fixes this comment.

Reported-by: Michalis Kokologiannakis <mixaskok@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_plugin.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 56583e764ebf..2f5541f6f031 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2366,8 +2366,9 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp)
 	}
 
 	/*
-	 * Each pass through this loop sets up one rcu_data structure and
-	 * spawns one rcu_nocb_kthread().
+	 * Each pass through this loop sets up one rcu_data structure.
+	 * Should the corresponding CPU come online in the future, then
+	 * we will spawn the needed set of rcu_nocb_kthread() kthreads.
 	 */
 	for_each_cpu(cpu, rcu_nocb_mask) {
 		rdp = per_cpu_ptr(rsp->rda, cpu);
-- 
2.5.2

[PATCH v3 tip/core/rcu 16/18] llist: Clarify comments about when locking is needed

[Paul E. McKenney]

From: Joel Fernandes <joelaf@google.com>

llist.h comments are confusing about when locking is needed versus when it
isn't. Clarify these comments by being more descriptive about why locking is
needed for llist_del_first.

Cc: Ingo Molnar <mingo@kernel.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Huang Ying <ying.huang@intel.com>
Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 include/linux/llist.h | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)

diff --git a/include/linux/llist.h b/include/linux/llist.h
index fd4ca0b4fe0f..171baa90f6f6 100644
--- a/include/linux/llist.h
+++ b/include/linux/llist.h
@@ -3,28 +3,33 @@
 /*
  * Lock-less NULL terminated single linked list
  *
- * If there are multiple producers and multiple consumers, llist_add
- * can be used in producers and llist_del_all can be used in
- * consumers.  They can work simultaneously without lock.  But
- * llist_del_first can not be used here.  Because llist_del_first
- * depends on list->first->next does not changed if list->first is not
- * changed during its operation, but llist_del_first, llist_add,
- * llist_add (or llist_del_all, llist_add, llist_add) sequence in
- * another consumer may violate that.
- *
- * If there are multiple producers and one consumer, llist_add can be
- * used in producers and llist_del_all or llist_del_first can be used
- * in the consumer.
- *
- * This can be summarized as follow:
+ * Cases where locking is not needed:
+ * If there are multiple producers and multiple consumers, llist_add can be
+ * used in producers and llist_del_all can be used in consumers simultaneously
+ * without locking. Also a single consumer can use llist_del_first while
+ * multiple producers simultaneously use llist_add, without any locking.
+ *
+ * Cases where locking is needed:
+ * If we have multiple consumers with llist_del_first used in one consumer, and
+ * llist_del_first or llist_del_all used in other consumers, then a lock is
+ * needed.  This is because llist_del_first depends on list->first->next not
+ * changing, but without lock protection, there's no way to be sure about that
+ * if a preemption happens in the middle of the delete operation and on being
+ * preempted back, the list->first is the same as before causing the cmpxchg in
+ * llist_del_first to succeed. For example, while a llist_del_first operation
+ * is in progress in one consumer, then a llist_del_first, llist_add,
+ * llist_add (or llist_del_all, llist_add, llist_add) sequence in another
+ * consumer may cause violations.
+ *
+ * This can be summarized as follows:
  *
  *           |   add    | del_first |  del_all
  * add       |    -     |     -     |     -
  * del_first |          |     L     |     L
  * del_all   |          |           |     -
  *
- * Where "-" stands for no lock is needed, while "L" stands for lock
- * is needed.
+ * Where, a particular row's operation can happen concurrently with a column's
+ * operation, with "-" being no lock needed, while "L" being lock is needed.
  *
  * The list entries deleted via llist_del_all can be traversed with
  * traversing function such as llist_for_each etc.  But the list
-- 
2.5.2

[PATCH v3 tip/core/rcu 17/18] rcu: Eliminate unused expedited_normal counter

[Paul E. McKenney]

Expedited grace periods no longer fall back to normal grace periods
in response to lock contention, given that expedited grace periods
now use the rcu_node tree so as to avoid contention.  This commit
therfore removes the expedited_normal counter.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 Documentation/RCU/trace.txt | 5 +----
 kernel/rcu/tree.h           | 1 -
 kernel/rcu/tree_trace.c     | 3 +--
 3 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/Documentation/RCU/trace.txt b/Documentation/RCU/trace.txt
index 00a3a38b375a..6549012033f9 100644
--- a/Documentation/RCU/trace.txt
+++ b/Documentation/RCU/trace.txt
@@ -237,7 +237,7 @@ o	"ktl" is the low-order 16 bits (in hexadecimal) of the count of
 
 The output of "cat rcu/rcu_preempt/rcuexp" looks as follows:
 
-s=21872 wd1=0 wd2=0 wd3=5 n=0 enq=0 sc=21872
+s=21872 wd1=0 wd2=0 wd3=5 enq=0 sc=21872
 
 These fields are as follows:
 
@@ -249,9 +249,6 @@ o	"wd1", "wd2", and "wd3" are the number of times that an attempt
 	completed an expedited grace period that satisfies the attempted
 	request.  "Our work is done."
 
-o	"n" is number of times that a concurrent CPU-hotplug operation
-	forced a fallback to a normal grace period.
-
 o	"enq" is the number of quiescent states still outstanding.
 
 o	"sc" is the number of times that the attempt to start a
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..8f750dffb0dd 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -521,7 +521,6 @@ struct rcu_state {
 	struct mutex exp_mutex;			/* Serialize expedited GP. */
 	struct mutex exp_wake_mutex;		/* Serialize wakeup. */
 	unsigned long expedited_sequence;	/* Take a ticket. */
-	atomic_long_t expedited_normal;		/* # fallbacks to normal. */
 	atomic_t expedited_need_qs;		/* # CPUs left to check in. */
 	struct swait_queue_head expedited_wq;	/* Wait for check-ins. */
 	int ncpus_snap;				/* # CPUs seen last time. */
diff --git a/kernel/rcu/tree_trace.c b/kernel/rcu/tree_trace.c
index b1f28972872c..2e932cd1da31 100644
--- a/kernel/rcu/tree_trace.c
+++ b/kernel/rcu/tree_trace.c
@@ -194,9 +194,8 @@ static int show_rcuexp(struct seq_file *m, void *v)
 		s2 += atomic_long_read(&rdp->exp_workdone2);
 		s3 += atomic_long_read(&rdp->exp_workdone3);
 	}
-	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu n=%lu enq=%d sc=%lu\n",
+	seq_printf(m, "s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu enq=%d sc=%lu\n",
 		   rsp->expedited_sequence, s0, s1, s2, s3,
-		   atomic_long_read(&rsp->expedited_normal),
 		   atomic_read(&rsp->expedited_need_qs),
 		   rsp->expedited_sequence / 2);
 	return 0;
-- 
2.5.2

[PATCH v3 tip/core/rcu 18/18] rcu: Add lockdep checks to synchronous expedited primitives

[Paul E. McKenney]

The non-expedited synchronize_*rcu() primitives have lockdep checks, but
their expedited counterparts lack these checks.  This commit therefore
adds these checks to the expedited synchronize_*rcu() primitives.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/rcu/tree_exp.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 303df97bbfc5..f3e214898e3a 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -639,6 +639,11 @@ void synchronize_sched_expedited(void)
 {
 	struct rcu_state *rsp = &rcu_sched_state;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_sched_expedited() in RCU read-side critical section");
+
 	/* If only one CPU, this is automatically a grace period. */
 	if (rcu_blocking_is_gp())
 		return;
@@ -708,6 +713,11 @@ void synchronize_rcu_expedited(void)
 {
 	struct rcu_state *rsp = rcu_state_p;
 
+	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
+			 lock_is_held(&rcu_lock_map) ||
+			 lock_is_held(&rcu_sched_lock_map),
+			 "Illegal synchronize_rcu_expedited() in RCU read-side critical section");
+
 	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)
 		return;
 	_synchronize_rcu_expedited(rsp, sync_rcu_exp_handler);
-- 
2.5.2