[PATCH tip/core/rcu 0/14] Fixups for 3.6

[PATCH tip/core/rcu 0/14] Fixups for 3.6

[Paul E. McKenney]

Hello!

This patch series has general fixups and improvements:

1.	Get rid of the false positives from the code that recognizes
	a CPU stall that ends just as it is detected.
2.	Consolidate open-coded initializations of RCU callback lists.
3.	Protect unsynchronized accesses to ->qlen with ACCESS_ONCE()
	to document the unsynchronized access and to prevent compiler
	mischief.
4.	Add INIT_RCU_POINTER() for static initialization to allow
	do-while formulation of the existing RCU_INIT_POINTER().
5.	Convert clever use of RCU_INIT_POINTER() for gcc-style
	initialization to INIT_RCU_POINTER().
6.	Wrap RCU_INIT_POINTER() macro in obligatory do-while.
7.	Remove extraneous parentheses from rcu_assign_keypointer()
	to allow rcu_assign_pointer() to be wrapped in do-while.
8.	Wrap rcu_assign_pointer() macro in obligatory do-while.
9.	Consolidate the identical CONFIG_TREE_PREEMPT_RCU and
	CONFIG_TINY_PREEMPT_RCU versions of __rcu_read_lock() and
	__rcu_read_unlock().
10.	Remove the old inline-function version of __kfree_rcu() and
	__is_kfree_rcu_offset().
11.	Make __call_rcu() correctly handle calls from the idle loop.
12.	Make __call_rcu() correctly handle calls from a CPU that is
	in the process of going offline.
13.	Split __call_rcu() into the part that actually deals with
	enqueueing the callback and the part that kicks RCU's
	grace-period machinery.
14.	Make RCU_TINY's rcu_is_cpu_idle() function safe for "make
	randconfig".

							Thanx, Paul

 b/include/linux/init_task.h |    4 -
 b/include/linux/key.h       |    2 
 b/include/linux/rcupdate.h  |    8 +++
 b/kernel/rcupdate.c         |   44 ++++++++++++++++++
 b/kernel/rcutiny.c          |    4 -
 b/kernel/rcutiny_plugin.h   |   47 -------------------
 b/kernel/rcutree.c          |    4 -
 b/kernel/rcutree_plugin.h   |   47 -------------------
 include/linux/rcupdate.h    |   40 +++-------------
 kernel/rcutree.c            |  107 ++++++++++++++++++++++++++++++++------------
 10 files changed, 149 insertions(+), 158 deletions(-)

[PATCH tip/core/rcu 01/14] rcu: Fix detection of abruptly-ending stall

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

The code that attempts to identify stalls that end just as we detect
them is broken by both flavors of initialization failure.  This commit
therefore properly initializes and computes the count of the number
of reasons why the RCU grace period is stalled.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 0da7b88..0f04480 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -732,7 +732,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)
 	int cpu;
 	long delta;
 	unsigned long flags;
-	int ndetected;
+	int ndetected = 0;
 	struct rcu_node *rnp = rcu_get_root(rsp);
 
 	/* Only let one CPU complain about others per time interval. */
@@ -773,7 +773,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)
 	 */
 	rnp = rcu_get_root(rsp);
 	raw_spin_lock_irqsave(&rnp->lock, flags);
-	ndetected = rcu_print_task_stall(rnp);
+	ndetected += rcu_print_task_stall(rnp);
 	raw_spin_unlock_irqrestore(&rnp->lock, flags);
 
 	print_cpu_stall_info_end();
-- 
1.7.8

[PATCH tip/core/rcu 02/14] rcu: Consolidate duplicate callback-list initialization

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

There are a couple of open-coded initializations of the rcu_data
structure's RCU callback list.  This commit therefore consolidates
them into a new init_callback_list() function.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |   23 ++++++++++++++---------
 1 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 0f04480..d938671 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -936,6 +936,18 @@ check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
 }
 
 /*
+ * Initialize the specified rcu_data structure's callback list to empty.
+ */
+static void init_callback_list(struct rcu_data *rdp)
+{
+	int i;
+
+	rdp->nxtlist = NULL;
+	for (i = 0; i < RCU_NEXT_SIZE; i++)
+		rdp->nxttail[i] = &rdp->nxtlist;
+}
+
+/*
  * Advance this CPU's callbacks, but only if the current grace period
  * has ended.  This may be called only from the CPU to whom the rdp
  * belongs.  In addition, the corresponding leaf rcu_node structure's
@@ -1327,8 +1339,6 @@ static void
 rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
 			  struct rcu_node *rnp, struct rcu_data *rdp)
 {
-	int i;
-
 	/*
 	 * Orphan the callbacks.  First adjust the counts.  This is safe
 	 * because ->onofflock excludes _rcu_barrier()'s adoption of
@@ -1368,9 +1378,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
 	}
 
 	/* Finally, initialize the rcu_data structure's list to empty.  */
-	rdp->nxtlist = NULL;
-	for (i = 0; i < RCU_NEXT_SIZE; i++)
-		rdp->nxttail[i] = &rdp->nxtlist;
+	init_callback_list(rdp);
 }
 
 /*
@@ -2404,16 +2412,13 @@ static void __init
 rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
-	int i;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rcu_get_root(rsp);
 
 	/* Set up local state, ensuring consistent view of global state. */
 	raw_spin_lock_irqsave(&rnp->lock, flags);
 	rdp->grpmask = 1UL << (cpu - rdp->mynode->grplo);
-	rdp->nxtlist = NULL;
-	for (i = 0; i < RCU_NEXT_SIZE; i++)
-		rdp->nxttail[i] = &rdp->nxtlist;
+	init_callback_list(rdp);
 	rdp->qlen_lazy = 0;
 	rdp->qlen = 0;
 	rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
-- 
1.7.8

[PATCH tip/core/rcu 03/14] rcu: Add ACCESS_ONCE() to ->qlen accesses

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

The _rcu_barrier() function accesses other CPUs' rcu_data structure's
->qlen field without benefit of locking.  This commit therefore adds
the required ACCESS_ONCE() wrappers around accesses and updates that
need it.

ACCESS_ONCE() is not needed when a CPU accesses its own ->qlen, or
in code that cannot run while _rcu_barrier() is sampling ->qlen fields.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index d938671..cdc101e 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1349,7 +1349,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
 		rsp->qlen += rdp->qlen;
 		rdp->n_cbs_orphaned += rdp->qlen;
 		rdp->qlen_lazy = 0;
-		rdp->qlen = 0;
+		ACCESS_ONCE(rdp->qlen) = 0;
 	}
 
 	/*
@@ -1597,7 +1597,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)
 	}
 	smp_mb(); /* List handling before counting for rcu_barrier(). */
 	rdp->qlen_lazy -= count_lazy;
-	rdp->qlen -= count;
+	ACCESS_ONCE(rdp->qlen) -= count;
 	rdp->n_cbs_invoked += count;
 
 	/* Reinstate batch limit if we have worked down the excess. */
@@ -1886,7 +1886,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	rdp = this_cpu_ptr(rsp->rda);
 
 	/* Add the callback to our list. */
-	rdp->qlen++;
+	ACCESS_ONCE(rdp->qlen)++;
 	if (lazy)
 		rdp->qlen_lazy++;
 	else
@@ -2420,7 +2420,7 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)
 	rdp->grpmask = 1UL << (cpu - rdp->mynode->grplo);
 	init_callback_list(rdp);
 	rdp->qlen_lazy = 0;
-	rdp->qlen = 0;
+	ACCESS_ONCE(rdp->qlen) = 0;
 	rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
 	WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_EXIT_IDLE);
 	WARN_ON_ONCE(atomic_read(&rdp->dynticks->dynticks) != 1);
-- 
1.7.8

[PATCH tip/core/rcu 04/14] rcu: Add a gcc-style structure initializer for RCU pointers

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

RCU_INIT_POINTER() returns a value that is never used, and which should
be abolished due to terminal ugliness:

	q = RCU_INIT_POINTER(global_p, p);

However, there are two uses that cannot be handled by a do-while
formulation because they do gcc-style initialization:

	RCU_INIT_POINTER(.real_cred, &init_cred),
	RCU_INIT_POINTER(.cred, &init_cred),

This usage is clever, but not necessarily the nicest approach.  This
commit therefore creates an INIT_RCU_POINTER() macro that is specifically
designed for gcc-style initialization.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 26d1a47..eb92c21 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -906,6 +906,14 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 #define RCU_INIT_POINTER(p, v) \
 		p = (typeof(*v) __force __rcu *)(v)
 
+/**
+ * INIT_RCU_POINTER() - statically initialize an RCU protected pointer
+ *
+ * GCC-style initialization for an RCU-protected pointer in a structure field.
+ */
+#define INIT_RCU_POINTER(p, v) \
+		.p = (typeof(*v) __force __rcu *)(v)
+
 static __always_inline bool __is_kfree_rcu_offset(unsigned long offset)
 {
 	return offset < 4096;
-- 
1.7.8

[PATCH tip/core/rcu 05/14] rcu: Use new INIT_RCU_POINTER for gcc-style initializations

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

This commit applies the INIT_RCU_POINTER() macro to all uses of
RCU_INIT_POINTER() that were all too cleverly creating gcc-style
initializations.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/init_task.h |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/init_task.h b/include/linux/init_task.h
index e4baff5..cdaa339 100644
--- a/include/linux/init_task.h
+++ b/include/linux/init_task.h
@@ -168,8 +168,8 @@ extern struct cred init_cred;
 	.children	= LIST_HEAD_INIT(tsk.children),			\
 	.sibling	= LIST_HEAD_INIT(tsk.sibling),			\
 	.group_leader	= &tsk,						\
-	RCU_INIT_POINTER(.real_cred, &init_cred),			\
-	RCU_INIT_POINTER(.cred, &init_cred),				\
+	INIT_RCU_POINTER(real_cred, &init_cred),			\
+	INIT_RCU_POINTER(cred, &init_cred),				\
 	.comm		= INIT_TASK_COMM,				\
 	.thread		= INIT_THREAD,					\
 	.fs		= &init_fs,					\
-- 
1.7.8

[PATCH tip/core/rcu 06/14] rcu: Remove return value from RCU_INIT_POINTER()

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

The return value from RCU_INIT_POINTER() is not used, and using it
would be quite ugly, for example:

	q = RCU_INIT_POINTER(global_p, p);

To prevent this sort of ugliness from appearing, this commit wraps
RCU_INIT_POINTER() in a do-while loop.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index eb92c21..f2cca02 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -904,7 +904,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  * the reader-accessible portions of the linked structure.
  */
 #define RCU_INIT_POINTER(p, v) \
-		p = (typeof(*v) __force __rcu *)(v)
+	do { \
+		p = (typeof(*v) __force __rcu *)(v); \
+	} while (0)
 
 /**
  * INIT_RCU_POINTER() - statically initialize an RCU protected pointer
-- 
1.7.8

[PATCH tip/core/rcu 07/14] key: Remove extraneous parentheses from rcu_assign_keypointer()

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

This commit removes the extraneous parentheses from rcu_assign_keypointer()
so that rcu_assign_pointer() can be wrapped in do-while.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/key.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/include/linux/key.h b/include/linux/key.h
index 4cd22ed..6fafcf4 100644
--- a/include/linux/key.h
+++ b/include/linux/key.h
@@ -303,7 +303,7 @@ static inline bool key_is_instantiated(const struct key *key)
 				   rwsem_is_locked(&((struct key *)(KEY))->sem)))
 
 #define rcu_assign_keypointer(KEY, PAYLOAD)				\
-	(rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD))
+	rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD)
 
 #ifdef CONFIG_SYSCTL
 extern ctl_table key_sysctls[];
-- 
1.7.8

[PATCH tip/core/rcu 08/14] rcu: Remove return value from rcu_assign_pointer()

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

The return value from rcu_assign_pointer() is not used, and using it
would be quite ugly, for example:

	q = rcu_assign_pointer(global_p, p);

To prevent this sort of ugliness from spreading, this commit wraps
rcu_assign_pointer() in a do-while loop.

Reported-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reported-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index f2cca02..2f6ec55 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -514,10 +514,10 @@ static inline void rcu_preempt_sleep_check(void)
 		(_________p1); \
 	})
 #define __rcu_assign_pointer(p, v, space) \
-	({ \
+	do { \
 		smp_wmb(); \
 		(p) = (typeof(*v) __force space *)(v); \
-	})
+	} while (0)
 
 
 /**
@@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  *
  * Assigns the specified value to the specified RCU-protected
  * pointer, ensuring that any concurrent RCU readers will see
- * any prior initialization.  Returns the value assigned.
+ * any prior initialization.
  *
  * Inserts memory barriers on architectures that require them
  * (which is most of them), and also prevents the compiler from
-- 
1.7.8

[PATCH tip/core/rcu 09/14] rcu: Consolidate tree/tiny __rcu_read_{,un}lock() implementations

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

The CONFIG_TREE_PREEMPT_RCU and CONFIG_TINY_PREEMPT_RCU versions of
__rcu_read_unlock() and __rcu_read_unlock() are identical, so this commit
consolidates them into kernel/rcupdate.h.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |    1 +
 kernel/rcupdate.c        |   44 +++++++++++++++++++++++++++++++++++++++++++
 kernel/rcutiny_plugin.h  |   47 +---------------------------------------------
 kernel/rcutree_plugin.h  |   47 +---------------------------------------------
 4 files changed, 47 insertions(+), 92 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 2f6ec55..f773a4a 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -147,6 +147,7 @@ extern void synchronize_sched(void);
 
 extern void __rcu_read_lock(void);
 extern void __rcu_read_unlock(void);
+extern void rcu_read_unlock_special(struct task_struct *t);
 void synchronize_rcu(void);
 
 /*
diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
index 95cba41..4e6a61b 100644
--- a/kernel/rcupdate.c
+++ b/kernel/rcupdate.c
@@ -54,6 +54,50 @@
 #ifdef CONFIG_PREEMPT_RCU
 
 /*
+ * Preemptible RCU implementation for rcu_read_lock().
+ * Just increment ->rcu_read_lock_nesting, shared state will be updated
+ * if we block.
+ */
+void __rcu_read_lock(void)
+{
+	current->rcu_read_lock_nesting++;
+	barrier();  /* critical section after entry code. */
+}
+EXPORT_SYMBOL_GPL(__rcu_read_lock);
+
+/*
+ * Preemptible RCU implementation for rcu_read_unlock().
+ * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
+ * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
+ * invoke rcu_read_unlock_special() to clean up after a context switch
+ * in an RCU read-side critical section and other special cases.
+ */
+void __rcu_read_unlock(void)
+{
+	struct task_struct *t = current;
+
+	if (t->rcu_read_lock_nesting != 1) {
+		--t->rcu_read_lock_nesting;
+	} else {
+		barrier();  /* critical section before exit code. */
+		t->rcu_read_lock_nesting = INT_MIN;
+		barrier();  /* assign before ->rcu_read_unlock_special load */
+		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
+			rcu_read_unlock_special(t);
+		barrier();  /* ->rcu_read_unlock_special load before assign */
+		t->rcu_read_lock_nesting = 0;
+	}
+#ifdef CONFIG_PROVE_LOCKING
+	{
+		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
+
+		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
+	}
+#endif /* #ifdef CONFIG_PROVE_LOCKING */
+}
+EXPORT_SYMBOL_GPL(__rcu_read_unlock);
+
+/*
  * Check for a task exiting while in a preemptible-RCU read-side
  * critical section, clean up if so.  No need to issue warnings,
  * as debug_check_no_locks_held() already does this if lockdep
diff --git a/kernel/rcutiny_plugin.h b/kernel/rcutiny_plugin.h
index fc31a2d..a269b0d 100644
--- a/kernel/rcutiny_plugin.h
+++ b/kernel/rcutiny_plugin.h
@@ -132,7 +132,6 @@ static struct rcu_preempt_ctrlblk rcu_preempt_ctrlblk = {
 	RCU_TRACE(.rcb.name = "rcu_preempt")
 };
 
-static void rcu_read_unlock_special(struct task_struct *t);
 static int rcu_preempted_readers_exp(void);
 static void rcu_report_exp_done(void);
 
@@ -527,23 +526,11 @@ void rcu_preempt_note_context_switch(void)
 }
 
 /*
- * Tiny-preemptible RCU implementation for rcu_read_lock().
- * Just increment ->rcu_read_lock_nesting, shared state will be updated
- * if we block.
- */
-void __rcu_read_lock(void)
-{
-	current->rcu_read_lock_nesting++;
-	barrier();  /* needed if we ever invoke rcu_read_lock in rcutiny.c */
-}
-EXPORT_SYMBOL_GPL(__rcu_read_lock);
-
-/*
  * Handle special cases during rcu_read_unlock(), such as needing to
  * notify RCU core processing or task having blocked during the RCU
  * read-side critical section.
  */
-static noinline void rcu_read_unlock_special(struct task_struct *t)
+void rcu_read_unlock_special(struct task_struct *t)
 {
 	int empty;
 	int empty_exp;
@@ -627,38 +614,6 @@ static noinline void rcu_read_unlock_special(struct task_struct *t)
 }
 
 /*
- * Tiny-preemptible RCU implementation for rcu_read_unlock().
- * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
- * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
- * invoke rcu_read_unlock_special() to clean up after a context switch
- * in an RCU read-side critical section and other special cases.
- */
-void __rcu_read_unlock(void)
-{
-	struct task_struct *t = current;
-
-	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutiny.c */
-	if (t->rcu_read_lock_nesting != 1)
-		--t->rcu_read_lock_nesting;
-	else {
-		t->rcu_read_lock_nesting = INT_MIN;
-		barrier();  /* assign before ->rcu_read_unlock_special load */
-		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
-			rcu_read_unlock_special(t);
-		barrier();  /* ->rcu_read_unlock_special load before assign */
-		t->rcu_read_lock_nesting = 0;
-	}
-#ifdef CONFIG_PROVE_LOCKING
-	{
-		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
-
-		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
-	}
-#endif /* #ifdef CONFIG_PROVE_LOCKING */
-}
-EXPORT_SYMBOL_GPL(__rcu_read_unlock);
-
-/*
  * Check for a quiescent state from the current CPU.  When a task blocks,
  * the task is recorded in the rcu_preempt_ctrlblk structure, which is
  * checked elsewhere.  This is called from the scheduling-clock interrupt.
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 2411000..d806186 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -78,7 +78,6 @@ struct rcu_state rcu_preempt_state = RCU_STATE_INITIALIZER(rcu_preempt);
 DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
 static struct rcu_state *rcu_state = &rcu_preempt_state;
 
-static void rcu_read_unlock_special(struct task_struct *t);
 static int rcu_preempted_readers_exp(struct rcu_node *rnp);
 
 /*
@@ -233,18 +232,6 @@ void rcu_preempt_note_context_switch(void)
 }
 
 /*
- * Tree-preemptible RCU implementation for rcu_read_lock().
- * Just increment ->rcu_read_lock_nesting, shared state will be updated
- * if we block.
- */
-void __rcu_read_lock(void)
-{
-	current->rcu_read_lock_nesting++;
-	barrier();  /* needed if we ever invoke rcu_read_lock in rcutree.c */
-}
-EXPORT_SYMBOL_GPL(__rcu_read_lock);
-
-/*
  * Check for preempted RCU readers blocking the current grace period
  * for the specified rcu_node structure.  If the caller needs a reliable
  * answer, it must hold the rcu_node's ->lock.
@@ -310,7 +297,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
  * notify RCU core processing or task having blocked during the RCU
  * read-side critical section.
  */
-static noinline void rcu_read_unlock_special(struct task_struct *t)
+void rcu_read_unlock_special(struct task_struct *t)
 {
 	int empty;
 	int empty_exp;
@@ -418,38 +405,6 @@ static noinline void rcu_read_unlock_special(struct task_struct *t)
 	}
 }
 
-/*
- * Tree-preemptible RCU implementation for rcu_read_unlock().
- * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
- * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
- * invoke rcu_read_unlock_special() to clean up after a context switch
- * in an RCU read-side critical section and other special cases.
- */
-void __rcu_read_unlock(void)
-{
-	struct task_struct *t = current;
-
-	if (t->rcu_read_lock_nesting != 1)
-		--t->rcu_read_lock_nesting;
-	else {
-		barrier();  /* critical section before exit code. */
-		t->rcu_read_lock_nesting = INT_MIN;
-		barrier();  /* assign before ->rcu_read_unlock_special load */
-		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
-			rcu_read_unlock_special(t);
-		barrier();  /* ->rcu_read_unlock_special load before assign */
-		t->rcu_read_lock_nesting = 0;
-	}
-#ifdef CONFIG_PROVE_LOCKING
-	{
-		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
-
-		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
-	}
-#endif /* #ifdef CONFIG_PROVE_LOCKING */
-}
-EXPORT_SYMBOL_GPL(__rcu_read_unlock);
-
 #ifdef CONFIG_RCU_CPU_STALL_VERBOSE
 
 /*
-- 
1.7.8

[PATCH tip/core/rcu 10/14] rcu: Remove function versions of __kfree_rcu and __is_kfree_rcu_offset

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

Commit d8169d4c (Make __kfree_rcu() less dependent on compiler choices)
added cpp macro versions of __kfree_rcu() and __is_kfree_rcu_offset(),
but failed to remove the old inline-function versions.  This commit does
this cleanup.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |   18 ------------------
 1 files changed, 0 insertions(+), 18 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index f773a4a..4874d26 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -917,24 +917,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 #define INIT_RCU_POINTER(p, v) \
 		.p = (typeof(*v) __force __rcu *)(v)
 
-static __always_inline bool __is_kfree_rcu_offset(unsigned long offset)
-{
-	return offset < 4096;
-}
-
-static __always_inline
-void __kfree_rcu(struct rcu_head *head, unsigned long offset)
-{
-	typedef void (*rcu_callback)(struct rcu_head *);
-
-	BUILD_BUG_ON(!__builtin_constant_p(offset));
-
-	/* See the kfree_rcu() header comment. */
-	BUILD_BUG_ON(!__is_kfree_rcu_offset(offset));
-
-	kfree_call_rcu(head, (rcu_callback)offset);
-}
-
 /*
  * Does the specified offset indicate that the corresponding rcu_head
  * structure can be handled by kfree_rcu()?
-- 
1.7.8

[PATCH tip/core/rcu 11/14] rcu: Make __call_rcu() handle invocation from idle

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

Although __call_rcu() is handled correctly when called from a momentary
non-idle period, if it is called on a CPU that RCU believes to be idle
on RCU_FAST_NO_HZ kernels, the callback might be indefinitely postponed.
This commit therefore ensures that RCU is aware of the new callback and
has a chance to force the CPU out of dyntick-idle mode when a new callback
is posted.

Reported-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |   11 ++---------
 kernel/rcutree.c         |   15 +++++++++------
 2 files changed, 11 insertions(+), 15 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 4874d26..a987595 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -257,6 +257,8 @@ static inline void destroy_rcu_head_on_stack(struct rcu_head *head)
 }
 #endif	/* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
 
+extern int rcu_is_cpu_idle(void);
+
 #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU)
 bool rcu_lockdep_current_cpu_online(void);
 #else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
@@ -268,15 +270,6 @@ static inline bool rcu_lockdep_current_cpu_online(void)
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 
-#ifdef CONFIG_PROVE_RCU
-extern int rcu_is_cpu_idle(void);
-#else /* !CONFIG_PROVE_RCU */
-static inline int rcu_is_cpu_idle(void)
-{
-	return 0;
-}
-#endif /* else !CONFIG_PROVE_RCU */
-
 static inline void rcu_lock_acquire(struct lockdep_map *map)
 {
 	lock_acquire(map, 0, 0, 2, 1, NULL, _THIS_IP_);
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index cdc101e..7720177 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -584,8 +584,6 @@ void rcu_nmi_exit(void)
 	WARN_ON_ONCE(atomic_read(&rdtp->dynticks) & 0x1);
 }
 
-#ifdef CONFIG_PROVE_RCU
-
 /**
  * rcu_is_cpu_idle - see if RCU thinks that the current CPU is idle
  *
@@ -603,7 +601,7 @@ int rcu_is_cpu_idle(void)
 }
 EXPORT_SYMBOL(rcu_is_cpu_idle);
 
-#ifdef CONFIG_HOTPLUG_CPU
+#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU)
 
 /*
  * Is the current CPU online?  Disable preemption to avoid false positives
@@ -644,9 +642,7 @@ bool rcu_lockdep_current_cpu_online(void)
 }
 EXPORT_SYMBOL_GPL(rcu_lockdep_current_cpu_online);
 
-#endif /* #ifdef CONFIG_HOTPLUG_CPU */
-
-#endif /* #ifdef CONFIG_PROVE_RCU */
+#endif /* #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) */
 
 /**
  * rcu_is_cpu_rrupt_from_idle - see if idle or immediately interrupted from idle
@@ -1901,6 +1897,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	else
 		trace_rcu_callback(rsp->name, head, rdp->qlen_lazy, rdp->qlen);
 
+	/*
+	 * If called from an extended quiescent state, invoke the RCU
+	 * core in order to force a re-evaluation of RCU's idleness.
+	 */
+	if (rcu_is_cpu_idle())
+		invoke_rcu_core();
+
 	/* If interrupts were disabled, don't dive into RCU core. */
 	if (irqs_disabled_flags(flags)) {
 		local_irq_restore(flags);
-- 
1.7.8

[PATCH tip/core/rcu 12/14] rcu: Prevent __call_rcu() from invoking RCU core on offline CPUs

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

The __call_rcu() function will invoke the RCU core, for example, if
it detects that the current CPU has too many callbacks.  However, this
can happen on an offline CPU that is on its way to the idle loop, in
which case it is an error to invoke the RCU core, and the excess callbacks
will be adopted in any case.  This commit therefore adds checks to
__call_rcu() for running on an offline CPU, refraining from invoking
the RCU core in this case.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 7720177..9419ebf 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1901,11 +1901,11 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	 * If called from an extended quiescent state, invoke the RCU
 	 * core in order to force a re-evaluation of RCU's idleness.
 	 */
-	if (rcu_is_cpu_idle())
+	if (rcu_is_cpu_idle() && cpu_online(smp_processor_id()))
 		invoke_rcu_core();
 
-	/* If interrupts were disabled, don't dive into RCU core. */
-	if (irqs_disabled_flags(flags)) {
+	/* If interrupts were disabled or CPU offline, don't invoke RCU core. */
+	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id())) {
 		local_irq_restore(flags);
 		return;
 	}
-- 
1.7.8

[PATCH tip/core/rcu 13/14] rcu: Split RCU core processing out of __call_rcu()

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

The __call_rcu() function is a bit overweight, so this commit splits
it into actual enqueuing of and accounting for the callback (__call_rcu())
and associated RCU-core processing (__call_rcu_core()).

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |   55 ++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 files changed, 49 insertions(+), 6 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 9419ebf..6940a81 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1858,9 +1858,11 @@ static void invoke_rcu_core(void)
 	raise_softirq(RCU_SOFTIRQ);
 }
 
-static void
-__call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
-	   struct rcu_state *rsp, bool lazy)
+/*
+ * Handle any core-RCU processing required by a call_rcu() invocation.
+ */
+static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,
+			    struct rcu_head *head, unsigned long flags)
 {
 	unsigned long flags;
 	struct rcu_data *rdp;
@@ -1905,10 +1907,8 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 		invoke_rcu_core();
 
 	/* If interrupts were disabled or CPU offline, don't invoke RCU core. */
-	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id())) {
-		local_irq_restore(flags);
+	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id()))
 		return;
-	}
 
 	/*
 	 * Force the grace period if too many callbacks or too long waiting.
@@ -1941,6 +1941,49 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 		}
 	} else if (ULONG_CMP_LT(ACCESS_ONCE(rsp->jiffies_force_qs), jiffies))
 		force_quiescent_state(rsp, 1);
+}
+
+static void
+__call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
+	   struct rcu_state *rsp, bool lazy)
+{
+	unsigned long flags;
+	struct rcu_data *rdp;
+
+	WARN_ON_ONCE((unsigned long)head & 0x3); /* Misaligned rcu_head! */
+	debug_rcu_head_queue(head);
+	head->func = func;
+	head->next = NULL;
+
+	smp_mb(); /* Ensure RCU update seen before callback registry. */
+
+	/*
+	 * Opportunistically note grace-period endings and beginnings.
+	 * Note that we might see a beginning right after we see an
+	 * end, but never vice versa, since this CPU has to pass through
+	 * a quiescent state betweentimes.
+	 */
+	local_irq_save(flags);
+	rdp = this_cpu_ptr(rsp->rda);
+
+	/* Add the callback to our list. */
+	ACCESS_ONCE(rdp->qlen)++;
+	if (lazy)
+		rdp->qlen_lazy++;
+	else
+		rcu_idle_count_callbacks_posted();
+	smp_mb();  /* Count before adding callback for rcu_barrier(). */
+	*rdp->nxttail[RCU_NEXT_TAIL] = head;
+	rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
+
+	if (__is_kfree_rcu_offset((unsigned long)func))
+		trace_rcu_kfree_callback(rsp->name, head, (unsigned long)func,
+					 rdp->qlen_lazy, rdp->qlen);
+	else
+		trace_rcu_callback(rsp->name, head, rdp->qlen_lazy, rdp->qlen);
+
+	/* Go handle any RCU core processing required. */
+	__call_rcu_core(rsp, rdp, head, flags);
 	local_irq_restore(flags);
 }
 
-- 
1.7.8

[PATCH tip/core/rcu 14/14] rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU

[Paul E. McKenney]

From: "Paul E. McKenney" <paul.mckenney@linaro.org>

The rcu_is_cpu_idle() function is used if CONFIG_DEBUG_LOCK_ALLOC,
but TINY_RCU defines it only when CONFIG_PROVE_RCU.  This causes
build failures when CONFIG_DEBUG_LOCK_ALLOC=y but CONFIG_PROVE_RCU=n.
This commit therefore adjusts the #ifdefs for rcu_is_cpu_idle() so
that it is defined when CONFIG_DEBUG_LOCK_ALLOC=y.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutiny.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/rcutiny.c b/kernel/rcutiny.c
index 37a5444..547b1fe 100644
--- a/kernel/rcutiny.c
+++ b/kernel/rcutiny.c
@@ -172,7 +172,7 @@ void rcu_irq_enter(void)
 	local_irq_restore(flags);
 }
 
-#ifdef CONFIG_PROVE_RCU
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
 
 /*
  * Test whether RCU thinks that the current CPU is idle.
@@ -183,7 +183,7 @@ int rcu_is_cpu_idle(void)
 }
 EXPORT_SYMBOL(rcu_is_cpu_idle);
 
-#endif /* #ifdef CONFIG_PROVE_RCU */
+#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
 
 /*
  * Test whether the current CPU was interrupted from idle.  Nested
-- 
1.7.8

Re: [PATCH tip/core/rcu 01/14] rcu: Fix detection of abruptly-ending stall

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:02PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> The code that attempts to identify stalls that end just as we detect
> them is broken by both flavors of initialization failure.  This commit
> therefore properly initializes and computes the count of the number
> of reasons why the RCU grace period is stalled.
> 
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

This approach definitely seems more robust against future code changes.

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

> ---
>  kernel/rcutree.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 0da7b88..0f04480 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -732,7 +732,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)
>  	int cpu;
>  	long delta;
>  	unsigned long flags;
> -	int ndetected;
> +	int ndetected = 0;
>  	struct rcu_node *rnp = rcu_get_root(rsp);
>  
>  	/* Only let one CPU complain about others per time interval. */
> @@ -773,7 +773,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)
>  	 */
>  	rnp = rcu_get_root(rsp);
>  	raw_spin_lock_irqsave(&rnp->lock, flags);
> -	ndetected = rcu_print_task_stall(rnp);
> +	ndetected += rcu_print_task_stall(rnp);
>  	raw_spin_unlock_irqrestore(&rnp->lock, flags);
>  
>  	print_cpu_stall_info_end();
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 02/14] rcu: Consolidate duplicate callback-list initialization

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:03PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> There are a couple of open-coded initializations of the rcu_data
> structure's RCU callback list.  This commit therefore consolidates
> them into a new init_callback_list() function.
> 
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcutree.c |   23 ++++++++++++++---------
>  1 files changed, 14 insertions(+), 9 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 0f04480..d938671 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -936,6 +936,18 @@ check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
>  }
>  
>  /*
> + * Initialize the specified rcu_data structure's callback list to empty.
> + */
> +static void init_callback_list(struct rcu_data *rdp)
> +{
> +	int i;
> +
> +	rdp->nxtlist = NULL;
> +	for (i = 0; i < RCU_NEXT_SIZE; i++)
> +		rdp->nxttail[i] = &rdp->nxtlist;
> +}
> +
> +/*
>   * Advance this CPU's callbacks, but only if the current grace period
>   * has ended.  This may be called only from the CPU to whom the rdp
>   * belongs.  In addition, the corresponding leaf rcu_node structure's
> @@ -1327,8 +1339,6 @@ static void
>  rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
>  			  struct rcu_node *rnp, struct rcu_data *rdp)
>  {
> -	int i;
> -
>  	/*
>  	 * Orphan the callbacks.  First adjust the counts.  This is safe
>  	 * because ->onofflock excludes _rcu_barrier()'s adoption of
> @@ -1368,9 +1378,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
>  	}
>  
>  	/* Finally, initialize the rcu_data structure's list to empty.  */
> -	rdp->nxtlist = NULL;
> -	for (i = 0; i < RCU_NEXT_SIZE; i++)
> -		rdp->nxttail[i] = &rdp->nxtlist;
> +	init_callback_list(rdp);
>  }
>  
>  /*
> @@ -2404,16 +2412,13 @@ static void __init
>  rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)
>  {
>  	unsigned long flags;
> -	int i;
>  	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
>  	struct rcu_node *rnp = rcu_get_root(rsp);
>  
>  	/* Set up local state, ensuring consistent view of global state. */
>  	raw_spin_lock_irqsave(&rnp->lock, flags);
>  	rdp->grpmask = 1UL << (cpu - rdp->mynode->grplo);
> -	rdp->nxtlist = NULL;
> -	for (i = 0; i < RCU_NEXT_SIZE; i++)
> -		rdp->nxttail[i] = &rdp->nxtlist;
> +	init_callback_list(rdp);
>  	rdp->qlen_lazy = 0;
>  	rdp->qlen = 0;
>  	rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 03/14] rcu: Add ACCESS_ONCE() to ->qlen accesses

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:04PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> The _rcu_barrier() function accesses other CPUs' rcu_data structure's
> ->qlen field without benefit of locking.  This commit therefore adds
> the required ACCESS_ONCE() wrappers around accesses and updates that
> need it.

This type of restriction makes me wonder if we could add some kind of
attribute to fields like qlen to make GCC or sparse help enforce this.

> ACCESS_ONCE() is not needed when a CPU accesses its own ->qlen, or
> in code that cannot run while _rcu_barrier() is sampling ->qlen fields.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcutree.c |    8 ++++----
>  1 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index d938671..cdc101e 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1349,7 +1349,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
>  		rsp->qlen += rdp->qlen;
>  		rdp->n_cbs_orphaned += rdp->qlen;
>  		rdp->qlen_lazy = 0;
> -		rdp->qlen = 0;
> +		ACCESS_ONCE(rdp->qlen) = 0;
>  	}
>  
>  	/*
> @@ -1597,7 +1597,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)
>  	}
>  	smp_mb(); /* List handling before counting for rcu_barrier(). */
>  	rdp->qlen_lazy -= count_lazy;
> -	rdp->qlen -= count;
> +	ACCESS_ONCE(rdp->qlen) -= count;
>  	rdp->n_cbs_invoked += count;
>  
>  	/* Reinstate batch limit if we have worked down the excess. */
> @@ -1886,7 +1886,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
>  	rdp = this_cpu_ptr(rsp->rda);
>  
>  	/* Add the callback to our list. */
> -	rdp->qlen++;
> +	ACCESS_ONCE(rdp->qlen)++;
>  	if (lazy)
>  		rdp->qlen_lazy++;
>  	else
> @@ -2420,7 +2420,7 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)
>  	rdp->grpmask = 1UL << (cpu - rdp->mynode->grplo);
>  	init_callback_list(rdp);
>  	rdp->qlen_lazy = 0;
> -	rdp->qlen = 0;
> +	ACCESS_ONCE(rdp->qlen) = 0;
>  	rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
>  	WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_EXIT_IDLE);
>  	WARN_ON_ONCE(atomic_read(&rdp->dynticks->dynticks) != 1);
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 04/14] rcu: Add a gcc-style structure initializer for RCU pointers

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:05PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> RCU_INIT_POINTER() returns a value that is never used, and which should
> be abolished due to terminal ugliness:
> 
> 	q = RCU_INIT_POINTER(global_p, p);
> 
> However, there are two uses that cannot be handled by a do-while
> formulation because they do gcc-style initialization:
> 
> 	RCU_INIT_POINTER(.real_cred, &init_cred),
> 	RCU_INIT_POINTER(.cred, &init_cred),
> 
> This usage is clever, but not necessarily the nicest approach.  This
> commit therefore creates an INIT_RCU_POINTER() macro that is specifically
> designed for gcc-style initialization.

The concept seems fine, but the names seem entirely non-obvious.
RCU_INIT_POINTER versus INIT_RCU_POINTER?

- Josh Triplett

Re: [PATCH tip/core/rcu 06/14] rcu: Remove return value from RCU_INIT_POINTER()

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:07PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> The return value from RCU_INIT_POINTER() is not used, and using it
> would be quite ugly, for example:
> 
> 	q = RCU_INIT_POINTER(global_p, p);
> 
> To prevent this sort of ugliness from appearing, this commit wraps
> RCU_INIT_POINTER() in a do-while loop.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  include/linux/rcupdate.h |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index eb92c21..f2cca02 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -904,7 +904,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
>   * the reader-accessible portions of the linked structure.
>   */
>  #define RCU_INIT_POINTER(p, v) \
> -		p = (typeof(*v) __force __rcu *)(v)
> +	do { \
> +		p = (typeof(*v) __force __rcu *)(v); \
> +	} while (0)
>  
>  /**
>   * INIT_RCU_POINTER() - statically initialize an RCU protected pointer
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 07/14] key: Remove extraneous parentheses from rcu_assign_keypointer()

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:08PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> This commit removes the extraneous parentheses from rcu_assign_keypointer()
> so that rcu_assign_pointer() can be wrapped in do-while.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  include/linux/key.h |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/include/linux/key.h b/include/linux/key.h
> index 4cd22ed..6fafcf4 100644
> --- a/include/linux/key.h
> +++ b/include/linux/key.h
> @@ -303,7 +303,7 @@ static inline bool key_is_instantiated(const struct key *key)
>  				   rwsem_is_locked(&((struct key *)(KEY))->sem)))
>  
>  #define rcu_assign_keypointer(KEY, PAYLOAD)				\
> -	(rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD))
> +	rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD)
>  
>  #ifdef CONFIG_SYSCTL
>  extern ctl_table key_sysctls[];
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 08/14] rcu: Remove return value from rcu_assign_pointer()

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:09PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> The return value from rcu_assign_pointer() is not used, and using it
> would be quite ugly, for example:
> 
> 	q = rcu_assign_pointer(global_p, p);
> 
> To prevent this sort of ugliness from spreading, this commit wraps
> rcu_assign_pointer() in a do-while loop.
> 
> Reported-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Reported-by: Josh Triplett <josh@joshtriplett.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  include/linux/rcupdate.h |    6 +++---
>  1 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index f2cca02..2f6ec55 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -514,10 +514,10 @@ static inline void rcu_preempt_sleep_check(void)
>  		(_________p1); \
>  	})
>  #define __rcu_assign_pointer(p, v, space) \
> -	({ \
> +	do { \
>  		smp_wmb(); \
>  		(p) = (typeof(*v) __force space *)(v); \
> -	})
> +	} while (0)
>  
>  
>  /**
> @@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
>   *
>   * Assigns the specified value to the specified RCU-protected
>   * pointer, ensuring that any concurrent RCU readers will see
> - * any prior initialization.  Returns the value assigned.
> + * any prior initialization.
>   *
>   * Inserts memory barriers on architectures that require them
>   * (which is most of them), and also prevents the compiler from
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 09/14] rcu: Consolidate tree/tiny __rcu_read_{,un}lock() implementations

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:10PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> The CONFIG_TREE_PREEMPT_RCU and CONFIG_TINY_PREEMPT_RCU versions of
> __rcu_read_unlock() and __rcu_read_unlock() are identical, so this commit
> consolidates them into kernel/rcupdate.h.

Copy-paste error in the commit message here; the first __rcu_read_unlock
should say __rcu_read_lock.  Other than that:

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  include/linux/rcupdate.h |    1 +
>  kernel/rcupdate.c        |   44 +++++++++++++++++++++++++++++++++++++++++++
>  kernel/rcutiny_plugin.h  |   47 +---------------------------------------------
>  kernel/rcutree_plugin.h  |   47 +---------------------------------------------
>  4 files changed, 47 insertions(+), 92 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 2f6ec55..f773a4a 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -147,6 +147,7 @@ extern void synchronize_sched(void);
>  
>  extern void __rcu_read_lock(void);
>  extern void __rcu_read_unlock(void);
> +extern void rcu_read_unlock_special(struct task_struct *t);
>  void synchronize_rcu(void);
>  
>  /*
> diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
> index 95cba41..4e6a61b 100644
> --- a/kernel/rcupdate.c
> +++ b/kernel/rcupdate.c
> @@ -54,6 +54,50 @@
>  #ifdef CONFIG_PREEMPT_RCU
>  
>  /*
> + * Preemptible RCU implementation for rcu_read_lock().
> + * Just increment ->rcu_read_lock_nesting, shared state will be updated
> + * if we block.
> + */
> +void __rcu_read_lock(void)
> +{
> +	current->rcu_read_lock_nesting++;
> +	barrier();  /* critical section after entry code. */
> +}
> +EXPORT_SYMBOL_GPL(__rcu_read_lock);
> +
> +/*
> + * Preemptible RCU implementation for rcu_read_unlock().
> + * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
> + * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
> + * invoke rcu_read_unlock_special() to clean up after a context switch
> + * in an RCU read-side critical section and other special cases.
> + */
> +void __rcu_read_unlock(void)
> +{
> +	struct task_struct *t = current;
> +
> +	if (t->rcu_read_lock_nesting != 1) {
> +		--t->rcu_read_lock_nesting;
> +	} else {
> +		barrier();  /* critical section before exit code. */
> +		t->rcu_read_lock_nesting = INT_MIN;
> +		barrier();  /* assign before ->rcu_read_unlock_special load */
> +		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> +			rcu_read_unlock_special(t);
> +		barrier();  /* ->rcu_read_unlock_special load before assign */
> +		t->rcu_read_lock_nesting = 0;
> +	}
> +#ifdef CONFIG_PROVE_LOCKING
> +	{
> +		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
> +
> +		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
> +	}
> +#endif /* #ifdef CONFIG_PROVE_LOCKING */
> +}
> +EXPORT_SYMBOL_GPL(__rcu_read_unlock);
> +
> +/*
>   * Check for a task exiting while in a preemptible-RCU read-side
>   * critical section, clean up if so.  No need to issue warnings,
>   * as debug_check_no_locks_held() already does this if lockdep
> diff --git a/kernel/rcutiny_plugin.h b/kernel/rcutiny_plugin.h
> index fc31a2d..a269b0d 100644
> --- a/kernel/rcutiny_plugin.h
> +++ b/kernel/rcutiny_plugin.h
> @@ -132,7 +132,6 @@ static struct rcu_preempt_ctrlblk rcu_preempt_ctrlblk = {
>  	RCU_TRACE(.rcb.name = "rcu_preempt")
>  };
>  
> -static void rcu_read_unlock_special(struct task_struct *t);
>  static int rcu_preempted_readers_exp(void);
>  static void rcu_report_exp_done(void);
>  
> @@ -527,23 +526,11 @@ void rcu_preempt_note_context_switch(void)
>  }
>  
>  /*
> - * Tiny-preemptible RCU implementation for rcu_read_lock().
> - * Just increment ->rcu_read_lock_nesting, shared state will be updated
> - * if we block.
> - */
> -void __rcu_read_lock(void)
> -{
> -	current->rcu_read_lock_nesting++;
> -	barrier();  /* needed if we ever invoke rcu_read_lock in rcutiny.c */
> -}
> -EXPORT_SYMBOL_GPL(__rcu_read_lock);
> -
> -/*
>   * Handle special cases during rcu_read_unlock(), such as needing to
>   * notify RCU core processing or task having blocked during the RCU
>   * read-side critical section.
>   */
> -static noinline void rcu_read_unlock_special(struct task_struct *t)
> +void rcu_read_unlock_special(struct task_struct *t)
>  {
>  	int empty;
>  	int empty_exp;
> @@ -627,38 +614,6 @@ static noinline void rcu_read_unlock_special(struct task_struct *t)
>  }
>  
>  /*
> - * Tiny-preemptible RCU implementation for rcu_read_unlock().
> - * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
> - * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
> - * invoke rcu_read_unlock_special() to clean up after a context switch
> - * in an RCU read-side critical section and other special cases.
> - */
> -void __rcu_read_unlock(void)
> -{
> -	struct task_struct *t = current;
> -
> -	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutiny.c */
> -	if (t->rcu_read_lock_nesting != 1)
> -		--t->rcu_read_lock_nesting;
> -	else {
> -		t->rcu_read_lock_nesting = INT_MIN;
> -		barrier();  /* assign before ->rcu_read_unlock_special load */
> -		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> -			rcu_read_unlock_special(t);
> -		barrier();  /* ->rcu_read_unlock_special load before assign */
> -		t->rcu_read_lock_nesting = 0;
> -	}
> -#ifdef CONFIG_PROVE_LOCKING
> -	{
> -		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
> -
> -		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
> -	}
> -#endif /* #ifdef CONFIG_PROVE_LOCKING */
> -}
> -EXPORT_SYMBOL_GPL(__rcu_read_unlock);
> -
> -/*
>   * Check for a quiescent state from the current CPU.  When a task blocks,
>   * the task is recorded in the rcu_preempt_ctrlblk structure, which is
>   * checked elsewhere.  This is called from the scheduling-clock interrupt.
> diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> index 2411000..d806186 100644
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -78,7 +78,6 @@ struct rcu_state rcu_preempt_state = RCU_STATE_INITIALIZER(rcu_preempt);
>  DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
>  static struct rcu_state *rcu_state = &rcu_preempt_state;
>  
> -static void rcu_read_unlock_special(struct task_struct *t);
>  static int rcu_preempted_readers_exp(struct rcu_node *rnp);
>  
>  /*
> @@ -233,18 +232,6 @@ void rcu_preempt_note_context_switch(void)
>  }
>  
>  /*
> - * Tree-preemptible RCU implementation for rcu_read_lock().
> - * Just increment ->rcu_read_lock_nesting, shared state will be updated
> - * if we block.
> - */
> -void __rcu_read_lock(void)
> -{
> -	current->rcu_read_lock_nesting++;
> -	barrier();  /* needed if we ever invoke rcu_read_lock in rcutree.c */
> -}
> -EXPORT_SYMBOL_GPL(__rcu_read_lock);
> -
> -/*
>   * Check for preempted RCU readers blocking the current grace period
>   * for the specified rcu_node structure.  If the caller needs a reliable
>   * answer, it must hold the rcu_node's ->lock.
> @@ -310,7 +297,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
>   * notify RCU core processing or task having blocked during the RCU
>   * read-side critical section.
>   */
> -static noinline void rcu_read_unlock_special(struct task_struct *t)
> +void rcu_read_unlock_special(struct task_struct *t)
>  {
>  	int empty;
>  	int empty_exp;
> @@ -418,38 +405,6 @@ static noinline void rcu_read_unlock_special(struct task_struct *t)
>  	}
>  }
>  
> -/*
> - * Tree-preemptible RCU implementation for rcu_read_unlock().
> - * Decrement ->rcu_read_lock_nesting.  If the result is zero (outermost
> - * rcu_read_unlock()) and ->rcu_read_unlock_special is non-zero, then
> - * invoke rcu_read_unlock_special() to clean up after a context switch
> - * in an RCU read-side critical section and other special cases.
> - */
> -void __rcu_read_unlock(void)
> -{
> -	struct task_struct *t = current;
> -
> -	if (t->rcu_read_lock_nesting != 1)
> -		--t->rcu_read_lock_nesting;
> -	else {
> -		barrier();  /* critical section before exit code. */
> -		t->rcu_read_lock_nesting = INT_MIN;
> -		barrier();  /* assign before ->rcu_read_unlock_special load */
> -		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> -			rcu_read_unlock_special(t);
> -		barrier();  /* ->rcu_read_unlock_special load before assign */
> -		t->rcu_read_lock_nesting = 0;
> -	}
> -#ifdef CONFIG_PROVE_LOCKING
> -	{
> -		int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
> -
> -		WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
> -	}
> -#endif /* #ifdef CONFIG_PROVE_LOCKING */
> -}
> -EXPORT_SYMBOL_GPL(__rcu_read_unlock);
> -
>  #ifdef CONFIG_RCU_CPU_STALL_VERBOSE
>  
>  /*
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 10/14] rcu: Remove function versions of __kfree_rcu and __is_kfree_rcu_offset

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:11PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> Commit d8169d4c (Make __kfree_rcu() less dependent on compiler choices)
> added cpp macro versions of __kfree_rcu() and __is_kfree_rcu_offset(),
> but failed to remove the old inline-function versions.  This commit does
> this cleanup.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  include/linux/rcupdate.h |   18 ------------------
>  1 files changed, 0 insertions(+), 18 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index f773a4a..4874d26 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -917,24 +917,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
>  #define INIT_RCU_POINTER(p, v) \
>  		.p = (typeof(*v) __force __rcu *)(v)
>  
> -static __always_inline bool __is_kfree_rcu_offset(unsigned long offset)
> -{
> -	return offset < 4096;
> -}
> -
> -static __always_inline
> -void __kfree_rcu(struct rcu_head *head, unsigned long offset)
> -{
> -	typedef void (*rcu_callback)(struct rcu_head *);
> -
> -	BUILD_BUG_ON(!__builtin_constant_p(offset));
> -
> -	/* See the kfree_rcu() header comment. */
> -	BUILD_BUG_ON(!__is_kfree_rcu_offset(offset));
> -
> -	kfree_call_rcu(head, (rcu_callback)offset);
> -}
> -
>  /*
>   * Does the specified offset indicate that the corresponding rcu_head
>   * structure can be handled by kfree_rcu()?
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 11/14] rcu: Make __call_rcu() handle invocation from idle

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:12PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> Although __call_rcu() is handled correctly when called from a momentary
> non-idle period, if it is called on a CPU that RCU believes to be idle
> on RCU_FAST_NO_HZ kernels, the callback might be indefinitely postponed.
> This commit therefore ensures that RCU is aware of the new callback and
> has a chance to force the CPU out of dyntick-idle mode when a new callback
> is posted.
> 
> Reported-by: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  include/linux/rcupdate.h |   11 ++---------
>  kernel/rcutree.c         |   15 +++++++++------
>  2 files changed, 11 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 4874d26..a987595 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -257,6 +257,8 @@ static inline void destroy_rcu_head_on_stack(struct rcu_head *head)
>  }
>  #endif	/* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
>  
> +extern int rcu_is_cpu_idle(void);
> +
>  #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU)
>  bool rcu_lockdep_current_cpu_online(void);
>  #else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
> @@ -268,15 +270,6 @@ static inline bool rcu_lockdep_current_cpu_online(void)
>  
>  #ifdef CONFIG_DEBUG_LOCK_ALLOC
>  
> -#ifdef CONFIG_PROVE_RCU
> -extern int rcu_is_cpu_idle(void);
> -#else /* !CONFIG_PROVE_RCU */
> -static inline int rcu_is_cpu_idle(void)
> -{
> -	return 0;
> -}
> -#endif /* else !CONFIG_PROVE_RCU */
> -
>  static inline void rcu_lock_acquire(struct lockdep_map *map)
>  {
>  	lock_acquire(map, 0, 0, 2, 1, NULL, _THIS_IP_);
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index cdc101e..7720177 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -584,8 +584,6 @@ void rcu_nmi_exit(void)
>  	WARN_ON_ONCE(atomic_read(&rdtp->dynticks) & 0x1);
>  }
>  
> -#ifdef CONFIG_PROVE_RCU
> -
>  /**
>   * rcu_is_cpu_idle - see if RCU thinks that the current CPU is idle
>   *
> @@ -603,7 +601,7 @@ int rcu_is_cpu_idle(void)
>  }
>  EXPORT_SYMBOL(rcu_is_cpu_idle);
>  
> -#ifdef CONFIG_HOTPLUG_CPU
> +#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU)
>  
>  /*
>   * Is the current CPU online?  Disable preemption to avoid false positives
> @@ -644,9 +642,7 @@ bool rcu_lockdep_current_cpu_online(void)
>  }
>  EXPORT_SYMBOL_GPL(rcu_lockdep_current_cpu_online);
>  
> -#endif /* #ifdef CONFIG_HOTPLUG_CPU */
> -
> -#endif /* #ifdef CONFIG_PROVE_RCU */
> +#endif /* #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) */
>  
>  /**
>   * rcu_is_cpu_rrupt_from_idle - see if idle or immediately interrupted from idle
> @@ -1901,6 +1897,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
>  	else
>  		trace_rcu_callback(rsp->name, head, rdp->qlen_lazy, rdp->qlen);
>  
> +	/*
> +	 * If called from an extended quiescent state, invoke the RCU
> +	 * core in order to force a re-evaluation of RCU's idleness.
> +	 */
> +	if (rcu_is_cpu_idle())
> +		invoke_rcu_core();
> +
>  	/* If interrupts were disabled, don't dive into RCU core. */
>  	if (irqs_disabled_flags(flags)) {
>  		local_irq_restore(flags);
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 12/14] rcu: Prevent __call_rcu() from invoking RCU core on offline CPUs

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:13PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> The __call_rcu() function will invoke the RCU core, for example, if
> it detects that the current CPU has too many callbacks.  However, this
> can happen on an offline CPU that is on its way to the idle loop, in
> which case it is an error to invoke the RCU core, and the excess callbacks
> will be adopted in any case.  This commit therefore adds checks to
> __call_rcu() for running on an offline CPU, refraining from invoking
> the RCU core in this case.
> 
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcutree.c |    6 +++---
>  1 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 7720177..9419ebf 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1901,11 +1901,11 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
>  	 * If called from an extended quiescent state, invoke the RCU
>  	 * core in order to force a re-evaluation of RCU's idleness.
>  	 */
> -	if (rcu_is_cpu_idle())
> +	if (rcu_is_cpu_idle() && cpu_online(smp_processor_id()))
>  		invoke_rcu_core();
>  
> -	/* If interrupts were disabled, don't dive into RCU core. */
> -	if (irqs_disabled_flags(flags)) {
> +	/* If interrupts were disabled or CPU offline, don't invoke RCU core. */
> +	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id())) {
>  		local_irq_restore(flags);
>  		return;
>  	}
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 04/14] rcu: Add a gcc-style structure initializer for RCU pointers

[Paul E. McKenney]

On Fri, Jun 15, 2012 at 01:48:14PM -0700, Josh Triplett wrote:
> On Fri, Jun 15, 2012 at 01:13:05PM -0700, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > 
> > RCU_INIT_POINTER() returns a value that is never used, and which should
> > be abolished due to terminal ugliness:
> > 
> > 	q = RCU_INIT_POINTER(global_p, p);
> > 
> > However, there are two uses that cannot be handled by a do-while
> > formulation because they do gcc-style initialization:
> > 
> > 	RCU_INIT_POINTER(.real_cred, &init_cred),
> > 	RCU_INIT_POINTER(.cred, &init_cred),
> > 
> > This usage is clever, but not necessarily the nicest approach.  This
> > commit therefore creates an INIT_RCU_POINTER() macro that is specifically
> > designed for gcc-style initialization.
> 
> The concept seems fine, but the names seem entirely non-obvious.
> RCU_INIT_POINTER versus INIT_RCU_POINTER?

I was following things like INIT_THREAD_INFO(), with the leading "INIT_"
for gcc initialization.

							Thanx, Paul

Re: [PATCH tip/core/rcu 13/14] rcu: Split RCU core processing out of __call_rcu()

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:14PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> The __call_rcu() function is a bit overweight, so this commit splits
> it into actual enqueuing of and accounting for the callback (__call_rcu())
> and associated RCU-core processing (__call_rcu_core()).
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcutree.c |   55 ++++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 files changed, 49 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 9419ebf..6940a81 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1858,9 +1858,11 @@ static void invoke_rcu_core(void)
>  	raise_softirq(RCU_SOFTIRQ);
>  }
>  
> -static void
> -__call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
> -	   struct rcu_state *rsp, bool lazy)
> +/*
> + * Handle any core-RCU processing required by a call_rcu() invocation.
> + */
> +static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,
> +			    struct rcu_head *head, unsigned long flags)
>  {
>  	unsigned long flags;
>  	struct rcu_data *rdp;
> @@ -1905,10 +1907,8 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
>  		invoke_rcu_core();
>  
>  	/* If interrupts were disabled or CPU offline, don't invoke RCU core. */
> -	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id())) {
> -		local_irq_restore(flags);
> +	if (irqs_disabled_flags(flags) || cpu_is_offline(smp_processor_id()))
>  		return;
> -	}
>  
>  	/*
>  	 * Force the grace period if too many callbacks or too long waiting.
> @@ -1941,6 +1941,49 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
>  		}
>  	} else if (ULONG_CMP_LT(ACCESS_ONCE(rsp->jiffies_force_qs), jiffies))
>  		force_quiescent_state(rsp, 1);
> +}
> +
> +static void
> +__call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
> +	   struct rcu_state *rsp, bool lazy)
> +{
> +	unsigned long flags;
> +	struct rcu_data *rdp;
> +
> +	WARN_ON_ONCE((unsigned long)head & 0x3); /* Misaligned rcu_head! */
> +	debug_rcu_head_queue(head);
> +	head->func = func;
> +	head->next = NULL;
> +
> +	smp_mb(); /* Ensure RCU update seen before callback registry. */
> +
> +	/*
> +	 * Opportunistically note grace-period endings and beginnings.
> +	 * Note that we might see a beginning right after we see an
> +	 * end, but never vice versa, since this CPU has to pass through
> +	 * a quiescent state betweentimes.
> +	 */
> +	local_irq_save(flags);
> +	rdp = this_cpu_ptr(rsp->rda);
> +
> +	/* Add the callback to our list. */
> +	ACCESS_ONCE(rdp->qlen)++;
> +	if (lazy)
> +		rdp->qlen_lazy++;
> +	else
> +		rcu_idle_count_callbacks_posted();
> +	smp_mb();  /* Count before adding callback for rcu_barrier(). */
> +	*rdp->nxttail[RCU_NEXT_TAIL] = head;
> +	rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
> +
> +	if (__is_kfree_rcu_offset((unsigned long)func))
> +		trace_rcu_kfree_callback(rsp->name, head, (unsigned long)func,
> +					 rdp->qlen_lazy, rdp->qlen);
> +	else
> +		trace_rcu_callback(rsp->name, head, rdp->qlen_lazy, rdp->qlen);
> +
> +	/* Go handle any RCU core processing required. */
> +	__call_rcu_core(rsp, rdp, head, flags);
>  	local_irq_restore(flags);
>  }
>  
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 14/14] rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:13:15PM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> 
> The rcu_is_cpu_idle() function is used if CONFIG_DEBUG_LOCK_ALLOC,
> but TINY_RCU defines it only when CONFIG_PROVE_RCU.  This causes
> build failures when CONFIG_DEBUG_LOCK_ALLOC=y but CONFIG_PROVE_RCU=n.
> This commit therefore adjusts the #ifdefs for rcu_is_cpu_idle() so
> that it is defined when CONFIG_DEBUG_LOCK_ALLOC=y.
> 
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Given that an earlier patch in this series makes rcupdate.h
unconditionally prototype rcu_is_cpu_idle, shouldn't rcutiny.c define it
unconditionally?  (And do so before the earlier patch in this series
makes it available unconditionally?)

- Josh Triplett

>  kernel/rcutiny.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/rcutiny.c b/kernel/rcutiny.c
> index 37a5444..547b1fe 100644
> --- a/kernel/rcutiny.c
> +++ b/kernel/rcutiny.c
> @@ -172,7 +172,7 @@ void rcu_irq_enter(void)
>  	local_irq_restore(flags);
>  }
>  
> -#ifdef CONFIG_PROVE_RCU
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
>  
>  /*
>   * Test whether RCU thinks that the current CPU is idle.
> @@ -183,7 +183,7 @@ int rcu_is_cpu_idle(void)
>  }
>  EXPORT_SYMBOL(rcu_is_cpu_idle);
>  
> -#endif /* #ifdef CONFIG_PROVE_RCU */
> +#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
>  
>  /*
>   * Test whether the current CPU was interrupted from idle.  Nested
> -- 
> 1.7.8
> 

Re: [PATCH tip/core/rcu 04/14] rcu: Add a gcc-style structure initializer for RCU pointers

[Josh Triplett]

On Fri, Jun 15, 2012 at 01:48:13PM -0700, Josh Triplett wrote:
> On Fri, Jun 15, 2012 at 01:13:05PM -0700, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > 
> > RCU_INIT_POINTER() returns a value that is never used, and which should
> > be abolished due to terminal ugliness:
> > 
> > 	q = RCU_INIT_POINTER(global_p, p);
> > 
> > However, there are two uses that cannot be handled by a do-while
> > formulation because they do gcc-style initialization:
> > 
> > 	RCU_INIT_POINTER(.real_cred, &init_cred),
> > 	RCU_INIT_POINTER(.cred, &init_cred),
> > 
> > This usage is clever, but not necessarily the nicest approach.  This
> > commit therefore creates an INIT_RCU_POINTER() macro that is specifically
> > designed for gcc-style initialization.
> 
> The concept seems fine, but the names seem entirely non-obvious.
> RCU_INIT_POINTER versus INIT_RCU_POINTER?

Suggestion: RCU_POINTER_INITIALIZER, following the example of many other
*_INITIALIZER macros elsewhere in the kernel.

- Josh Triplett

Re: [PATCH tip/core/rcu 03/14] rcu: Add ACCESS_ONCE() to ->qlen accesses

[Paul E. McKenney]

On Fri, Jun 15, 2012 at 01:45:00PM -0700, Josh Triplett wrote:
> On Fri, Jun 15, 2012 at 01:13:04PM -0700, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> > 
> > The _rcu_barrier() function accesses other CPUs' rcu_data structure's
> > ->qlen field without benefit of locking.  This commit therefore adds
> > the required ACCESS_ONCE() wrappers around accesses and updates that
> > need it.
> 
> This type of restriction makes me wonder if we could add some kind of
> attribute to fields like qlen to make GCC or sparse help enforce this.

It is worth thinking about.  Should spark some spirited discussions.  ;-)

> > ACCESS_ONCE() is not needed when a CPU accesses its own ->qlen, or
> > in code that cannot run while _rcu_barrier() is sampling ->qlen fields.
> >
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Thank you!

							Thanx, Paul

> >  kernel/rcutree.c |    8 ++++----
> >  1 files changed, 4 insertions(+), 4 deletions(-)
> > 
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index d938671..cdc101e 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -1349,7 +1349,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
> >  		rsp->qlen += rdp->qlen;
> >  		rdp->n_cbs_orphaned += rdp->qlen;
> >  		rdp->qlen_lazy = 0;
> > -		rdp->qlen = 0;
> > +		ACCESS_ONCE(rdp->qlen) = 0;
> >  	}
> >  
> >  	/*
> > @@ -1597,7 +1597,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)
> >  	}
> >  	smp_mb(); /* List handling before counting for rcu_barrier(). */
> >  	rdp->qlen_lazy -= count_lazy;
> > -	rdp->qlen -= count;
> > +	ACCESS_ONCE(rdp->qlen) -= count;
> >  	rdp->n_cbs_invoked += count;
> >  
> >  	/* Reinstate batch limit if we have worked down the excess. */
> > @@ -1886,7 +1886,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
> >  	rdp = this_cpu_ptr(rsp->rda);
> >  
> >  	/* Add the callback to our list. */
> > -	rdp->qlen++;
> > +	ACCESS_ONCE(rdp->qlen)++;
> >  	if (lazy)
> >  		rdp->qlen_lazy++;
> >  	else
> > @@ -2420,7 +2420,7 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)
> >  	rdp->grpmask = 1UL << (cpu - rdp->mynode->grplo);
> >  	init_callback_list(rdp);
> >  	rdp->qlen_lazy = 0;
> > -	rdp->qlen = 0;
> > +	ACCESS_ONCE(rdp->qlen) = 0;
> >  	rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
> >  	WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_EXIT_IDLE);
> >  	WARN_ON_ONCE(atomic_read(&rdp->dynticks->dynticks) != 1);
> > -- 
> > 1.7.8
> > 
> 

Re: [PATCH tip/core/rcu 04/14] rcu: Add a gcc-style structure initializer for RCU pointers

[Paul E. McKenney]

On Fri, Jun 15, 2012 at 02:50:21PM -0700, Josh Triplett wrote:
> On Fri, Jun 15, 2012 at 01:48:13PM -0700, Josh Triplett wrote:
> > On Fri, Jun 15, 2012 at 01:13:05PM -0700, Paul E. McKenney wrote:
> > > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > > 
> > > RCU_INIT_POINTER() returns a value that is never used, and which should
> > > be abolished due to terminal ugliness:
> > > 
> > > 	q = RCU_INIT_POINTER(global_p, p);
> > > 
> > > However, there are two uses that cannot be handled by a do-while
> > > formulation because they do gcc-style initialization:
> > > 
> > > 	RCU_INIT_POINTER(.real_cred, &init_cred),
> > > 	RCU_INIT_POINTER(.cred, &init_cred),
> > > 
> > > This usage is clever, but not necessarily the nicest approach.  This
> > > commit therefore creates an INIT_RCU_POINTER() macro that is specifically
> > > designed for gcc-style initialization.
> > 
> > The concept seems fine, but the names seem entirely non-obvious.
> > RCU_INIT_POINTER versus INIT_RCU_POINTER?
> 
> Suggestion: RCU_POINTER_INITIALIZER, following the example of many other
> *_INITIALIZER macros elsewhere in the kernel.

Sold!

							Thanx, Paul

Re: [PATCH tip/core/rcu 14/14] rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU

[Paul E. McKenney]

On Fri, Jun 15, 2012 at 02:28:28PM -0700, Josh Triplett wrote:
> On Fri, Jun 15, 2012 at 01:13:15PM -0700, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> > 
> > The rcu_is_cpu_idle() function is used if CONFIG_DEBUG_LOCK_ALLOC,
> > but TINY_RCU defines it only when CONFIG_PROVE_RCU.  This causes
> > build failures when CONFIG_DEBUG_LOCK_ALLOC=y but CONFIG_PROVE_RCU=n.
> > This commit therefore adjusts the #ifdefs for rcu_is_cpu_idle() so
> > that it is defined when CONFIG_DEBUG_LOCK_ALLOC=y.
> > 
> > Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Given that an earlier patch in this series makes rcupdate.h
> unconditionally prototype rcu_is_cpu_idle, shouldn't rcutiny.c define it
> unconditionally?  (And do so before the earlier patch in this series
> makes it available unconditionally?)

Or update that earlier patch so that it keeps rcu_is_cpu_idle() under
#ifdef CONFIG_DEBUG_LOCK_ALLOC.  Either way, good catch!

Hmmm...  I want it under CONFIG_DEBUG_LOCK_ALLOC for TINY_RCU (memory
footprint and all that), but unconditional for TREE_RCU.  That can
be arranged...

						Thanx, Paul

> - Josh Triplett
> 
> >  kernel/rcutiny.c |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/kernel/rcutiny.c b/kernel/rcutiny.c
> > index 37a5444..547b1fe 100644
> > --- a/kernel/rcutiny.c
> > +++ b/kernel/rcutiny.c
> > @@ -172,7 +172,7 @@ void rcu_irq_enter(void)
> >  	local_irq_restore(flags);
> >  }
> >  
> > -#ifdef CONFIG_PROVE_RCU
> > +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> >  
> >  /*
> >   * Test whether RCU thinks that the current CPU is idle.
> > @@ -183,7 +183,7 @@ int rcu_is_cpu_idle(void)
> >  }
> >  EXPORT_SYMBOL(rcu_is_cpu_idle);
> >  
> > -#endif /* #ifdef CONFIG_PROVE_RCU */
> > +#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
> >  
> >  /*
> >   * Test whether the current CPU was interrupted from idle.  Nested
> > -- 
> > 1.7.8
> > 
> 

Re: [PATCH tip/core/rcu 14/14] rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU

[Josh Triplett]

On Fri, Jun 15, 2012 at 03:57:49PM -0700, Paul E. McKenney wrote:
> On Fri, Jun 15, 2012 at 02:28:28PM -0700, Josh Triplett wrote:
> > On Fri, Jun 15, 2012 at 01:13:15PM -0700, Paul E. McKenney wrote:
> > > From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> > > 
> > > The rcu_is_cpu_idle() function is used if CONFIG_DEBUG_LOCK_ALLOC,
> > > but TINY_RCU defines it only when CONFIG_PROVE_RCU.  This causes
> > > build failures when CONFIG_DEBUG_LOCK_ALLOC=y but CONFIG_PROVE_RCU=n.
> > > This commit therefore adjusts the #ifdefs for rcu_is_cpu_idle() so
> > > that it is defined when CONFIG_DEBUG_LOCK_ALLOC=y.
> > > 
> > > Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > 
> > Given that an earlier patch in this series makes rcupdate.h
> > unconditionally prototype rcu_is_cpu_idle, shouldn't rcutiny.c define it
> > unconditionally?  (And do so before the earlier patch in this series
> > makes it available unconditionally?)
> 
> Or update that earlier patch so that it keeps rcu_is_cpu_idle() under
> #ifdef CONFIG_DEBUG_LOCK_ALLOC.  Either way, good catch!
> 
> Hmmm...  I want it under CONFIG_DEBUG_LOCK_ALLOC for TINY_RCU (memory
> footprint and all that), but unconditional for TREE_RCU.  That can
> be arranged...

Seems reasonable to me, as long as the availability of the prototype at
compile time always matches the availability of the function at link
time.

- Josh Triplett