* [PATCH tip/core/rcu 0/3] SRCU updates for 4.11
@ 2017-01-14 9:19 Paul E. McKenney
2017-01-14 9:19 ` [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts Paul E. McKenney
` (3 more replies)
0 siblings, 4 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 9:19 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani
Hello!
This series provides updates to SRCU:
1. This is a rewrite of the algorithm simplifying reader-count
tracking. Algorithm courtesy of Mathieu Desnoyers, implementation
courtesy of Lance Roy.
2. Force full grace-period ordering in SRCU.
3. Add CBMC-based formal verification for SRCU, courtesy of Lance Roy.
Thanx, Paul
------------------------------------------------------------------------
include/linux/rcupdate.h | 12
include/linux/srcu.h | 4
kernel/rcu/rcutorture.c | 18
kernel/rcu/srcu.c | 122 +--
kernel/rcu/tree.h | 12
tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h | 155 ++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 ++++++++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 27
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h | 220 +++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h | 92 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c | 78 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c | 72 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh | 102 ++
34 files changed, 1661 insertions(+), 89 deletions(-)
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts.
2017-01-14 9:19 [PATCH tip/core/rcu 0/3] SRCU updates for 4.11 Paul E. McKenney
@ 2017-01-14 9:19 ` Paul E. McKenney
2017-01-14 9:31 ` Ingo Molnar
2017-01-14 9:20 ` [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
` (2 subsequent siblings)
3 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 9:19 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
SRCU uses two per-cpu counters: a nesting counter to count the number of
active critical sections, and a sequence counter to ensure that the nesting
counters don't change while they are being added together in
srcu_readers_active_idx_check().
This patch instead uses per-cpu lock and unlock counters. Because the both
counters only increase and srcu_readers_active_idx_check() reads the unlock
counter before the lock counter, this achieves the same end without having
to increment two different counters in srcu_read_lock(). This also saves a
smp_mb() in srcu_readers_active_idx_check().
A possible problem with this patch is that it can only handle
ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
handle up to ULONG_MAX.
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
include/linux/srcu.h | 4 +-
kernel/rcu/rcutorture.c | 18 +++++++-
kernel/rcu/srcu.c | 117 ++++++++++++++++++------------------------------
3 files changed, 62 insertions(+), 77 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index dc8eb63c6568..0caea34d8c5f 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -34,8 +34,8 @@
#include <linux/workqueue.h>
struct srcu_struct_array {
- unsigned long c[2];
- unsigned long seq[2];
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
};
struct rcu_batch {
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 87c51225ceec..6e4fd7680c70 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -564,10 +564,24 @@ static void srcu_torture_stats(void)
pr_alert("%s%s per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
+ unsigned long l0, l1;
+ unsigned long u0, u1;
long c0, c1;
+ struct srcu_struct_array* counts =
+ per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
- c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
+ u0 = counts->unlock_count[!idx];
+ u1 = counts->unlock_count[idx];
+
+ /* Make sure that a lock is always counted if the corresponding
+ unlock is counted. */
+ smp_rmb();
+
+ l0 = counts->lock_count[!idx];
+ l1 = counts->lock_count[idx];
+
+ c0 = (long)(l0 - u0);
+ c1 = (long)(l1 - u1);
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 9b9cdd549caa..edfdfadec821 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -141,34 +141,38 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
- * Returns approximate total of the readers' ->seq[] values for the
+ * Returns approximate total of the readers' ->lock_count[] values for the
* rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
+ struct srcu_struct_array* cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ t = READ_ONCE(cpu_counts->lock_count[idx]);
sum += t;
}
return sum;
}
/*
- * Returns approximate number of readers active on the specified rank
- * of the per-CPU ->c[] counters.
+ * Returns approximate total of the readers' ->unlock_count[] values for the
+ * rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
+ struct srcu_struct_array* cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ t = READ_ONCE(cpu_counts->unlock_count[idx]);
sum += t;
}
return sum;
@@ -176,79 +180,43 @@ static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
/*
* Return true if the number of pre-existing readers is determined to
- * be stably zero. An example unstable zero can occur if the call
- * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
- * but due to task migration, sees the corresponding __srcu_read_unlock()
- * decrement. This can happen because srcu_readers_active_idx() takes
- * time to sum the array, and might in fact be interrupted or preempted
- * partway through the summation.
+ * be zero.
*/
static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
{
- unsigned long seq;
+ unsigned long unlocks;
- seq = srcu_readers_seq_idx(sp, idx);
+ unlocks = srcu_readers_unlock_idx(sp, idx);
/*
- * The following smp_mb() A pairs with the smp_mb() B located in
- * __srcu_read_lock(). This pairing ensures that if an
- * __srcu_read_lock() increments its counter after the summation
- * in srcu_readers_active_idx(), then the corresponding SRCU read-side
- * critical section will see any changes made prior to the start
- * of the current SRCU grace period.
+ * Make sure that a lock is always counted if the corresponding unlock
+ * is counted. Needs to be a smp_mb() as the read side may contain a
+ * read from a variable that is written to before the synchronize_srcu()
+ * in the write side. In this case smp_mb()s A and B act like the store
+ * buffering pattern.
*
- * Also, if the above call to srcu_readers_seq_idx() saw the
- * increment of ->seq[], then the call to srcu_readers_active_idx()
- * must see the increment of ->c[].
+ * This smp_mb() also pairs with smp_mb() C to prevent writes after the
+ * synchronize_srcu() from being executed before the grace period ends.
*/
smp_mb(); /* A */
/*
- * Note that srcu_readers_active_idx() can incorrectly return
- * zero even though there is a pre-existing reader throughout.
- * To see this, suppose that task A is in a very long SRCU
- * read-side critical section that started on CPU 0, and that
- * no other reader exists, so that the sum of the counters
- * is equal to one. Then suppose that task B starts executing
- * srcu_readers_active_idx(), summing up to CPU 1, and then that
- * task C starts reading on CPU 0, so that its increment is not
- * summed, but finishes reading on CPU 2, so that its decrement
- * -is- summed. Then when task B completes its sum, it will
- * incorrectly get zero, despite the fact that task A has been
- * in its SRCU read-side critical section the whole time.
- *
- * We therefore do a validation step should srcu_readers_active_idx()
- * return zero.
- */
- if (srcu_readers_active_idx(sp, idx) != 0)
- return false;
-
- /*
- * The remainder of this function is the validation step.
- * The following smp_mb() D pairs with the smp_mb() C in
- * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
- * by srcu_readers_active_idx() above, then any destructive
- * operation performed after the grace period will happen after
- * the corresponding SRCU read-side critical section.
+ * If the locks are the same as the unlocks, then there must of have
+ * been no readers on this index at some time in between. This does not
+ * mean that there are no more readers, as one could have read the
+ * current index but have incremented the lock counter yet.
*
- * Note that there can be at most NR_CPUS worth of readers using
- * the old index, which is not enough to overflow even a 32-bit
- * integer. (Yes, this does mean that systems having more than
- * a billion or so CPUs need to be 64-bit systems.) Therefore,
- * the sum of the ->seq[] counters cannot possibly overflow.
- * Therefore, the only way that the return values of the two
- * calls to srcu_readers_seq_idx() can be equal is if there were
- * no increments of the corresponding rank of ->seq[] counts
- * in the interim. But the missed-increment scenario laid out
- * above includes an increment of the ->seq[] counter by
- * the corresponding __srcu_read_lock(). Therefore, if this
- * scenario occurs, the return values from the two calls to
- * srcu_readers_seq_idx() will differ, and thus the validation
- * step below suffices.
+ * Note that there can be at most NR_CPUS worth of readers using the old
+ * index that haven't incremented ->lock_count[] yet. Therefore, the
+ * sum of the ->lock_count[]s cannot increment enough times to overflow
+ * and end up equal the sum of the ->unlock_count[]s, as long as there
+ * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this does
+ * mean that systems having more than a billion or so CPUs need to be
+ * 64-bit systems.) Therefore, the only way that the return values of
+ * the two calls to srcu_readers_(un)lock_idx() can be equal is if there
+ * are no active readers using this index.
*/
- smp_mb(); /* D */
-
- return srcu_readers_seq_idx(sp, idx) == seq;
+ return srcu_readers_lock_idx(sp, idx) == unlocks;
}
/**
@@ -266,8 +234,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
unsigned long sum = 0;
for_each_possible_cpu(cpu) {
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
+ struct srcu_struct_array* cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ sum += READ_ONCE(cpu_counts->lock_count[0]);
+ sum += READ_ONCE(cpu_counts->lock_count[1]);
+ sum -= READ_ONCE(cpu_counts->unlock_count[0]);
+ sum -= READ_ONCE(cpu_counts->unlock_count[1]);
}
return sum;
}
@@ -298,9 +270,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
int idx;
idx = READ_ONCE(sp->completed) & 0x1;
- __this_cpu_inc(sp->per_cpu_ref->c[idx]);
+ __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
smp_mb(); /* B */ /* Avoid leaking the critical section. */
- __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
return idx;
}
EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -314,7 +285,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
void __srcu_read_unlock(struct srcu_struct *sp, int idx)
{
smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_dec(sp->per_cpu_ref->c[idx]);
+ this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
}
EXPORT_SYMBOL_GPL(__srcu_read_unlock);
@@ -349,7 +320,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
/*
* Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->c[] and ->seq[] arrays. This allows
+ * use the other rank of the ->(un)lock_count[] arrays. This allows
* us to wait for pre-existing readers in a starvation-free manner.
*/
static void srcu_flip(struct srcu_struct *sp)
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 9:19 [PATCH tip/core/rcu 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-14 9:19 ` [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts Paul E. McKenney
@ 2017-01-14 9:20 ` Paul E. McKenney
2017-01-14 9:35 ` Ingo Molnar
2017-01-14 9:20 ` [PATCH tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
3 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 9:20 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Paul E. McKenney
If a process invokes synchronize_srcu(), is delayed just the right amount
of time, and thus does not sleep when waiting for the grace period to
complete, there is no ordering between the end of the grace period and
the code following the synchronize_srcu(). Similarly, there can be a
lack of ordering between the end of the SRCU grace period and callback
invocation.
This commit adds the necessary ordering.
Reported-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcupdate.h | 12 ++++++++++++
kernel/rcu/srcu.c | 5 +++++
kernel/rcu/tree.h | 12 ------------
3 files changed, 17 insertions(+), 12 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 01f71e1d2e94..608d56f908f2 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1161,5 +1161,17 @@ do { \
ftrace_dump(oops_dump_mode); \
} while (0)
+/*
+ * Place this after a lock-acquisition primitive to guarantee that
+ * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
+ * if the UNLOCK and LOCK are executed by the same CPU or if the
+ * UNLOCK and LOCK operate on the same lock variable.
+ */
+#ifdef CONFIG_PPC
+#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
+#else /* #ifdef CONFIG_PPC */
+#define smp_mb__after_unlock_lock() do { } while (0)
+#endif /* #else #ifdef CONFIG_PPC */
+
#endif /* __LINUX_RCUPDATE_H */
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index edfdfadec821..2c9265035154 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -363,6 +363,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
head->next = NULL;
head->func = func;
spin_lock_irqsave(&sp->queue_lock, flags);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
rcu_batch_queue(&sp->batch_queue, head);
if (!sp->running) {
sp->running = true;
@@ -396,6 +397,7 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
head->next = NULL;
head->func = wakeme_after_rcu;
spin_lock_irq(&sp->queue_lock);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
if (!sp->running) {
/* steal the processing owner */
sp->running = true;
@@ -417,6 +419,8 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
if (!done)
wait_for_completion(&rcu.completion);
+
+ smp_mb(); /* Caller's later accesses after GP. */
}
/**
@@ -591,6 +595,7 @@ static void srcu_invoke_callbacks(struct srcu_struct *sp)
int i;
struct rcu_head *head;
+ smp_mb(); /* Callback accesses after GP. */
for (i = 0; i < SRCU_CALLBACK_BATCH; i++) {
head = rcu_batch_dequeue(&sp->batch_done);
if (!head)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..abcc25bdcb29 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -688,18 +688,6 @@ static inline void rcu_nocb_q_lengths(struct rcu_data *rdp, long *ql, long *qll)
#endif /* #ifdef CONFIG_RCU_TRACE */
/*
- * Place this after a lock-acquisition primitive to guarantee that
- * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
- * if the UNLOCK and LOCK are executed by the same CPU or if the
- * UNLOCK and LOCK operate on the same lock variable.
- */
-#ifdef CONFIG_PPC
-#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
-#else /* #ifdef CONFIG_PPC */
-#define smp_mb__after_unlock_lock() do { } while (0)
-#endif /* #else #ifdef CONFIG_PPC */
-
-/*
* Wrappers for the rcu_node::lock acquire and release.
*
* Because the rcu_nodes form a tree, the tree traversal locking will observe
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU
2017-01-14 9:19 [PATCH tip/core/rcu 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-14 9:19 ` [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts Paul E. McKenney
2017-01-14 9:20 ` [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
@ 2017-01-14 9:20 ` Paul E. McKenney
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
3 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 9:20 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
This commit creates a formal/srcu-cbmc directory containing scripts that
pull SRCU in from the source code, filter it to remove things that CBMC
cannot handle, and run a series of verifications on it. This has a number
of shortcomings:
1. It does not yet hook into the upper-level self-test Makefiles.
2. It tests only a single scenario, store buffering.
3. There is no gcc-based syntax-error prefiltering.
Nevertheless, it does fully verify a piece of SRCU under a moderately
weak memory model (PSO).
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
.../rcutorture/formal/srcu-cbmc/.gitignore | 1 +
.../selftests/rcutorture/formal/srcu-cbmc/Makefile | 16 +
.../formal/srcu-cbmc/empty_includes/linux/delay.h | 0
.../formal/srcu-cbmc/empty_includes/linux/export.h | 0
.../formal/srcu-cbmc/empty_includes/linux/mutex.h | 0
.../formal/srcu-cbmc/empty_includes/linux/percpu.h | 0
.../srcu-cbmc/empty_includes/linux/preempt.h | 0
.../srcu-cbmc/empty_includes/linux/rcupdate.h | 0
.../formal/srcu-cbmc/empty_includes/linux/sched.h | 0
.../formal/srcu-cbmc/empty_includes/linux/smp.h | 0
.../srcu-cbmc/empty_includes/linux/workqueue.h | 0
.../srcu-cbmc/empty_includes/uapi/linux/types.h | 0
.../formal/srcu-cbmc/include/linux/.gitignore | 1 +
.../formal/srcu-cbmc/include/linux/kconfig.h | 1 +
.../formal/srcu-cbmc/include/linux/types.h | 155 +++++++++
.../rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 +++++++++++++++++++++
.../rcutorture/formal/srcu-cbmc/src/assume.h | 16 +
.../rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +++
.../rcutorture/formal/srcu-cbmc/src/bug_on.h | 13 +
.../formal/srcu-cbmc/src/combined_source.c | 13 +
.../rcutorture/formal/srcu-cbmc/src/config.h | 27 ++
.../rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31 ++
.../rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33 ++
.../rcutorture/formal/srcu-cbmc/src/locks.h | 220 ++++++++++++
.../rcutorture/formal/srcu-cbmc/src/misc.c | 11 +
.../rcutorture/formal/srcu-cbmc/src/misc.h | 58 ++++
.../rcutorture/formal/srcu-cbmc/src/percpu.h | 92 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.c | 78 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.h | 58 ++++
.../formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +++
.../rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++++++
.../srcu-cbmc/tests/store_buffering/.gitignore | 1 +
.../srcu-cbmc/tests/store_buffering/Makefile | 11 +
.../tests/store_buffering/assert_end.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force2.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force3.fail | 1 +
.../srcu-cbmc/tests/store_buffering/main.pass | 0
.../formal/srcu-cbmc/tests/store_buffering/test.c | 72 ++++
.../formal/srcu-cbmc/tests/test_script.sh | 102 ++++++
40 files changed, 1582 insertions(+)
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
new file mode 100644
index 000000000000..712a3d41a325
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
@@ -0,0 +1 @@
+srcu.c
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
new file mode 100644
index 000000000000..16b01559fa55
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
@@ -0,0 +1,16 @@
+all: srcu.c store_buffering
+
+LINUX_SOURCE = ../../../../../..
+
+modified_srcu_input = $(LINUX_SOURCE)/include/linux/srcu.h \
+ $(LINUX_SOURCE)/kernel/rcu/srcu.c
+
+modified_srcu_output = include/linux/srcu.h srcu.c
+
+include/linux/srcu.h: srcu.c
+
+srcu.c: modify_srcu.awk Makefile $(modified_srcu_input)
+ awk -f modify_srcu.awk $(modified_srcu_input) $(modified_srcu_output)
+
+store_buffering:
+ @cd tests/store_buffering; make
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
new file mode 100644
index 000000000000..1d016e66980a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
@@ -0,0 +1 @@
+srcu.h
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
new file mode 100644
index 000000000000..f2860dd1b407
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
@@ -0,0 +1 @@
+#include <LINUX_SOURCE/linux/kconfig.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
new file mode 100644
index 000000000000..4a3d538fef12
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
@@ -0,0 +1,155 @@
+/*
+ * This header has been modifies to remove definitions of types that
+ * are defined in standard userspace headers or are problematic for some
+ * other reason.
+ */
+
+#ifndef _LINUX_TYPES_H
+#define _LINUX_TYPES_H
+
+#define __EXPORTED_HEADERS__
+#include <uapi/linux/types.h>
+
+#ifndef __ASSEMBLY__
+
+#define DECLARE_BITMAP(name, bits) \
+ unsigned long name[BITS_TO_LONGS(bits)]
+
+typedef __u32 __kernel_dev_t;
+
+/* bsd */
+typedef unsigned char u_char;
+typedef unsigned short u_short;
+typedef unsigned int u_int;
+typedef unsigned long u_long;
+
+/* sysv */
+typedef unsigned char unchar;
+typedef unsigned short ushort;
+typedef unsigned int uint;
+typedef unsigned long ulong;
+
+#ifndef __BIT_TYPES_DEFINED__
+#define __BIT_TYPES_DEFINED__
+
+typedef __u8 u_int8_t;
+typedef __s8 int8_t;
+typedef __u16 u_int16_t;
+typedef __s16 int16_t;
+typedef __u32 u_int32_t;
+typedef __s32 int32_t;
+
+#endif /* !(__BIT_TYPES_DEFINED__) */
+
+typedef __u8 uint8_t;
+typedef __u16 uint16_t;
+typedef __u32 uint32_t;
+
+/* this is a special 64bit data type that is 8-byte aligned */
+#define aligned_u64 __u64 __attribute__((aligned(8)))
+#define aligned_be64 __be64 __attribute__((aligned(8)))
+#define aligned_le64 __le64 __attribute__((aligned(8)))
+
+/**
+ * The type used for indexing onto a disc or disc partition.
+ *
+ * Linux always considers sectors to be 512 bytes long independently
+ * of the devices real block size.
+ *
+ * blkcnt_t is the type of the inode's block count.
+ */
+#ifdef CONFIG_LBDAF
+typedef u64 sector_t;
+#else
+typedef unsigned long sector_t;
+#endif
+
+/*
+ * The type of an index into the pagecache.
+ */
+#define pgoff_t unsigned long
+
+/*
+ * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * by the DMA API.
+ *
+ * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
+ * bits wide. Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
+ * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
+ * so they don't care about the size of the actual bus addresses.
+ */
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+typedef u64 dma_addr_t;
+#else
+typedef u32 dma_addr_t;
+#endif
+
+#ifdef CONFIG_PHYS_ADDR_T_64BIT
+typedef u64 phys_addr_t;
+#else
+typedef u32 phys_addr_t;
+#endif
+
+typedef phys_addr_t resource_size_t;
+
+/*
+ * This type is the placeholder for a hardware interrupt number. It has to be
+ * big enough to enclose whatever representation is used by a given platform.
+ */
+typedef unsigned long irq_hw_number_t;
+
+typedef struct {
+ int counter;
+} atomic_t;
+
+#ifdef CONFIG_64BIT
+typedef struct {
+ long counter;
+} atomic64_t;
+#endif
+
+struct list_head {
+ struct list_head *next, *prev;
+};
+
+struct hlist_head {
+ struct hlist_node *first;
+};
+
+struct hlist_node {
+ struct hlist_node *next, **pprev;
+};
+
+/**
+ * struct callback_head - callback structure for use with RCU and task_work
+ * @next: next update requests in a list
+ * @func: actual update function to call after the grace period.
+ *
+ * The struct is aligned to size of pointer. On most architectures it happens
+ * naturally due ABI requirements, but some architectures (like CRIS) have
+ * weird ABI and we need to ask it explicitly.
+ *
+ * The alignment is required to guarantee that bits 0 and 1 of @next will be
+ * clear under normal conditions -- as long as we use call_rcu(),
+ * call_rcu_bh(), call_rcu_sched(), or call_srcu() to queue callback.
+ *
+ * This guarantee is important for few reasons:
+ * - future call_rcu_lazy() will make use of lower bits in the pointer;
+ * - the structure shares storage spacer in struct page with @compound_head,
+ * which encode PageTail() in bit 0. The guarantee is needed to avoid
+ * false-positive PageTail().
+ */
+struct callback_head {
+ struct callback_head *next;
+ void (*func)(struct callback_head *head);
+} __attribute__((aligned(sizeof(void *))));
+#define rcu_head callback_head
+
+typedef void (*rcu_callback_t)(struct rcu_head *head);
+typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
+
+/* clocksource cycle base type */
+typedef u64 cycle_t;
+
+#endif /* __ASSEMBLY__ */
+#endif /* _LINUX_TYPES_H */
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
new file mode 100755
index 000000000000..8ff89043d0a9
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
@@ -0,0 +1,375 @@
+#!/bin/awk -f
+
+# Modify SRCU for formal verification. The first argument should be srcu.h and
+# the second should be srcu.c. Outputs modified srcu.h and srcu.c into the
+# current directory.
+
+BEGIN {
+ if (ARGC != 5) {
+ print "Usange: input.h input.c output.h output.c" > "/dev/stderr";
+ exit 1;
+ }
+ h_output = ARGV[3];
+ c_output = ARGV[4];
+ ARGC = 3;
+
+ # Tokenize using FS and not RS as FS supports regular expressions. Each
+ # record is one line of source, except that backslashed lines are
+ # combined. Comments are treated as field separators, as are quotes.
+ quote_regexp="\"([^\\\\\"]|\\\\.)*\"";
+ comment_regexp="\\/\\*([^*]|\\*+[^*/])*\\*\\/|\\/\\/.*(\n|$)";
+ FS="([ \\\\\t\n\v\f;,.=(){}+*/<>&|^-]|\\[|\\]|" comment_regexp "|" quote_regexp ")+";
+
+ inside_srcu_struct = 0;
+ inside_srcu_init_def = 0;
+ srcu_init_param_name = "";
+ in_macro = 0;
+ brace_nesting = 0;
+ paren_nesting = 0;
+
+ # Allow the manipulation of the last field separator after has been
+ # seen.
+ last_fs = "";
+ # Whether the last field separator was intended to be output.
+ last_fs_print = 0;
+
+ # rcu_batches stores the initialization for each instance of struct
+ # rcu_batch
+
+ in_comment = 0;
+
+ outputfile = "";
+}
+
+{
+ prev_outputfile = outputfile;
+ if (FILENAME ~ /\.h$/) {
+ outputfile = h_output;
+ if (FNR != NR) {
+ print "Incorrect file order" > "/dev/stderr";
+ exit 1;
+ }
+ }
+ else
+ outputfile = c_output;
+
+ if (prev_outputfile && outputfile != prev_outputfile) {
+ new_outputfile = outputfile;
+ outputfile = prev_outputfile;
+ update_fieldsep("", 0);
+ outputfile = new_outputfile;
+ }
+}
+
+# Combine the next line into $0.
+function combine_line() {
+ ret = getline next_line;
+ if (ret == 0) {
+ # Don't allow two consecutive getlines at the end of the file
+ if (eof_found) {
+ print "Error: expected more input." > "/dev/stderr";
+ exit 1;
+ } else {
+ eof_found = 1;
+ }
+ } else if (ret == -1) {
+ print "Error reading next line of file" FILENAME > "/dev/stderr";
+ exit 1;
+ }
+ $0 = $0 "\n" next_line;
+}
+
+# Combine backslashed lines and multiline comments.
+function combine_backslashes() {
+ while (/\\$|\/\*([^*]|\*+[^*\/])*\**$/) {
+ combine_line();
+ }
+}
+
+function read_line() {
+ combine_line();
+ combine_backslashes();
+}
+
+# Print out field separators and update variables that depend on them. Only
+# print if p is true. Call with sep="" and p=0 to print out the last field
+# separator.
+function update_fieldsep(sep, p) {
+ # Count braces
+ sep_tmp = sep;
+ gsub(quote_regexp "|" comment_regexp, "", sep_tmp);
+ while (1)
+ {
+ if (sub("[^{}()]*\\{", "", sep_tmp)) {
+ brace_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\}", "", sep_tmp)) {
+ brace_nesting--;
+ if (brace_nesting < 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+ if (sub("[^{}()]*\\(", "", sep_tmp)) {
+ paren_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\)", "", sep_tmp)) {
+ paren_nesting--;
+ if (paren_nesting < 0) {
+ print "Unbalanced parenthesis!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+
+ break;
+ }
+
+ if (last_fs_print)
+ printf("%s", last_fs) > outputfile;
+ last_fs = sep;
+ last_fs_print = p;
+}
+
+# Shifts the fields down by n positions. Calls next if there are no more. If p
+# is true then print out field separators.
+function shift_fields(n, p) {
+ do {
+ if (match($0, FS) > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), p);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ } else {
+ update_fieldsep("", 0);
+ print "" > outputfile;
+ next;
+ }
+ } while (--n > 0);
+}
+
+# Shifts and prints the first n fields.
+function print_fields(n) {
+ do {
+ update_fieldsep("", 0);
+ printf("%s", $1) > outputfile;
+ shift_fields(1, 1);
+ } while (--n > 0);
+}
+
+{
+ combine_backslashes();
+}
+
+# Print leading FS
+{
+ if (match($0, "^(" FS ")+") > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), 1);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ }
+}
+
+# Parse the line.
+{
+ while (NF > 0) {
+ if ($1 == "struct" && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_struct &&
+ brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "srcu_struct" &&
+ $0 ~ "^struct(" FS ")+srcu_struct(" FS ")+\\{") {
+ inside_srcu_struct = 1;
+ print_fields(2);
+ continue;
+ }
+ if (inside_srcu_struct && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_struct = 0;
+ update_fieldsep("", 0);
+ for (name in rcu_batches)
+ print "extern struct rcu_batch " name ";" > outputfile;
+ }
+
+ if (inside_srcu_struct && $1 == "struct" && $2 == "rcu_batch") {
+ # Move rcu_batches outside of the struct.
+ rcu_batches[$3] = "";
+ shift_fields(3, 1);
+ sub(/;[[:space:]]*$/, "", last_fs);
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_init_def &&
+ $1 == "#define" && $2 == "__SRCU_STRUCT_INIT") {
+ inside_srcu_init_def = 1;
+ srcu_init_param_name = $3;
+ in_macro = 1;
+ print_fields(3);
+ continue;
+ }
+ if (inside_srcu_init_def && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_init_def = 0;
+ in_macro = 0;
+ continue;
+ }
+
+ if (inside_srcu_init_def && brace_nesting == 1 &&
+ paren_nesting == 0 && last_fs ~ /\.[[:space:]]*$/ &&
+ $1 ~ /^[[:alnum:]_]+$/) {
+ name = $1;
+ if (name in rcu_batches) {
+ # Remove the dot.
+ sub(/\.[[:space:]]*$/, "", last_fs);
+
+ old_record = $0;
+ do
+ shift_fields(1, 0);
+ while (last_fs !~ /,/ || paren_nesting > 0);
+ end_loc = length(old_record) - length($0);
+ end_loc += index(last_fs, ",") - length(last_fs);
+
+ last_fs = substr(last_fs, index(last_fs, ",") + 1);
+ last_fs_print = 1;
+
+ match(old_record, "^"name"("FS")+=");
+ start_loc = RSTART + RLENGTH;
+
+ len = end_loc - start_loc;
+ initializer = substr(old_record, start_loc, len);
+ gsub(srcu_init_param_name "\\.", "", initializer);
+ rcu_batches[name] = initializer;
+ continue;
+ }
+ }
+
+ # Don't include a nonexistent file
+ if (!in_macro && $1 == "#include" && /^#include[[:space:]]+"rcu\.h"/) {
+ update_fieldsep("", 0);
+ next;
+ }
+
+ # Ignore most preprocessor stuff.
+ if (!in_macro && $1 ~ /#/) {
+ break;
+ }
+
+ if (brace_nesting > 0 && $1 ~ "^[[:alnum:]_]+$" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (brace_nesting > 0 &&
+ $0 ~ "^[[:alnum:]_]+[[:space:]]*(\\.|->)[[:space:]]*[[:alnum:]_]+" &&
+ $2 in rcu_batches) {
+ # Make uses of rcu_batches global. Somewhat unreliable.
+ shift_fields(1, 0);
+ print_fields(1);
+ continue;
+ }
+
+ if ($1 == "static" && NF < 3) {
+ read_line();
+ continue;
+ }
+ if ($1 == "static" && ($2 == "bool" && $3 == "try_check_zero" ||
+ $2 == "void" && $3 == "srcu_flip")) {
+ shift_fields(1, 1);
+ print_fields(2);
+ continue;
+ }
+
+ # Distinguish between read-side and write-side memory barriers.
+ if ($1 == "smp_mb" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (match($0, /^smp_mb[[:space:]();\/*]*[[:alnum:]]/)) {
+ barrier_letter = substr($0, RLENGTH, 1);
+ if (barrier_letter ~ /A|D/)
+ new_barrier_name = "sync_smp_mb";
+ else if (barrier_letter ~ /B|C/)
+ new_barrier_name = "rs_smp_mb";
+ else {
+ print "Unrecognized memory barrier." > "/dev/null";
+ exit 1;
+ }
+
+ shift_fields(1, 1);
+ printf("%s", new_barrier_name) > outputfile;
+ continue;
+ }
+
+ # Skip definition of rcu_synchronize, since it is already
+ # defined in misc.h. Only present in old versions of srcu.
+ if (brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "rcu_synchronize" &&
+ $0 ~ "^struct(" FS ")+rcu_synchronize(" FS ")+\\{") {
+ shift_fields(2, 0);
+ while (brace_nesting) {
+ if (NF < 2)
+ read_line();
+ shift_fields(1, 0);
+ }
+ }
+
+ # Skip definition of wakeme_after_rcu for the same reason
+ if (brace_nesting == 0 && $1 == "static" && $2 == "void" &&
+ $3 == "wakeme_after_rcu") {
+ while (NF < 5)
+ read_line();
+ shift_fields(3, 0);
+ do {
+ while (NF < 3)
+ read_line();
+ shift_fields(1, 0);
+ } while (paren_nesting || brace_nesting);
+ }
+
+ if ($1 ~ /^(unsigned|long)$/ && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ # Give srcu_batches_completed the correct type for old SRCU.
+ if (brace_nesting == 0 && $1 == "long" &&
+ $2 == "srcu_batches_completed") {
+ update_fieldsep("", 0);
+ printf("unsigned ") > outputfile;
+ print_fields(2);
+ continue;
+ }
+ if (brace_nesting == 0 && $1 == "unsigned" && $2 == "long" &&
+ $3 == "srcu_batches_completed") {
+ print_fields(3);
+ continue;
+ }
+
+ # Just print out the input code by default.
+ print_fields(1);
+ }
+ update_fieldsep("", 0);
+ print > outputfile;
+ next;
+}
+
+END {
+ update_fieldsep("", 0);
+
+ if (brace_nesting != 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+
+ # Define the rcu_batches
+ for (name in rcu_batches)
+ print "struct rcu_batch " name " = " rcu_batches[name] ";" > c_output;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
new file mode 100644
index 000000000000..a64955447995
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
@@ -0,0 +1,16 @@
+#ifndef ASSUME_H
+#define ASSUME_H
+
+/* Provide an assumption macro that can be disabled for gcc. */
+#ifdef RUN
+#define assume(x) \
+ do { \
+ /* Evaluate x to suppress warnings. */ \
+ (void) (x); \
+ } while (0)
+
+#else
+#define assume(x) __CPROVER_assume(x)
+#endif
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
new file mode 100644
index 000000000000..6687acc08e6d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
@@ -0,0 +1,41 @@
+#ifndef BARRIERS_H
+#define BARRIERS_H
+
+#define barrier() __asm__ __volatile__("" : : : "memory")
+
+#ifdef RUN
+#define smp_mb() __sync_synchronize()
+#define smp_mb__after_unlock_lock() __sync_synchronize()
+#else
+/*
+ * Copied from CBMC's implementation of __sync_synchronize(), which
+ * seems to be disabled by default.
+ */
+#define smp_mb() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#define smp_mb__after_unlock_lock() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#endif
+
+/*
+ * Allow memory barriers to be disabled in either the read or write side
+ * of SRCU individually.
+ */
+
+#ifndef NO_SYNC_SMP_MB
+#define sync_smp_mb() smp_mb()
+#else
+#define sync_smp_mb() do {} while (0)
+#endif
+
+#ifndef NO_READ_SIDE_SMP_MB
+#define rs_smp_mb() smp_mb()
+#else
+#define rs_smp_mb() do {} while (0)
+#endif
+
+#define ACCESS_ONCE(x) (*(volatile typeof(x) *) &(x))
+#define READ_ONCE(x) ACCESS_ONCE(x)
+#define WRITE_ONCE(x, val) (ACCESS_ONCE(x) = (val))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
new file mode 100644
index 000000000000..2a80e91f78e7
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
@@ -0,0 +1,13 @@
+#ifndef BUG_ON_H
+#define BUG_ON_H
+
+#include <assert.h>
+
+#define BUG() assert(0)
+#define BUG_ON(x) assert(!(x))
+
+/* Does it make sense to treat warnings as errors? */
+#define WARN() BUG()
+#define WARN_ON(x) (BUG_ON(x), false)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
new file mode 100644
index 000000000000..29eb5d2697ed
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
@@ -0,0 +1,13 @@
+#include <config.h>
+
+/* Include all source files. */
+
+#include "include_srcu.c"
+
+#include "preempt.c"
+#include "misc.c"
+
+/* Used by test.c files */
+#include <pthread.h>
+#include <stdlib.h>
+#include <linux/srcu.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
new file mode 100644
index 000000000000..a60038aeea7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
@@ -0,0 +1,27 @@
+/* "Cheater" definitions based on restricted Kconfig choices. */
+
+#undef CONFIG_TINY_RCU
+#undef __CHECKER__
+#undef CONFIG_DEBUG_LOCK_ALLOC
+#undef CONFIG_DEBUG_OBJECTS_RCU_HEAD
+#undef CONFIG_HOTPLUG_CPU
+#undef CONFIG_MODULES
+#undef CONFIG_NO_HZ_FULL_SYSIDLE
+#undef CONFIG_PREEMPT_COUNT
+#undef CONFIG_PREEMPT_RCU
+#undef CONFIG_PROVE_RCU
+#undef CONFIG_RCU_NOCB_CPU
+#undef CONFIG_RCU_NOCB_CPU_ALL
+#undef CONFIG_RCU_STALL_COMMON
+#undef CONFIG_RCU_TRACE
+#undef CONFIG_RCU_USER_QS
+#undef CONFIG_TASKS_RCU
+#define CONFIG_TREE_RCU
+
+#define CONFIG_GENERIC_ATOMIC64
+
+#if NR_CPUS > 1
+#define CONFIG_SMP
+#else
+#undef CONFIG_SMP
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
new file mode 100644
index 000000000000..5ec582a53018
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
@@ -0,0 +1,31 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#define synchronize_srcu(sp) synchronize_srcu_original(sp)
+#endif
+
+#include <srcu.c>
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#undef synchronize_srcu
+
+#include "simple_sync_srcu.c"
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
new file mode 100644
index 000000000000..3aad63917858
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
@@ -0,0 +1,33 @@
+#ifndef INT_TYPEDEFS_H
+#define INT_TYPEDEFS_H
+
+#include <inttypes.h>
+
+typedef int8_t s8;
+typedef uint8_t u8;
+typedef int16_t s16;
+typedef uint16_t u16;
+typedef int32_t s32;
+typedef uint32_t u32;
+typedef int64_t s64;
+typedef uint64_t u64;
+
+typedef int8_t __s8;
+typedef uint8_t __u8;
+typedef int16_t __s16;
+typedef uint16_t __u16;
+typedef int32_t __s32;
+typedef uint32_t __u32;
+typedef int64_t __s64;
+typedef uint64_t __u64;
+
+#define S8_C(x) INT8_C(x)
+#define U8_C(x) UINT8_C(x)
+#define S16_C(x) INT16_C(x)
+#define U16_C(x) UINT16_C(x)
+#define S32_C(x) INT32_C(x)
+#define U32_C(x) UINT32_C(x)
+#define S64_C(x) INT64_C(x)
+#define U64_C(x) UINT64_C(x)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
new file mode 100644
index 000000000000..356004665576
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
@@ -0,0 +1,220 @@
+#ifndef LOCKS_H
+#define LOCKS_H
+
+#include <limits.h>
+#include <pthread.h>
+#include <stdbool.h>
+
+#include "assume.h"
+#include "bug_on.h"
+#include "preempt.h"
+
+int nondet_int(void);
+
+#define __acquire(x)
+#define __acquires(x)
+#define __release(x)
+#define __releases(x)
+
+/* Only use one lock mechanism. Select which one. */
+#ifdef PTHREAD_LOCK
+struct lock_impl {
+ pthread_mutex_t mutex;
+};
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_lock(&lock->mutex));
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_unlock(&lock->mutex));
+}
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+ int err = pthread_mutex_trylock(&lock->mutex);
+
+ if (!err)
+ return true;
+ else if (err == EBUSY)
+ return false;
+ BUG();
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ pthread_mutex_init(&lock->mutex, NULL);
+}
+
+#define LOCK_IMPL_INITIALIZER {.mutex = PTHREAD_MUTEX_INITIALIZER}
+
+#else /* !defined(PTHREAD_LOCK) */
+/* Spinlock that assumes that it always gets the lock immediately. */
+
+struct lock_impl {
+ bool locked;
+};
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+#ifdef RUN
+ /* TODO: Should this be a test and set? */
+ return __sync_bool_compare_and_swap(&lock->locked, false, true);
+#else
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = true;
+ __CPROVER_atomic_end();
+
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RRfence", "RWfence");
+
+ return !old_locked;
+#endif
+}
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ /*
+ * CBMC doesn't support busy waiting, so just assume that the
+ * lock is available.
+ */
+ assume(lock_impl_trylock(lock));
+
+ /*
+ * If the lock was already held by this thread then the assumption
+ * is unsatisfiable (deadlock).
+ */
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+#ifdef RUN
+ BUG_ON(!__sync_bool_compare_and_swap(&lock->locked, true, false));
+#else
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RWfence", "WWfence");
+
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = false;
+ __CPROVER_atomic_end();
+
+ BUG_ON(!old_locked);
+#endif
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ lock->locked = false;
+}
+
+#define LOCK_IMPL_INITIALIZER {.locked = false}
+
+#endif /* !defined(PTHREAD_LOCK) */
+
+/*
+ * Implement spinlocks using the lock mechanism. Wrap the lock to prevent mixing
+ * locks of different types.
+ */
+typedef struct {
+ struct lock_impl internal_lock;
+} spinlock_t;
+
+#define SPIN_LOCK_UNLOCKED {.internal_lock = LOCK_IMPL_INITIALIZER}
+#define __SPIN_LOCK_UNLOCKED(x) SPIN_LOCK_UNLOCKED
+#define DEFINE_SPINLOCK(x) spinlock_t x = SPIN_LOCK_UNLOCKED
+
+static inline void spin_lock_init(spinlock_t *lock)
+{
+ lock_impl_init(&lock->internal_lock);
+}
+
+static inline void spin_lock(spinlock_t *lock)
+{
+ /*
+ * Spin locks also need to be removed in order to eliminate all
+ * memory barriers. They are only used by the write side anyway.
+ */
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ lock_impl_lock(&lock->internal_lock);
+#endif
+}
+
+static inline void spin_unlock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ lock_impl_unlock(&lock->internal_lock);
+ preempt_enable();
+#endif
+}
+
+/* Don't bother with interrupts */
+#define spin_lock_irq(lock) spin_lock(lock)
+#define spin_unlock_irq(lock) spin_unlock(lock)
+#define spin_lock_irqsave(lock, flags) spin_lock(lock)
+#define spin_unlock_irqrestore(lock, flags) spin_unlock(lock)
+
+/*
+ * This is supposed to return an int, but I think that a bool should work as
+ * well.
+ */
+static inline bool spin_trylock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ return lock_impl_trylock(&lock->internal_lock);
+#else
+ return true;
+#endif
+}
+
+struct completion {
+ /* Hopefuly this won't overflow. */
+ unsigned int count;
+};
+
+#define COMPLETION_INITIALIZER(x) {.count = 0}
+#define DECLARE_COMPLETION(x) struct completion x = COMPLETION_INITIALIZER(x)
+#define DECLARE_COMPLETION_ONSTACK(x) DECLARE_COMPLETION(x)
+
+static inline void init_completion(struct completion *c)
+{
+ c->count = 0;
+}
+
+static inline void wait_for_completion(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_sub(&c->count, 1);
+
+ assume(prev_count);
+}
+
+static inline void complete(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_add(&c->count, 1);
+
+ BUG_ON(prev_count == UINT_MAX);
+}
+
+/* This function probably isn't very useful for CBMC. */
+static inline bool try_wait_for_completion(struct completion *c)
+{
+ BUG();
+}
+
+static inline bool completion_done(struct completion *c)
+{
+ return c->count;
+}
+
+/* TODO: Implement complete_all */
+static inline void complete_all(struct completion *c)
+{
+ BUG();
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
new file mode 100644
index 000000000000..ca892e3b2351
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
@@ -0,0 +1,11 @@
+#include <config.h>
+
+#include "misc.h"
+#include "bug_on.h"
+
+struct rcu_head;
+
+void wakeme_after_rcu(struct rcu_head *head)
+{
+ BUG();
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
new file mode 100644
index 000000000000..aca50030f954
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
@@ -0,0 +1,58 @@
+#ifndef MISC_H
+#define MISC_H
+
+#include "assume.h"
+#include "int_typedefs.h"
+#include "locks.h"
+
+#include <linux/types.h>
+
+/* Probably won't need to deal with bottom halves. */
+static inline void local_bh_disable(void) {}
+static inline void local_bh_enable(void) {}
+
+#define MODULE_ALIAS(X)
+#define module_param(...)
+#define EXPORT_SYMBOL_GPL(x)
+
+#define container_of(ptr, type, member) ({ \
+ const typeof(((type *)0)->member) *__mptr = (ptr); \
+ (type *)((char *)__mptr - offsetof(type, member)); \
+})
+
+#ifndef USE_SIMPLE_SYNC_SRCU
+/* Abuse udelay to make sure that busy loops terminate. */
+#define udelay(x) assume(0)
+
+#else
+
+/* The simple custom synchronize_srcu is ok with try_check_zero failing. */
+#define udelay(x) do { } while (0)
+#endif
+
+#define trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
+ do { } while (0)
+
+#define notrace
+
+/* Avoid including rcupdate.h */
+struct rcu_synchronize {
+ struct rcu_head head;
+ struct completion completion;
+};
+
+void wakeme_after_rcu(struct rcu_head *head);
+
+#define rcu_lock_acquire(a) do { } while (0)
+#define rcu_lock_release(a) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
+#define RCU_LOCKDEP_WARN(c, s) do { } while (0)
+
+/* Let CBMC non-deterministically choose switch between normal and expedited. */
+bool rcu_gp_is_normal(void);
+bool rcu_gp_is_expedited(void);
+
+/* Do the same for old versions of rcu. */
+#define rcu_expedited (rcu_gp_is_expedited())
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
new file mode 100644
index 000000000000..3de5a49de49b
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
@@ -0,0 +1,92 @@
+#ifndef PERCPU_H
+#define PERCPU_H
+
+#include <stddef.h>
+#include "bug_on.h"
+#include "preempt.h"
+
+#define __percpu
+
+/* Maximum size of any percpu data. */
+#define PERCPU_OFFSET (4 * sizeof(long))
+
+/* Ignore alignment, as CBMC doesn't care about false sharing. */
+#define alloc_percpu(type) __alloc_percpu(sizeof(type), 1)
+
+static inline void *__alloc_percpu(size_t size, size_t align)
+{
+ BUG();
+ return NULL;
+}
+
+static inline void free_percpu(void *ptr)
+{
+ BUG();
+}
+
+#define per_cpu_ptr(ptr, cpu) \
+ ((typeof(ptr)) ((char *) (ptr) + PERCPU_OFFSET * cpu))
+
+#define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1)
+#define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1)
+#define __this_cpu_sub(pcp, n) __this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+#define this_cpu_inc(pcp) this_cpu_add(pcp, 1)
+#define this_cpu_dec(pcp) this_cpu_sub(pcp, 1)
+#define this_cpu_sub(pcp, n) this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+/* Make CBMC use atomics to work around bug. */
+#ifdef RUN
+#define THIS_CPU_ADD_HELPER(ptr, x) (*(ptr) += (x))
+#else
+/*
+ * Split the atomic into a read and a write so that it has the least
+ * possible ordering.
+ */
+#define THIS_CPU_ADD_HELPER(ptr, x) \
+ do { \
+ typeof(ptr) this_cpu_add_helper_ptr = (ptr); \
+ typeof(ptr) this_cpu_add_helper_x = (x); \
+ typeof(*ptr) this_cpu_add_helper_temp; \
+ __CPROVER_atomic_begin(); \
+ this_cpu_add_helper_temp = *(this_cpu_add_helper_ptr); \
+ __CPROVER_atomic_end(); \
+ this_cpu_add_helper_temp += this_cpu_add_helper_x; \
+ __CPROVER_atomic_begin(); \
+ *(this_cpu_add_helper_ptr) = this_cpu_add_helper_temp; \
+ __CPROVER_atomic_end(); \
+ } while (0)
+#endif
+
+/*
+ * For some reason CBMC needs an atomic operation even though this is percpu
+ * data.
+ */
+#define __this_cpu_add(pcp, n) \
+ do { \
+ BUG_ON(preemptible()); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), thread_cpu_id), \
+ (typeof(pcp)) (n)); \
+ } while (0)
+
+#define this_cpu_add(pcp, n) \
+ do { \
+ int this_cpu_add_impl_cpu = get_cpu(); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), this_cpu_add_impl_cpu), \
+ (typeof(pcp)) (n)); \
+ put_cpu(); \
+ } while (0)
+
+/*
+ * This will cause a compiler warning because of the cast from char[][] to
+ * type*. This will cause a compile time error if type is too big.
+ */
+#define DEFINE_PER_CPU(type, name) \
+ char name[NR_CPUS][PERCPU_OFFSET]; \
+ typedef char percpu_too_big_##name \
+ [sizeof(type) > PERCPU_OFFSET ? -1 : 1]
+
+#define for_each_possible_cpu(cpu) \
+ for ((cpu) = 0; (cpu) < NR_CPUS; ++(cpu))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
new file mode 100644
index 000000000000..4f1b068e9b7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
@@ -0,0 +1,78 @@
+#include <config.h>
+
+#include "preempt.h"
+
+#include "assume.h"
+#include "locks.h"
+
+/* Support NR_CPUS of at most 64 */
+#define CPU_PREEMPTION_LOCKS_INIT0 LOCK_IMPL_INITIALIZER
+#define CPU_PREEMPTION_LOCKS_INIT1 \
+ CPU_PREEMPTION_LOCKS_INIT0, CPU_PREEMPTION_LOCKS_INIT0
+#define CPU_PREEMPTION_LOCKS_INIT2 \
+ CPU_PREEMPTION_LOCKS_INIT1, CPU_PREEMPTION_LOCKS_INIT1
+#define CPU_PREEMPTION_LOCKS_INIT3 \
+ CPU_PREEMPTION_LOCKS_INIT2, CPU_PREEMPTION_LOCKS_INIT2
+#define CPU_PREEMPTION_LOCKS_INIT4 \
+ CPU_PREEMPTION_LOCKS_INIT3, CPU_PREEMPTION_LOCKS_INIT3
+#define CPU_PREEMPTION_LOCKS_INIT5 \
+ CPU_PREEMPTION_LOCKS_INIT4, CPU_PREEMPTION_LOCKS_INIT4
+
+/*
+ * Simulate disabling preemption by locking a particular cpu. NR_CPUS
+ * should be the actual number of cpus, not just the maximum.
+ */
+struct lock_impl cpu_preemption_locks[NR_CPUS] = {
+ CPU_PREEMPTION_LOCKS_INIT0
+#if (NR_CPUS - 1) & 1
+ , CPU_PREEMPTION_LOCKS_INIT0
+#endif
+#if (NR_CPUS - 1) & 2
+ , CPU_PREEMPTION_LOCKS_INIT1
+#endif
+#if (NR_CPUS - 1) & 4
+ , CPU_PREEMPTION_LOCKS_INIT2
+#endif
+#if (NR_CPUS - 1) & 8
+ , CPU_PREEMPTION_LOCKS_INIT3
+#endif
+#if (NR_CPUS - 1) & 16
+ , CPU_PREEMPTION_LOCKS_INIT4
+#endif
+#if (NR_CPUS - 1) & 32
+ , CPU_PREEMPTION_LOCKS_INIT5
+#endif
+};
+
+#undef CPU_PREEMPTION_LOCKS_INIT0
+#undef CPU_PREEMPTION_LOCKS_INIT1
+#undef CPU_PREEMPTION_LOCKS_INIT2
+#undef CPU_PREEMPTION_LOCKS_INIT3
+#undef CPU_PREEMPTION_LOCKS_INIT4
+#undef CPU_PREEMPTION_LOCKS_INIT5
+
+__thread int thread_cpu_id;
+__thread int preempt_disable_count;
+
+void preempt_disable(void)
+{
+ BUG_ON(preempt_disable_count < 0 || preempt_disable_count == INT_MAX);
+
+ if (preempt_disable_count++)
+ return;
+
+ thread_cpu_id = nondet_int();
+ assume(thread_cpu_id >= 0);
+ assume(thread_cpu_id < NR_CPUS);
+ lock_impl_lock(&cpu_preemption_locks[thread_cpu_id]);
+}
+
+void preempt_enable(void)
+{
+ BUG_ON(preempt_disable_count < 1);
+
+ if (--preempt_disable_count)
+ return;
+
+ lock_impl_unlock(&cpu_preemption_locks[thread_cpu_id]);
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
new file mode 100644
index 000000000000..2f95ee0e4dd5
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
@@ -0,0 +1,58 @@
+#ifndef PREEMPT_H
+#define PREEMPT_H
+
+#include <stdbool.h>
+
+#include "bug_on.h"
+
+/* This flag contains garbage if preempt_disable_count is 0. */
+extern __thread int thread_cpu_id;
+
+/* Support recursive preemption disabling. */
+extern __thread int preempt_disable_count;
+
+void preempt_disable(void);
+void preempt_enable(void);
+
+static inline void preempt_disable_notrace(void)
+{
+ preempt_disable();
+}
+
+static inline void preempt_enable_no_resched(void)
+{
+ preempt_enable();
+}
+
+static inline void preempt_enable_notrace(void)
+{
+ preempt_enable();
+}
+
+static inline int preempt_count(void)
+{
+ return preempt_disable_count;
+}
+
+static inline bool preemptible(void)
+{
+ return !preempt_count();
+}
+
+static inline int get_cpu(void)
+{
+ preempt_disable();
+ return thread_cpu_id;
+}
+
+static inline void put_cpu(void)
+{
+ preempt_enable();
+}
+
+static inline void might_sleep(void)
+{
+ BUG_ON(preempt_disable_count);
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
new file mode 100644
index 000000000000..ac9cbc62b411
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
@@ -0,0 +1,50 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#include <linux/srcu.h>
+
+/* Functions needed from modify_srcu.c */
+bool try_check_zero(struct srcu_struct *sp, int idx, int trycount);
+void srcu_flip(struct srcu_struct *sp);
+
+/* Simpler implementation of synchronize_srcu that ignores batching. */
+void synchronize_srcu(struct srcu_struct *sp)
+{
+ int idx;
+ /*
+ * This code assumes that try_check_zero will succeed anyway,
+ * so there is no point in multiple tries.
+ */
+ const int trycount = 1;
+
+ might_sleep();
+
+ /* Ignore the lock, as multiple writers aren't working yet anyway. */
+
+ idx = 1 ^ (sp->completed & 1);
+
+ /* For comments see srcu_advance_batches. */
+
+ assume(try_check_zero(sp, idx, trycount));
+
+ srcu_flip(sp);
+
+ assume(try_check_zero(sp, idx^1, trycount));
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
new file mode 100644
index 000000000000..e58c8dfd3e90
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
@@ -0,0 +1,102 @@
+#ifndef WORKQUEUES_H
+#define WORKQUEUES_H
+
+#include <stdbool.h>
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "int_typedefs.h"
+
+#include <linux/types.h>
+
+/* Stub workqueue implementation. */
+
+struct work_struct;
+typedef void (*work_func_t)(struct work_struct *work);
+void delayed_work_timer_fn(unsigned long __data);
+
+struct work_struct {
+/* atomic_long_t data; */
+ unsigned long data;
+
+ struct list_head entry;
+ work_func_t func;
+#ifdef CONFIG_LOCKDEP
+ struct lockdep_map lockdep_map;
+#endif
+};
+
+struct timer_list {
+ struct hlist_node entry;
+ unsigned long expires;
+ void (*function)(unsigned long);
+ unsigned long data;
+ u32 flags;
+ int slack;
+};
+
+struct delayed_work {
+ struct work_struct work;
+ struct timer_list timer;
+
+ /* target workqueue and CPU ->timer uses to queue ->work */
+ struct workqueue_struct *wq;
+ int cpu;
+};
+
+
+static inline bool schedule_work(struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool schedule_work_on(int cpu, struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_work(struct workqueue_struct *wq,
+ struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_delayed_work(struct workqueue_struct *wq,
+ struct delayed_work *dwork,
+ unsigned long delay)
+{
+ BUG();
+ return true;
+}
+
+#define INIT_WORK(w, f) \
+ do { \
+ (w)->data = 0; \
+ (w)->func = (f); \
+ } while (0)
+
+#define INIT_DELAYED_WORK(w, f) INIT_WORK(&(w)->work, (f))
+
+#define __WORK_INITIALIZER(n, f) { \
+ .data = 0, \
+ .entry = { &(n).entry, &(n).entry }, \
+ .func = f \
+ }
+
+/* Don't bother initializing timer. */
+#define __DELAYED_WORK_INITIALIZER(n, f, tflags) { \
+ .work = __WORK_INITIALIZER((n).work, (f)), \
+ }
+
+#define DECLARE_WORK(n, f) \
+ struct workqueue_struct n = __WORK_INITIALIZER
+
+#define DECLARE_DELAYED_WORK(n, f) \
+ struct delayed_work n = __DELAYED_WORK_INITIALIZER(n, f, 0)
+
+#define system_power_efficient_wq ((struct workqueue_struct *) NULL)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
new file mode 100644
index 000000000000..f47cb2045f13
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
@@ -0,0 +1 @@
+*.out
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
new file mode 100644
index 000000000000..3a3aee149225
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
@@ -0,0 +1,11 @@
+CBMC_FLAGS = -I../.. -I../../src -I../../include -I../../empty_includes -32 -pointer-check -mm pso
+
+all:
+ for i in ./*.pass; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-pass $$i > $$i.out 2>&1 ; \
+ done
+ for i in ./*.fail; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-fail $$i > $$i.out 2>&1 ; \
+ done
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
new file mode 100644
index 000000000000..40c8075919d1
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DASSERT_END"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
new file mode 100644
index 000000000000..ada5baf0b60d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
new file mode 100644
index 000000000000..8fe00c8db466
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_2"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
new file mode 100644
index 000000000000..612ed6772844
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_3"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
new file mode 100644
index 000000000000..470b1105a112
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
@@ -0,0 +1,72 @@
+#include <src/combined_source.c>
+
+int x;
+int y;
+
+int __unbuffered_tpr_x;
+int __unbuffered_tpr_y;
+
+DEFINE_SRCU(ss);
+
+void rcu_reader(void)
+{
+ int idx;
+
+#ifndef FORCE_FAILURE_3
+ idx = srcu_read_lock(&ss);
+#endif
+ might_sleep();
+
+ __unbuffered_tpr_y = READ_ONCE(y);
+#ifdef FORCE_FAILURE
+ srcu_read_unlock(&ss, idx);
+ idx = srcu_read_lock(&ss);
+#endif
+ WRITE_ONCE(x, 1);
+
+#ifndef FORCE_FAILURE_3
+ srcu_read_unlock(&ss, idx);
+#endif
+ might_sleep();
+}
+
+void *thread_update(void *arg)
+{
+ WRITE_ONCE(y, 1);
+#ifndef FORCE_FAILURE_2
+ synchronize_srcu(&ss);
+#endif
+ might_sleep();
+ __unbuffered_tpr_x = READ_ONCE(x);
+
+ return NULL;
+}
+
+void *thread_process_reader(void *arg)
+{
+ rcu_reader();
+
+ return NULL;
+}
+
+int main(int argc, char *argv[])
+{
+ pthread_t tu;
+ pthread_t tpr;
+
+ if (pthread_create(&tu, NULL, thread_update, NULL))
+ abort();
+ if (pthread_create(&tpr, NULL, thread_process_reader, NULL))
+ abort();
+ if (pthread_join(tu, NULL))
+ abort();
+ if (pthread_join(tpr, NULL))
+ abort();
+ assert(__unbuffered_tpr_y != 0 || __unbuffered_tpr_x != 0);
+
+#ifdef ASSERT_END
+ assert(0);
+#endif
+
+ return 0;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
new file mode 100755
index 000000000000..d1545972a0fa
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+
+# This script expects a mode (either --should-pass or --should-fail) followed by
+# an input file. The script uses the following environment variables. The test C
+# source file is expected to be named test.c in the directory containing the
+# input file.
+#
+# CBMC: The command to run CBMC. Default: cbmc
+# CBMC_FLAGS: Additional flags to pass to CBMC
+# NR_CPUS: Number of cpus to run tests with. Default specified by the test
+# SYNC_SRCU_MODE: Choose implementation of synchronize_srcu. Defaults to simple.
+# kernel: Version included in the linux kernel source.
+# simple: Use try_check_zero directly.
+#
+# The input file is a script that is sourced by this file. It can define any of
+# the following variables to configure the test.
+#
+# test_cbmc_options: Extra options to pass to CBMC.
+# min_cpus_fail: Minimum number of CPUs (NR_CPUS) for verification to fail.
+# The test is expected to pass if it is run with fewer. (Only
+# useful for .fail files)
+# default_cpus: Quantity of CPUs to use for the test, if not specified on the
+# command line. Default: Larger of 2 and MIN_CPUS_FAIL.
+
+set -e
+
+if test "$#" -ne 2; then
+ echo "Expected one option followed by an input file" 1>&2
+ exit 99
+fi
+
+if test "x$1" = "x--should-pass"; then
+ should_pass="yes"
+elif test "x$1" = "x--should-fail"; then
+ should_pass="no"
+else
+ echo "Unrecognized argument '$1'" 1>&2
+
+ # Exit code 99 indicates a hard error.
+ exit 99
+fi
+
+CBMC=${CBMC:-cbmc}
+
+SYNC_SRCU_MODE=${SYNC_SRCU_MODE:-simple}
+
+case ${SYNC_SRCU_MODE} in
+kernel) sync_srcu_mode_flags="" ;;
+simple) sync_srcu_mode_flags="-DUSE_SIMPLE_SYNC_SRCU" ;;
+
+*)
+ echo "Unrecognized argument '${SYNC_SRCU_MODE}'" 1>&2
+ exit 99
+ ;;
+esac
+
+min_cpus_fail=1
+
+c_file=`dirname "$2"`/test.c
+
+# Source the input file.
+. $2
+
+if test ${min_cpus_fail} -gt 2; then
+ default_default_cpus=${min_cpus_fail}
+else
+ default_default_cpus=2
+fi
+default_cpus=${default_cpus:-${default_default_cpus}}
+cpus=${NR_CPUS:-${default_cpus}}
+
+# Check if there are two few cpus to make the test fail.
+if test $cpus -lt ${min_cpus_fail:-0}; then
+ should_pass="yes"
+fi
+
+cbmc_opts="-DNR_CPUS=${cpus} ${sync_srcu_mode_flags} ${test_cbmc_options} ${CBMC_FLAGS}"
+
+echo "Running CBMC: ${CBMC} ${cbmc_opts} ${c_file}"
+if ${CBMC} ${cbmc_opts} "${c_file}"; then
+ # Verification successful. Make sure that it was supposed to verify.
+ test "x${should_pass}" = xyes
+else
+ cbmc_exit_status=$?
+
+ # An exit status of 10 indicates a failed verification.
+ # (see cbmc_parse_optionst::do_bmc in the CBMC source code)
+ if test ${cbmc_exit_status} -eq 10 && test "x${should_pass}" = xno; then
+ :
+ else
+ echo "CBMC returned ${cbmc_exit_status} exit status" 1>&2
+
+ # Parse errors have exit status 6. Any other type of error
+ # should be considered a hard error.
+ if test ${cbmc_exit_status} -ne 6 && \
+ test ${cbmc_exit_status} -ne 10; then
+ exit 99
+ else
+ exit 1
+ fi
+ fi
+fi
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts.
2017-01-14 9:19 ` [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts Paul E. McKenney
@ 2017-01-14 9:31 ` Ingo Molnar
2017-01-14 19:48 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2017-01-14 9:31 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, Lance Roy
Noticed a few minor nits:
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> From: Lance Roy <ldr709@gmail.com>
>
> SRCU uses two per-cpu counters: a nesting counter to count the number of
> active critical sections, and a sequence counter to ensure that the nesting
> counters don't change while they are being added together in
> srcu_readers_active_idx_check().
>
> This patch instead uses per-cpu lock and unlock counters. Because the both
> counters only increase and srcu_readers_active_idx_check() reads the unlock
> counter before the lock counter, this achieves the same end without having
> to increment two different counters in srcu_read_lock(). This also saves a
> smp_mb() in srcu_readers_active_idx_check().
typo:
s/Because the both counters
Because both counters
>
> A possible problem with this patch is that it can only handle
> ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
> handle up to ULONG_MAX.
I don't think this is a problem! :-)
>
> Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Signed-off-by: Lance Roy <ldr709@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> ---
> include/linux/srcu.h | 4 +-
> kernel/rcu/rcutorture.c | 18 +++++++-
> kernel/rcu/srcu.c | 117 ++++++++++++++++++------------------------------
> 3 files changed, 62 insertions(+), 77 deletions(-)
>
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index dc8eb63c6568..0caea34d8c5f 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -34,8 +34,8 @@
> #include <linux/workqueue.h>
>
> struct srcu_struct_array {
> - unsigned long c[2];
> - unsigned long seq[2];
> + unsigned long lock_count[2];
> + unsigned long unlock_count[2];
> };
>
> struct rcu_batch {
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 87c51225ceec..6e4fd7680c70 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -564,10 +564,24 @@ static void srcu_torture_stats(void)
> pr_alert("%s%s per-CPU(idx=%d):",
> torture_type, TORTURE_FLAG, idx);
> for_each_possible_cpu(cpu) {
> + unsigned long l0, l1;
> + unsigned long u0, u1;
> long c0, c1;
> + struct srcu_struct_array* counts =
> + per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
Please don't break the line to pacify checkpatch - if the line is too long then
maybe split out the loop body into a helper function - but keeping it a bit longer
than 80 cols is fine as well.
>
> - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> + u0 = counts->unlock_count[!idx];
> + u1 = counts->unlock_count[idx];
> +
> + /* Make sure that a lock is always counted if the corresponding
> + unlock is counted. */
> + smp_rmb();
That's not the standard kernel code comment style.
> +
> + l0 = counts->lock_count[!idx];
> + l1 = counts->lock_count[idx];
> +
> + c0 = (long)(l0 - u0);
> + c1 = (long)(l1 - u1);
These type casts look unnecessary to me.
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> + struct srcu_struct_array* cpu_counts =
> + per_cpu_ptr(sp->per_cpu_ref, cpu);
> + t = READ_ONCE(cpu_counts->lock_count[idx]);
> sum += t;
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> + struct srcu_struct_array* cpu_counts =
> + per_cpu_ptr(sp->per_cpu_ref, cpu);
> + t = READ_ONCE(cpu_counts->unlock_count[idx]);
> sum += t;
These linebreak look ugly as well. Some abbreviation of types and variables might
help:
s/srcu_struct_array/srcu_array
s/cpu_counts/cpuc
?
> + * If the locks are the same as the unlocks, then there must of have
> + * been no readers on this index at some time in between. This does not
> + * mean that there are no more readers, as one could have read the
> + * current index but have incremented the lock counter yet.
>
> + * Note that there can be at most NR_CPUS worth of readers using the old
> + * index that haven't incremented ->lock_count[] yet. Therefore, the
> + * sum of the ->lock_count[]s cannot increment enough times to overflow
> + * and end up equal the sum of the ->unlock_count[]s, as long as there
> + * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this does
> + * mean that systems having more than a billion or so CPUs need to be
> + * 64-bit systems.) Therefore, the only way that the return values of
> + * the two calls to srcu_readers_(un)lock_idx() can be equal is if there
> + * are no active readers using this index.
typo:
s/must of have been no readers/
must have been no readers
Also, maybe I'm misreading it, but shouldn't it be:
s/as one could have read the current index but have incremented the lock counter yet.
/as one could have read the current index but not have incremented the lock counter yet.
?
Also, the title:
srcu: More efficient reader counts.
should have a verb and no full stop, i.e. something like:
srcu: Implement more efficient reader counts
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 9:20 ` [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
@ 2017-01-14 9:35 ` Ingo Molnar
2017-01-14 19:54 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2017-01-14 9:35 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> If a process invokes synchronize_srcu(), is delayed just the right amount
> of time, and thus does not sleep when waiting for the grace period to
> complete, there is no ordering between the end of the grace period and
> the code following the synchronize_srcu(). Similarly, there can be a
> lack of ordering between the end of the SRCU grace period and callback
> invocation.
>
> This commit adds the necessary ordering.
>
> Reported-by: Lance Roy <ldr709@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> include/linux/rcupdate.h | 12 ++++++++++++
> kernel/rcu/srcu.c | 5 +++++
> kernel/rcu/tree.h | 12 ------------
> 3 files changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 01f71e1d2e94..608d56f908f2 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -1161,5 +1161,17 @@ do { \
> ftrace_dump(oops_dump_mode); \
> } while (0)
>
> +/*
> + * Place this after a lock-acquisition primitive to guarantee that
> + * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
> + * if the UNLOCK and LOCK are executed by the same CPU or if the
> + * UNLOCK and LOCK operate on the same lock variable.
minor typo:
s/an UNLOCK+LOCK pair act as
an UNLOCK+LOCK pair acts as
> + */
> +#ifdef CONFIG_PPC
> +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> +#else /* #ifdef CONFIG_PPC */
> +#define smp_mb__after_unlock_lock() do { } while (0)
> +#endif /* #else #ifdef CONFIG_PPC */
Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
#ifdefs into generic headers is generally frowned upon.
The canonical approach would be either to define a helper Kconfig variable that
can be set by PPC (but other architectures don't need to set it), or to expose a
suitable macro (function) for architectures to define in their barrier.h arch
header file.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts.
2017-01-14 9:31 ` Ingo Molnar
@ 2017-01-14 19:48 ` Paul E. McKenney
0 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 19:48 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, Lance Roy
On Sat, Jan 14, 2017 at 10:31:15AM +0100, Ingo Molnar wrote:
>
> Noticed a few minor nits:
And thank you for the review and comments!
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > From: Lance Roy <ldr709@gmail.com>
> >
> > SRCU uses two per-cpu counters: a nesting counter to count the number of
> > active critical sections, and a sequence counter to ensure that the nesting
> > counters don't change while they are being added together in
> > srcu_readers_active_idx_check().
> >
> > This patch instead uses per-cpu lock and unlock counters. Because the both
> > counters only increase and srcu_readers_active_idx_check() reads the unlock
> > counter before the lock counter, this achieves the same end without having
> > to increment two different counters in srcu_read_lock(). This also saves a
> > smp_mb() in srcu_readers_active_idx_check().
>
> typo:
>
> s/Because the both counters
> Because both counters
Fixed!
> > A possible problem with this patch is that it can only handle
> > ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
> > handle up to ULONG_MAX.
>
> I don't think this is a problem! :-)
Here is hoping! ;-)
> > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Signed-off-by: Lance Roy <ldr709@gmail.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > ---
> > include/linux/srcu.h | 4 +-
> > kernel/rcu/rcutorture.c | 18 +++++++-
> > kernel/rcu/srcu.c | 117 ++++++++++++++++++------------------------------
> > 3 files changed, 62 insertions(+), 77 deletions(-)
> >
> > diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> > index dc8eb63c6568..0caea34d8c5f 100644
> > --- a/include/linux/srcu.h
> > +++ b/include/linux/srcu.h
> > @@ -34,8 +34,8 @@
> > #include <linux/workqueue.h>
> >
> > struct srcu_struct_array {
> > - unsigned long c[2];
> > - unsigned long seq[2];
> > + unsigned long lock_count[2];
> > + unsigned long unlock_count[2];
> > };
> >
> > struct rcu_batch {
> > diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> > index 87c51225ceec..6e4fd7680c70 100644
> > --- a/kernel/rcu/rcutorture.c
> > +++ b/kernel/rcu/rcutorture.c
> > @@ -564,10 +564,24 @@ static void srcu_torture_stats(void)
> > pr_alert("%s%s per-CPU(idx=%d):",
> > torture_type, TORTURE_FLAG, idx);
> > for_each_possible_cpu(cpu) {
> > + unsigned long l0, l1;
> > + unsigned long u0, u1;
> > long c0, c1;
> > + struct srcu_struct_array* counts =
> > + per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
>
> Please don't break the line to pacify checkpatch - if the line is too long then
> maybe split out the loop body into a helper function - but keeping it a bit longer
> than 80 cols is fine as well.
Creating a helper function woujld leave me several characters over still,
so I just created the long line. Another approach would be to split the
definition and the initialization into two statements, but that would
add a line.
> > - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> > - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> > + u0 = counts->unlock_count[!idx];
> > + u1 = counts->unlock_count[idx];
> > +
> > + /* Make sure that a lock is always counted if the corresponding
> > + unlock is counted. */
> > + smp_rmb();
>
> That's not the standard kernel code comment style.
That is embarrassing! Fixed.
> > +
> > + l0 = counts->lock_count[!idx];
> > + l1 = counts->lock_count[idx];
> > +
> > + c0 = (long)(l0 - u0);
> > + c1 = (long)(l1 - u1);
>
> These type casts look unnecessary to me.
Indeed, given that we are assigning to a long rather than just computing.
> > for_each_possible_cpu(cpu) {
> > - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> > + struct srcu_struct_array* cpu_counts =
> > + per_cpu_ptr(sp->per_cpu_ref, cpu);
> > + t = READ_ONCE(cpu_counts->lock_count[idx]);
> > sum += t;
>
>
> > for_each_possible_cpu(cpu) {
> > - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> > + struct srcu_struct_array* cpu_counts =
> > + per_cpu_ptr(sp->per_cpu_ref, cpu);
> > + t = READ_ONCE(cpu_counts->unlock_count[idx]);
> > sum += t;
>
> These linebreak look ugly as well. Some abbreviation of types and variables might
> help:
>
> s/srcu_struct_array/srcu_array
> s/cpu_counts/cpuc
>
> ?
Why not? Fixed. ;-)
> > + * If the locks are the same as the unlocks, then there must of have
> > + * been no readers on this index at some time in between. This does not
> > + * mean that there are no more readers, as one could have read the
> > + * current index but have incremented the lock counter yet.
> >
> > + * Note that there can be at most NR_CPUS worth of readers using the old
> > + * index that haven't incremented ->lock_count[] yet. Therefore, the
> > + * sum of the ->lock_count[]s cannot increment enough times to overflow
> > + * and end up equal the sum of the ->unlock_count[]s, as long as there
> > + * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this does
> > + * mean that systems having more than a billion or so CPUs need to be
> > + * 64-bit systems.) Therefore, the only way that the return values of
> > + * the two calls to srcu_readers_(un)lock_idx() can be equal is if there
> > + * are no active readers using this index.
>
> typo:
>
> s/must of have been no readers/
> must have been no readers
>
> Also, maybe I'm misreading it, but shouldn't it be:
>
> s/as one could have read the current index but have incremented the lock counter yet.
> /as one could have read the current index but not have incremented the lock counter yet.
>
> ?
Agreed on both, fixed.
> Also, the title:
>
> srcu: More efficient reader counts.
>
> should have a verb and no full stop, i.e. something like:
>
> srcu: Implement more efficient reader counts
And this one as well.
Thanx, Paul
> Thanks,
>
> Ingo
>
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 9:35 ` Ingo Molnar
@ 2017-01-14 19:54 ` Paul E. McKenney
2017-01-14 21:41 ` Paul E. McKenney
2017-01-15 6:54 ` Ingo Molnar
0 siblings, 2 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 19:54 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani
On Sat, Jan 14, 2017 at 10:35:50AM +0100, Ingo Molnar wrote:
>
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > If a process invokes synchronize_srcu(), is delayed just the right amount
> > of time, and thus does not sleep when waiting for the grace period to
> > complete, there is no ordering between the end of the grace period and
> > the code following the synchronize_srcu(). Similarly, there can be a
> > lack of ordering between the end of the SRCU grace period and callback
> > invocation.
> >
> > This commit adds the necessary ordering.
> >
> > Reported-by: Lance Roy <ldr709@gmail.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> > include/linux/rcupdate.h | 12 ++++++++++++
> > kernel/rcu/srcu.c | 5 +++++
> > kernel/rcu/tree.h | 12 ------------
> > 3 files changed, 17 insertions(+), 12 deletions(-)
> >
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index 01f71e1d2e94..608d56f908f2 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -1161,5 +1161,17 @@ do { \
> > ftrace_dump(oops_dump_mode); \
> > } while (0)
> >
> > +/*
> > + * Place this after a lock-acquisition primitive to guarantee that
> > + * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
> > + * if the UNLOCK and LOCK are executed by the same CPU or if the
> > + * UNLOCK and LOCK operate on the same lock variable.
>
> minor typo:
>
> s/an UNLOCK+LOCK pair act as
> an UNLOCK+LOCK pair acts as
Fixed.
> > + */
> > +#ifdef CONFIG_PPC
> > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > +#else /* #ifdef CONFIG_PPC */
> > +#define smp_mb__after_unlock_lock() do { } while (0)
> > +#endif /* #else #ifdef CONFIG_PPC */
>
> Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
> #ifdefs into generic headers is generally frowned upon.
>
> The canonical approach would be either to define a helper Kconfig variable that
> can be set by PPC (but other architectures don't need to set it), or to expose a
> suitable macro (function) for architectures to define in their barrier.h arch
> header file.
Very well, I will add a separate commit for this. 4.11 OK?
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 19:54 ` Paul E. McKenney
@ 2017-01-14 21:41 ` Paul E. McKenney
2017-01-15 7:11 ` Ingo Molnar
2017-01-23 8:12 ` Michael Ellerman
2017-01-15 6:54 ` Ingo Molnar
1 sibling, 2 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-14 21:41 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev
On Sat, Jan 14, 2017 at 11:54:17AM -0800, Paul E. McKenney wrote:
> On Sat, Jan 14, 2017 at 10:35:50AM +0100, Ingo Molnar wrote:
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
[ . . . ]
> > > + */
> > > +#ifdef CONFIG_PPC
> > > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > > +#else /* #ifdef CONFIG_PPC */
> > > +#define smp_mb__after_unlock_lock() do { } while (0)
> > > +#endif /* #else #ifdef CONFIG_PPC */
> >
> > Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
> > #ifdefs into generic headers is generally frowned upon.
> >
> > The canonical approach would be either to define a helper Kconfig variable that
> > can be set by PPC (but other architectures don't need to set it), or to expose a
> > suitable macro (function) for architectures to define in their barrier.h arch
> > header file.
>
> Very well, I will add a separate commit for this. 4.11 OK?
Does the patch below seem reasonable?
Thanx, Paul
------------------------------------------------------------------------
commit 271c0601237c41a279f975563e13837bace0df03
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Sat Jan 14 13:32:50 2017 -0800
rcu: Make arch select smp_mb__after_unlock_lock() strength
The definition of smp_mb__after_unlock_lock() is currently smp_mb()
for CONFIG_PPC and a no-op otherwise. It would be better to instead
provide an architecture-selectable Kconfig option, and select the
strength of smp_mb__after_unlock_lock() based on that option. This
commit therefore creates CONFIG_ARCH_WEAK_RELACQ, has PPC select it,
and bases the definition of smp_mb__after_unlock_lock() on this new
CONFIG_ARCH_WEAK_RELACQ Kconfig option.
Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: <linuxppc-dev@lists.ozlabs.org>
diff --git a/arch/Kconfig b/arch/Kconfig
index 99839c23d453..94dd90d33f95 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -316,6 +316,9 @@ config HAVE_CMPXCHG_LOCAL
config HAVE_CMPXCHG_DOUBLE
bool
+config ARCH_WEAK_RELACQ
+ bool
+
config ARCH_WANT_IPC_PARSE_VERSION
bool
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index a8ee573fe610..e7083d27271e 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -165,6 +165,7 @@ config PPC
select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_KERNEL_GZIP
select HAVE_CC_STACKPROTECTOR
+ select ARCH_WEAK_RELACQ
config GENERIC_CSUM
def_bool CPU_LITTLE_ENDIAN
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 357b32aaea48..5fdfe874229e 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1175,11 +1175,11 @@ do { \
* if the UNLOCK and LOCK are executed by the same CPU or if the
* UNLOCK and LOCK operate on the same lock variable.
*/
-#ifdef CONFIG_PPC
+#ifdef CONFIG_ARCH_WEAK_RELACQ
#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
-#else /* #ifdef CONFIG_PPC */
+#else /* #ifdef CONFIG_ARCH_WEAK_RELACQ */
#define smp_mb__after_unlock_lock() do { } while (0)
-#endif /* #else #ifdef CONFIG_PPC */
+#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELACQ */
#endif /* __LINUX_RCUPDATE_H */
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 19:54 ` Paul E. McKenney
2017-01-14 21:41 ` Paul E. McKenney
@ 2017-01-15 6:54 ` Ingo Molnar
1 sibling, 0 replies; 43+ messages in thread
From: Ingo Molnar @ 2017-01-15 6:54 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > + */
> > > +#ifdef CONFIG_PPC
> > > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > > +#else /* #ifdef CONFIG_PPC */
> > > +#define smp_mb__after_unlock_lock() do { } while (0)
> > > +#endif /* #else #ifdef CONFIG_PPC */
> >
> > Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
> > #ifdefs into generic headers is generally frowned upon.
> >
> > The canonical approach would be either to define a helper Kconfig variable that
> > can be set by PPC (but other architectures don't need to set it), or to expose a
> > suitable macro (function) for architectures to define in their barrier.h arch
> > header file.
>
> Very well, I will add a separate commit for this. 4.11 OK?
Sure!
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 21:41 ` Paul E. McKenney
@ 2017-01-15 7:11 ` Ingo Molnar
2017-01-15 7:40 ` Paul E. McKenney
2017-01-23 8:12 ` Michael Ellerman
1 sibling, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2017-01-15 7:11 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 357b32aaea48..5fdfe874229e 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -1175,11 +1175,11 @@ do { \
> * if the UNLOCK and LOCK are executed by the same CPU or if the
> * UNLOCK and LOCK operate on the same lock variable.
> */
> -#ifdef CONFIG_PPC
> +#ifdef CONFIG_ARCH_WEAK_RELACQ
> #define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> -#else /* #ifdef CONFIG_PPC */
> +#else /* #ifdef CONFIG_ARCH_WEAK_RELACQ */
> #define smp_mb__after_unlock_lock() do { } while (0)
> -#endif /* #else #ifdef CONFIG_PPC */
> +#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELACQ */
>
>
So at the risk of sounding totally pedantic, why not structure it like the
existing smp_mb__before/after*() primitives in barrier.h?
That allows asm-generic/barrier.h to pick up the definition - for example in the
case of smp_acquire__after_ctrl_dep() we do:
#ifndef smp_acquire__after_ctrl_dep
#define smp_acquire__after_ctrl_dep() smp_rmb()
#endif
Which allows Tile to relax it:
arch/tile/include/asm/barrier.h:#define smp_acquire__after_ctrl_dep() barrier()
I.e. I'd move the API definition out of rcupdate.h and into barrier.h - even
though tree-RCU is the only user of this barrier type.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 7:11 ` Ingo Molnar
@ 2017-01-15 7:40 ` Paul E. McKenney
2017-01-15 7:57 ` Ingo Molnar
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 7:40 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
On Sun, Jan 15, 2017 at 08:11:23AM +0100, Ingo Molnar wrote:
>
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index 357b32aaea48..5fdfe874229e 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -1175,11 +1175,11 @@ do { \
> > * if the UNLOCK and LOCK are executed by the same CPU or if the
> > * UNLOCK and LOCK operate on the same lock variable.
> > */
> > -#ifdef CONFIG_PPC
> > +#ifdef CONFIG_ARCH_WEAK_RELACQ
> > #define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > -#else /* #ifdef CONFIG_PPC */
> > +#else /* #ifdef CONFIG_ARCH_WEAK_RELACQ */
> > #define smp_mb__after_unlock_lock() do { } while (0)
> > -#endif /* #else #ifdef CONFIG_PPC */
> > +#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELACQ */
> >
> >
>
> So at the risk of sounding totally pedantic, why not structure it like the
> existing smp_mb__before/after*() primitives in barrier.h?
>
> That allows asm-generic/barrier.h to pick up the definition - for example in the
> case of smp_acquire__after_ctrl_dep() we do:
>
> #ifndef smp_acquire__after_ctrl_dep
> #define smp_acquire__after_ctrl_dep() smp_rmb()
> #endif
>
> Which allows Tile to relax it:
>
> arch/tile/include/asm/barrier.h:#define smp_acquire__after_ctrl_dep() barrier()
>
> I.e. I'd move the API definition out of rcupdate.h and into barrier.h - even
> though tree-RCU is the only user of this barrier type.
I wouldn't have any problem with that, however, some time back it was
moved into RCU because (you guessed it!) RCU is the only user. ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 7:40 ` Paul E. McKenney
@ 2017-01-15 7:57 ` Ingo Molnar
2017-01-15 9:24 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2017-01-15 7:57 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Jan 15, 2017 at 08:11:23AM +0100, Ingo Molnar wrote:
> >
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> >
> > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > index 357b32aaea48..5fdfe874229e 100644
> > > --- a/include/linux/rcupdate.h
> > > +++ b/include/linux/rcupdate.h
> > > @@ -1175,11 +1175,11 @@ do { \
> > > * if the UNLOCK and LOCK are executed by the same CPU or if the
> > > * UNLOCK and LOCK operate on the same lock variable.
> > > */
> > > -#ifdef CONFIG_PPC
> > > +#ifdef CONFIG_ARCH_WEAK_RELACQ
> > > #define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > > -#else /* #ifdef CONFIG_PPC */
> > > +#else /* #ifdef CONFIG_ARCH_WEAK_RELACQ */
> > > #define smp_mb__after_unlock_lock() do { } while (0)
> > > -#endif /* #else #ifdef CONFIG_PPC */
> > > +#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELACQ */
> > >
> > >
> >
> > So at the risk of sounding totally pedantic, why not structure it like the
> > existing smp_mb__before/after*() primitives in barrier.h?
> >
> > That allows asm-generic/barrier.h to pick up the definition - for example in the
> > case of smp_acquire__after_ctrl_dep() we do:
> >
> > #ifndef smp_acquire__after_ctrl_dep
> > #define smp_acquire__after_ctrl_dep() smp_rmb()
> > #endif
> >
> > Which allows Tile to relax it:
> >
> > arch/tile/include/asm/barrier.h:#define smp_acquire__after_ctrl_dep() barrier()
> >
> > I.e. I'd move the API definition out of rcupdate.h and into barrier.h - even
> > though tree-RCU is the only user of this barrier type.
>
> I wouldn't have any problem with that, however, some time back it was
> moved into RCU because (you guessed it!) RCU is the only user. ;-)
Indeed ...
[sounds of rummaging around in the Git tree]
I found this commit of yours from ancient history (more than a year ago!):
commit 12d560f4ea87030667438a169912380be00cea4b
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Tue Jul 14 18:35:23 2015 -0700
rcu,locking: Privatize smp_mb__after_unlock_lock()
RCU is the only thing that uses smp_mb__after_unlock_lock(), and is
likely the only thing that ever will use it, so this commit makes this
macro private to RCU.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>
So I concur and I'm fine with your patch - or with the status quo code as well.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 7:57 ` Ingo Molnar
@ 2017-01-15 9:24 ` Paul E. McKenney
2017-01-15 9:40 ` Ingo Molnar
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 9:24 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
On Sun, Jan 15, 2017 at 08:57:11AM +0100, Ingo Molnar wrote:
>
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > On Sun, Jan 15, 2017 at 08:11:23AM +0100, Ingo Molnar wrote:
> > >
> > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > >
> > > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > > index 357b32aaea48..5fdfe874229e 100644
> > > > --- a/include/linux/rcupdate.h
> > > > +++ b/include/linux/rcupdate.h
> > > > @@ -1175,11 +1175,11 @@ do { \
> > > > * if the UNLOCK and LOCK are executed by the same CPU or if the
> > > > * UNLOCK and LOCK operate on the same lock variable.
> > > > */
> > > > -#ifdef CONFIG_PPC
> > > > +#ifdef CONFIG_ARCH_WEAK_RELACQ
> > > > #define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> > > > -#else /* #ifdef CONFIG_PPC */
> > > > +#else /* #ifdef CONFIG_ARCH_WEAK_RELACQ */
> > > > #define smp_mb__after_unlock_lock() do { } while (0)
> > > > -#endif /* #else #ifdef CONFIG_PPC */
> > > > +#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELACQ */
> > > >
> > > >
> > >
> > > So at the risk of sounding totally pedantic, why not structure it like the
> > > existing smp_mb__before/after*() primitives in barrier.h?
> > >
> > > That allows asm-generic/barrier.h to pick up the definition - for example in the
> > > case of smp_acquire__after_ctrl_dep() we do:
> > >
> > > #ifndef smp_acquire__after_ctrl_dep
> > > #define smp_acquire__after_ctrl_dep() smp_rmb()
> > > #endif
> > >
> > > Which allows Tile to relax it:
> > >
> > > arch/tile/include/asm/barrier.h:#define smp_acquire__after_ctrl_dep() barrier()
> > >
> > > I.e. I'd move the API definition out of rcupdate.h and into barrier.h - even
> > > though tree-RCU is the only user of this barrier type.
> >
> > I wouldn't have any problem with that, however, some time back it was
> > moved into RCU because (you guessed it!) RCU is the only user. ;-)
>
> Indeed ...
>
> [sounds of rummaging around in the Git tree]
>
> I found this commit of yours from ancient history (more than a year ago!):
>
> commit 12d560f4ea87030667438a169912380be00cea4b
> Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Date: Tue Jul 14 18:35:23 2015 -0700
>
> rcu,locking: Privatize smp_mb__after_unlock_lock()
>
> RCU is the only thing that uses smp_mb__after_unlock_lock(), and is
> likely the only thing that ever will use it, so this commit makes this
> macro private to RCU.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>
>
> So I concur and I'm fine with your patch - or with the status quo code as well.
I already have the patch queued, so how about I keep it if I get an ack
from the powerpc guys and drop it otherwise?
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 9:24 ` Paul E. McKenney
@ 2017-01-15 9:40 ` Ingo Molnar
2017-01-15 19:45 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2017-01-15 9:40 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > [sounds of rummaging around in the Git tree]
> >
> > I found this commit of yours from ancient history (more than a year ago!):
> >
> > commit 12d560f4ea87030667438a169912380be00cea4b
> > Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Date: Tue Jul 14 18:35:23 2015 -0700
> >
> > rcu,locking: Privatize smp_mb__after_unlock_lock()
> >
> > RCU is the only thing that uses smp_mb__after_unlock_lock(), and is
> > likely the only thing that ever will use it, so this commit makes this
> > macro private to RCU.
> >
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>
> >
> > So I concur and I'm fine with your patch - or with the status quo code as well.
>
> I already have the patch queued, so how about I keep it if I get an ack
> from the powerpc guys and drop it otherwise?
Yeah, sounds good! Your patch made me look up 'RelAcq' so it has documentation
value as well ;-)
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 9:40 ` Ingo Molnar
@ 2017-01-15 19:45 ` Paul E. McKenney
2017-01-16 6:56 ` Ingo Molnar
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 19:45 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
On Sun, Jan 15, 2017 at 10:40:58AM +0100, Ingo Molnar wrote:
>
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > > [sounds of rummaging around in the Git tree]
> > >
> > > I found this commit of yours from ancient history (more than a year ago!):
> > >
> > > commit 12d560f4ea87030667438a169912380be00cea4b
> > > Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > Date: Tue Jul 14 18:35:23 2015 -0700
> > >
> > > rcu,locking: Privatize smp_mb__after_unlock_lock()
> > >
> > > RCU is the only thing that uses smp_mb__after_unlock_lock(), and is
> > > likely the only thing that ever will use it, so this commit makes this
> > > macro private to RCU.
> > >
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > Cc: Will Deacon <will.deacon@arm.com>
> > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > > Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>
> > >
> > > So I concur and I'm fine with your patch - or with the status quo code as well.
> >
> > I already have the patch queued, so how about I keep it if I get an ack
> > from the powerpc guys and drop it otherwise?
>
> Yeah, sounds good! Your patch made me look up 'RelAcq' so it has documentation
> value as well ;-)
;-) ;-) ;-)
Looking forward, my guess would be that if some other code needs
smp_mb__after_unlock_lock() or if some other architecture needs
non-smb_mb() special handling, I should consider making it work the
same as smp_mb__after_atomic() and friends. Does that seem like a
reasonable thought?
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11
2017-01-14 9:19 [PATCH tip/core/rcu 0/3] SRCU updates for 4.11 Paul E. McKenney
` (2 preceding siblings ...)
2017-01-14 9:20 ` [PATCH tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
@ 2017-01-15 22:41 ` Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts Paul E. McKenney
` (3 more replies)
3 siblings, 4 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 22:41 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani
Hello!
This series provides updates to SRCU:
1. This is a rewrite of the algorithm simplifying reader-count
tracking. Algorithm courtesy of Mathieu Desnoyers, implementation
courtesy of Lance Roy.
2. Force full grace-period ordering in SRCU.
3. Add CBMC-based formal verification for SRCU, courtesy of Lance Roy.
Updates since v1:
o Applied Ingo Molnar feedback.
o Fix some checkpatch issues.
Thanx, Paul
------------------------------------------------------------------------
include/linux/rcupdate.h | 12
include/linux/srcu.h | 10
kernel/rcu/rcutorture.c | 19
kernel/rcu/srcu.c | 128 +--
kernel/rcu/tree.h | 12
tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h | 155 ++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 ++++++++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 27
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h | 220 +++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h | 92 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c | 78 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c | 72 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh | 102 ++
34 files changed, 1666 insertions(+), 97 deletions(-)
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
@ 2017-01-15 22:42 ` Paul E. McKenney
2017-01-23 20:17 ` Lance Roy
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
` (2 subsequent siblings)
3 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 22:42 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
SRCU uses two per-cpu counters: a nesting counter to count the number of
active critical sections, and a sequence counter to ensure that the nesting
counters don't change while they are being added together in
srcu_readers_active_idx_check().
This patch instead uses per-cpu lock and unlock counters. Because both
counters only increase and srcu_readers_active_idx_check() reads the unlock
counter before the lock counter, this achieves the same end without having
to increment two different counters in srcu_read_lock(). This also saves a
smp_mb() in srcu_readers_active_idx_check().
A possible problem with this patch is that it can only handle
ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
handle up to ULONG_MAX.
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
include/linux/srcu.h | 10 ++--
kernel/rcu/rcutorture.c | 19 +++++++-
kernel/rcu/srcu.c | 123 ++++++++++++++++++------------------------------
3 files changed, 67 insertions(+), 85 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index dc8eb63c6568..a598cf3ac70c 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -33,9 +33,9 @@
#include <linux/rcupdate.h>
#include <linux/workqueue.h>
-struct srcu_struct_array {
- unsigned long c[2];
- unsigned long seq[2];
+struct srcu_array {
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
};
struct rcu_batch {
@@ -46,7 +46,7 @@ struct rcu_batch {
struct srcu_struct {
unsigned long completed;
- struct srcu_struct_array __percpu *per_cpu_ref;
+ struct srcu_array __percpu *per_cpu_ref;
spinlock_t queue_lock; /* protect ->batch_queue, ->running */
bool running;
/* callbacks just queued */
@@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
* See include/linux/percpu-defs.h for the rules on per-CPU variables.
*/
#define __DEFINE_SRCU(name, is_static) \
- static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
+ static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 87c51225ceec..d81345be730e 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
pr_alert("%s%s per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
+ unsigned long l0, l1;
+ unsigned long u0, u1;
long c0, c1;
+ struct srcu_array *counts = per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
- c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
+ u0 = counts->unlock_count[!idx];
+ u1 = counts->unlock_count[idx];
+
+ /*
+ * Make sure that a lock is always counted if the corresponding
+ * unlock is counted.
+ */
+ smp_rmb();
+
+ l0 = counts->lock_count[!idx];
+ l1 = counts->lock_count[idx];
+
+ c0 = l0 - u0;
+ c1 = l1 - u1;
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 9b9cdd549caa..ddabf5fbf562 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
rcu_batch_init(&sp->batch_check1);
rcu_batch_init(&sp->batch_done);
INIT_DELAYED_WORK(&sp->work, process_srcu);
- sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
+ sp->per_cpu_ref = alloc_percpu(struct srcu_array);
return sp->per_cpu_ref ? 0 : -ENOMEM;
}
@@ -141,114 +141,78 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
- * Returns approximate total of the readers' ->seq[] values for the
+ * Returns approximate total of the readers' ->lock_count[] values for the
* rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[idx]);
}
return sum;
}
/*
- * Returns approximate number of readers active on the specified rank
- * of the per-CPU ->c[] counters.
+ * Returns approximate total of the readers' ->unlock_count[] values for the
+ * rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->unlock_count[idx]);
}
return sum;
}
/*
* Return true if the number of pre-existing readers is determined to
- * be stably zero. An example unstable zero can occur if the call
- * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
- * but due to task migration, sees the corresponding __srcu_read_unlock()
- * decrement. This can happen because srcu_readers_active_idx() takes
- * time to sum the array, and might in fact be interrupted or preempted
- * partway through the summation.
+ * be zero.
*/
static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
{
- unsigned long seq;
+ unsigned long unlocks;
- seq = srcu_readers_seq_idx(sp, idx);
+ unlocks = srcu_readers_unlock_idx(sp, idx);
/*
- * The following smp_mb() A pairs with the smp_mb() B located in
- * __srcu_read_lock(). This pairing ensures that if an
- * __srcu_read_lock() increments its counter after the summation
- * in srcu_readers_active_idx(), then the corresponding SRCU read-side
- * critical section will see any changes made prior to the start
- * of the current SRCU grace period.
+ * Make sure that a lock is always counted if the corresponding unlock
+ * is counted. Needs to be a smp_mb() as the read side may contain a
+ * read from a variable that is written to before the synchronize_srcu()
+ * in the write side. In this case smp_mb()s A and B act like the store
+ * buffering pattern.
*
- * Also, if the above call to srcu_readers_seq_idx() saw the
- * increment of ->seq[], then the call to srcu_readers_active_idx()
- * must see the increment of ->c[].
+ * This smp_mb() also pairs with smp_mb() C to prevent writes after the
+ * synchronize_srcu() from being executed before the grace period ends.
*/
smp_mb(); /* A */
/*
- * Note that srcu_readers_active_idx() can incorrectly return
- * zero even though there is a pre-existing reader throughout.
- * To see this, suppose that task A is in a very long SRCU
- * read-side critical section that started on CPU 0, and that
- * no other reader exists, so that the sum of the counters
- * is equal to one. Then suppose that task B starts executing
- * srcu_readers_active_idx(), summing up to CPU 1, and then that
- * task C starts reading on CPU 0, so that its increment is not
- * summed, but finishes reading on CPU 2, so that its decrement
- * -is- summed. Then when task B completes its sum, it will
- * incorrectly get zero, despite the fact that task A has been
- * in its SRCU read-side critical section the whole time.
+ * If the locks are the same as the unlocks, then there must have
+ * been no readers on this index at some time in between. This does not
+ * mean that there are no more readers, as one could have read the
+ * current index but not have incremented the lock counter yet.
*
- * We therefore do a validation step should srcu_readers_active_idx()
- * return zero.
+ * Note that there can be at most NR_CPUS worth of readers using the old
+ * index that haven't incremented ->lock_count[] yet. Therefore, the
+ * sum of the ->lock_count[]s cannot increment enough times to overflow
+ * and end up equal the sum of the ->unlock_count[]s, as long as there
+ * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this does
+ * mean that systems having more than a billion or so CPUs need to be
+ * 64-bit systems.) Therefore, the only way that the return values of
+ * the two calls to srcu_readers_(un)lock_idx() can be equal is if there
+ * are no active readers using this index.
*/
- if (srcu_readers_active_idx(sp, idx) != 0)
- return false;
-
- /*
- * The remainder of this function is the validation step.
- * The following smp_mb() D pairs with the smp_mb() C in
- * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
- * by srcu_readers_active_idx() above, then any destructive
- * operation performed after the grace period will happen after
- * the corresponding SRCU read-side critical section.
- *
- * Note that there can be at most NR_CPUS worth of readers using
- * the old index, which is not enough to overflow even a 32-bit
- * integer. (Yes, this does mean that systems having more than
- * a billion or so CPUs need to be 64-bit systems.) Therefore,
- * the sum of the ->seq[] counters cannot possibly overflow.
- * Therefore, the only way that the return values of the two
- * calls to srcu_readers_seq_idx() can be equal is if there were
- * no increments of the corresponding rank of ->seq[] counts
- * in the interim. But the missed-increment scenario laid out
- * above includes an increment of the ->seq[] counter by
- * the corresponding __srcu_read_lock(). Therefore, if this
- * scenario occurs, the return values from the two calls to
- * srcu_readers_seq_idx() will differ, and thus the validation
- * step below suffices.
- */
- smp_mb(); /* D */
-
- return srcu_readers_seq_idx(sp, idx) == seq;
+ return srcu_readers_lock_idx(sp, idx) == unlocks;
}
/**
@@ -266,8 +230,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
unsigned long sum = 0;
for_each_possible_cpu(cpu) {
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[0]);
+ sum += READ_ONCE(cpuc->lock_count[1]);
+ sum -= READ_ONCE(cpuc->unlock_count[0]);
+ sum -= READ_ONCE(cpuc->unlock_count[1]);
}
return sum;
}
@@ -298,9 +266,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
int idx;
idx = READ_ONCE(sp->completed) & 0x1;
- __this_cpu_inc(sp->per_cpu_ref->c[idx]);
+ __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
smp_mb(); /* B */ /* Avoid leaking the critical section. */
- __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
return idx;
}
EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -314,7 +281,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
void __srcu_read_unlock(struct srcu_struct *sp, int idx)
{
smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_dec(sp->per_cpu_ref->c[idx]);
+ this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
}
EXPORT_SYMBOL_GPL(__srcu_read_unlock);
@@ -349,7 +316,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
/*
* Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->c[] and ->seq[] arrays. This allows
+ * use the other rank of the ->(un)lock_count[] arrays. This allows
* us to wait for pre-existing readers in a starvation-free manner.
*/
static void srcu_flip(struct srcu_struct *sp)
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts Paul E. McKenney
@ 2017-01-15 22:42 ` Paul E. McKenney
2017-01-23 8:38 ` Lance Roy
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
3 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 22:42 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Paul E. McKenney
If a process invokes synchronize_srcu(), is delayed just the right amount
of time, and thus does not sleep when waiting for the grace period to
complete, there is no ordering between the end of the grace period and
the code following the synchronize_srcu(). Similarly, there can be a
lack of ordering between the end of the SRCU grace period and callback
invocation.
This commit adds the necessary ordering.
Reported-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcupdate.h | 12 ++++++++++++
kernel/rcu/srcu.c | 5 +++++
kernel/rcu/tree.h | 12 ------------
3 files changed, 17 insertions(+), 12 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 01f71e1d2e94..6ade6a52d9d4 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1161,5 +1161,17 @@ do { \
ftrace_dump(oops_dump_mode); \
} while (0)
+/*
+ * Place this after a lock-acquisition primitive to guarantee that
+ * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
+ * if the UNLOCK and LOCK are executed by the same CPU or if the
+ * UNLOCK and LOCK operate on the same lock variable.
+ */
+#ifdef CONFIG_PPC
+#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
+#else /* #ifdef CONFIG_PPC */
+#define smp_mb__after_unlock_lock() do { } while (0)
+#endif /* #else #ifdef CONFIG_PPC */
+
#endif /* __LINUX_RCUPDATE_H */
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index ddabf5fbf562..f2abfbae258c 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -359,6 +359,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
head->next = NULL;
head->func = func;
spin_lock_irqsave(&sp->queue_lock, flags);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
rcu_batch_queue(&sp->batch_queue, head);
if (!sp->running) {
sp->running = true;
@@ -392,6 +393,7 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
head->next = NULL;
head->func = wakeme_after_rcu;
spin_lock_irq(&sp->queue_lock);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
if (!sp->running) {
/* steal the processing owner */
sp->running = true;
@@ -413,6 +415,8 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
if (!done)
wait_for_completion(&rcu.completion);
+
+ smp_mb(); /* Caller's later accesses after GP. */
}
/**
@@ -587,6 +591,7 @@ static void srcu_invoke_callbacks(struct srcu_struct *sp)
int i;
struct rcu_head *head;
+ smp_mb(); /* Callback accesses after GP. */
for (i = 0; i < SRCU_CALLBACK_BATCH; i++) {
head = rcu_batch_dequeue(&sp->batch_done);
if (!head)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..abcc25bdcb29 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -688,18 +688,6 @@ static inline void rcu_nocb_q_lengths(struct rcu_data *rdp, long *ql, long *qll)
#endif /* #ifdef CONFIG_RCU_TRACE */
/*
- * Place this after a lock-acquisition primitive to guarantee that
- * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
- * if the UNLOCK and LOCK are executed by the same CPU or if the
- * UNLOCK and LOCK operate on the same lock variable.
- */
-#ifdef CONFIG_PPC
-#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
-#else /* #ifdef CONFIG_PPC */
-#define smp_mb__after_unlock_lock() do { } while (0)
-#endif /* #else #ifdef CONFIG_PPC */
-
-/*
* Wrappers for the rcu_node::lock acquire and release.
*
* Because the rcu_nodes form a tree, the tree traversal locking will observe
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v2 tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
@ 2017-01-15 22:42 ` Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
3 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-15 22:42 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
This commit creates a formal/srcu-cbmc directory containing scripts that
pull SRCU in from the source code, filter it to remove things that CBMC
cannot handle, and run a series of verifications on it. This has a number
of shortcomings:
1. It does not yet hook into the upper-level self-test Makefiles.
2. It tests only a single scenario, store buffering.
3. There is no gcc-based syntax-error prefiltering.
Nevertheless, it does fully verify a piece of SRCU under a moderately
weak memory model (PSO).
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
.../rcutorture/formal/srcu-cbmc/.gitignore | 1 +
.../selftests/rcutorture/formal/srcu-cbmc/Makefile | 16 +
.../formal/srcu-cbmc/empty_includes/linux/delay.h | 0
.../formal/srcu-cbmc/empty_includes/linux/export.h | 0
.../formal/srcu-cbmc/empty_includes/linux/mutex.h | 0
.../formal/srcu-cbmc/empty_includes/linux/percpu.h | 0
.../srcu-cbmc/empty_includes/linux/preempt.h | 0
.../srcu-cbmc/empty_includes/linux/rcupdate.h | 0
.../formal/srcu-cbmc/empty_includes/linux/sched.h | 0
.../formal/srcu-cbmc/empty_includes/linux/smp.h | 0
.../srcu-cbmc/empty_includes/linux/workqueue.h | 0
.../srcu-cbmc/empty_includes/uapi/linux/types.h | 0
.../formal/srcu-cbmc/include/linux/.gitignore | 1 +
.../formal/srcu-cbmc/include/linux/kconfig.h | 1 +
.../formal/srcu-cbmc/include/linux/types.h | 155 +++++++++
.../rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 +++++++++++++++++++++
.../rcutorture/formal/srcu-cbmc/src/assume.h | 16 +
.../rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +++
.../rcutorture/formal/srcu-cbmc/src/bug_on.h | 13 +
.../formal/srcu-cbmc/src/combined_source.c | 13 +
.../rcutorture/formal/srcu-cbmc/src/config.h | 27 ++
.../rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31 ++
.../rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33 ++
.../rcutorture/formal/srcu-cbmc/src/locks.h | 220 ++++++++++++
.../rcutorture/formal/srcu-cbmc/src/misc.c | 11 +
.../rcutorture/formal/srcu-cbmc/src/misc.h | 58 ++++
.../rcutorture/formal/srcu-cbmc/src/percpu.h | 92 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.c | 78 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.h | 58 ++++
.../formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +++
.../rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++++++
.../srcu-cbmc/tests/store_buffering/.gitignore | 1 +
.../srcu-cbmc/tests/store_buffering/Makefile | 11 +
.../tests/store_buffering/assert_end.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force2.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force3.fail | 1 +
.../srcu-cbmc/tests/store_buffering/main.pass | 0
.../formal/srcu-cbmc/tests/store_buffering/test.c | 72 ++++
.../formal/srcu-cbmc/tests/test_script.sh | 102 ++++++
40 files changed, 1582 insertions(+)
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
new file mode 100644
index 000000000000..712a3d41a325
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
@@ -0,0 +1 @@
+srcu.c
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
new file mode 100644
index 000000000000..16b01559fa55
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
@@ -0,0 +1,16 @@
+all: srcu.c store_buffering
+
+LINUX_SOURCE = ../../../../../..
+
+modified_srcu_input = $(LINUX_SOURCE)/include/linux/srcu.h \
+ $(LINUX_SOURCE)/kernel/rcu/srcu.c
+
+modified_srcu_output = include/linux/srcu.h srcu.c
+
+include/linux/srcu.h: srcu.c
+
+srcu.c: modify_srcu.awk Makefile $(modified_srcu_input)
+ awk -f modify_srcu.awk $(modified_srcu_input) $(modified_srcu_output)
+
+store_buffering:
+ @cd tests/store_buffering; make
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
new file mode 100644
index 000000000000..1d016e66980a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
@@ -0,0 +1 @@
+srcu.h
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
new file mode 100644
index 000000000000..f2860dd1b407
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
@@ -0,0 +1 @@
+#include <LINUX_SOURCE/linux/kconfig.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
new file mode 100644
index 000000000000..4a3d538fef12
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
@@ -0,0 +1,155 @@
+/*
+ * This header has been modifies to remove definitions of types that
+ * are defined in standard userspace headers or are problematic for some
+ * other reason.
+ */
+
+#ifndef _LINUX_TYPES_H
+#define _LINUX_TYPES_H
+
+#define __EXPORTED_HEADERS__
+#include <uapi/linux/types.h>
+
+#ifndef __ASSEMBLY__
+
+#define DECLARE_BITMAP(name, bits) \
+ unsigned long name[BITS_TO_LONGS(bits)]
+
+typedef __u32 __kernel_dev_t;
+
+/* bsd */
+typedef unsigned char u_char;
+typedef unsigned short u_short;
+typedef unsigned int u_int;
+typedef unsigned long u_long;
+
+/* sysv */
+typedef unsigned char unchar;
+typedef unsigned short ushort;
+typedef unsigned int uint;
+typedef unsigned long ulong;
+
+#ifndef __BIT_TYPES_DEFINED__
+#define __BIT_TYPES_DEFINED__
+
+typedef __u8 u_int8_t;
+typedef __s8 int8_t;
+typedef __u16 u_int16_t;
+typedef __s16 int16_t;
+typedef __u32 u_int32_t;
+typedef __s32 int32_t;
+
+#endif /* !(__BIT_TYPES_DEFINED__) */
+
+typedef __u8 uint8_t;
+typedef __u16 uint16_t;
+typedef __u32 uint32_t;
+
+/* this is a special 64bit data type that is 8-byte aligned */
+#define aligned_u64 __u64 __attribute__((aligned(8)))
+#define aligned_be64 __be64 __attribute__((aligned(8)))
+#define aligned_le64 __le64 __attribute__((aligned(8)))
+
+/**
+ * The type used for indexing onto a disc or disc partition.
+ *
+ * Linux always considers sectors to be 512 bytes long independently
+ * of the devices real block size.
+ *
+ * blkcnt_t is the type of the inode's block count.
+ */
+#ifdef CONFIG_LBDAF
+typedef u64 sector_t;
+#else
+typedef unsigned long sector_t;
+#endif
+
+/*
+ * The type of an index into the pagecache.
+ */
+#define pgoff_t unsigned long
+
+/*
+ * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * by the DMA API.
+ *
+ * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
+ * bits wide. Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
+ * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
+ * so they don't care about the size of the actual bus addresses.
+ */
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+typedef u64 dma_addr_t;
+#else
+typedef u32 dma_addr_t;
+#endif
+
+#ifdef CONFIG_PHYS_ADDR_T_64BIT
+typedef u64 phys_addr_t;
+#else
+typedef u32 phys_addr_t;
+#endif
+
+typedef phys_addr_t resource_size_t;
+
+/*
+ * This type is the placeholder for a hardware interrupt number. It has to be
+ * big enough to enclose whatever representation is used by a given platform.
+ */
+typedef unsigned long irq_hw_number_t;
+
+typedef struct {
+ int counter;
+} atomic_t;
+
+#ifdef CONFIG_64BIT
+typedef struct {
+ long counter;
+} atomic64_t;
+#endif
+
+struct list_head {
+ struct list_head *next, *prev;
+};
+
+struct hlist_head {
+ struct hlist_node *first;
+};
+
+struct hlist_node {
+ struct hlist_node *next, **pprev;
+};
+
+/**
+ * struct callback_head - callback structure for use with RCU and task_work
+ * @next: next update requests in a list
+ * @func: actual update function to call after the grace period.
+ *
+ * The struct is aligned to size of pointer. On most architectures it happens
+ * naturally due ABI requirements, but some architectures (like CRIS) have
+ * weird ABI and we need to ask it explicitly.
+ *
+ * The alignment is required to guarantee that bits 0 and 1 of @next will be
+ * clear under normal conditions -- as long as we use call_rcu(),
+ * call_rcu_bh(), call_rcu_sched(), or call_srcu() to queue callback.
+ *
+ * This guarantee is important for few reasons:
+ * - future call_rcu_lazy() will make use of lower bits in the pointer;
+ * - the structure shares storage spacer in struct page with @compound_head,
+ * which encode PageTail() in bit 0. The guarantee is needed to avoid
+ * false-positive PageTail().
+ */
+struct callback_head {
+ struct callback_head *next;
+ void (*func)(struct callback_head *head);
+} __attribute__((aligned(sizeof(void *))));
+#define rcu_head callback_head
+
+typedef void (*rcu_callback_t)(struct rcu_head *head);
+typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
+
+/* clocksource cycle base type */
+typedef u64 cycle_t;
+
+#endif /* __ASSEMBLY__ */
+#endif /* _LINUX_TYPES_H */
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
new file mode 100755
index 000000000000..8ff89043d0a9
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
@@ -0,0 +1,375 @@
+#!/bin/awk -f
+
+# Modify SRCU for formal verification. The first argument should be srcu.h and
+# the second should be srcu.c. Outputs modified srcu.h and srcu.c into the
+# current directory.
+
+BEGIN {
+ if (ARGC != 5) {
+ print "Usange: input.h input.c output.h output.c" > "/dev/stderr";
+ exit 1;
+ }
+ h_output = ARGV[3];
+ c_output = ARGV[4];
+ ARGC = 3;
+
+ # Tokenize using FS and not RS as FS supports regular expressions. Each
+ # record is one line of source, except that backslashed lines are
+ # combined. Comments are treated as field separators, as are quotes.
+ quote_regexp="\"([^\\\\\"]|\\\\.)*\"";
+ comment_regexp="\\/\\*([^*]|\\*+[^*/])*\\*\\/|\\/\\/.*(\n|$)";
+ FS="([ \\\\\t\n\v\f;,.=(){}+*/<>&|^-]|\\[|\\]|" comment_regexp "|" quote_regexp ")+";
+
+ inside_srcu_struct = 0;
+ inside_srcu_init_def = 0;
+ srcu_init_param_name = "";
+ in_macro = 0;
+ brace_nesting = 0;
+ paren_nesting = 0;
+
+ # Allow the manipulation of the last field separator after has been
+ # seen.
+ last_fs = "";
+ # Whether the last field separator was intended to be output.
+ last_fs_print = 0;
+
+ # rcu_batches stores the initialization for each instance of struct
+ # rcu_batch
+
+ in_comment = 0;
+
+ outputfile = "";
+}
+
+{
+ prev_outputfile = outputfile;
+ if (FILENAME ~ /\.h$/) {
+ outputfile = h_output;
+ if (FNR != NR) {
+ print "Incorrect file order" > "/dev/stderr";
+ exit 1;
+ }
+ }
+ else
+ outputfile = c_output;
+
+ if (prev_outputfile && outputfile != prev_outputfile) {
+ new_outputfile = outputfile;
+ outputfile = prev_outputfile;
+ update_fieldsep("", 0);
+ outputfile = new_outputfile;
+ }
+}
+
+# Combine the next line into $0.
+function combine_line() {
+ ret = getline next_line;
+ if (ret == 0) {
+ # Don't allow two consecutive getlines at the end of the file
+ if (eof_found) {
+ print "Error: expected more input." > "/dev/stderr";
+ exit 1;
+ } else {
+ eof_found = 1;
+ }
+ } else if (ret == -1) {
+ print "Error reading next line of file" FILENAME > "/dev/stderr";
+ exit 1;
+ }
+ $0 = $0 "\n" next_line;
+}
+
+# Combine backslashed lines and multiline comments.
+function combine_backslashes() {
+ while (/\\$|\/\*([^*]|\*+[^*\/])*\**$/) {
+ combine_line();
+ }
+}
+
+function read_line() {
+ combine_line();
+ combine_backslashes();
+}
+
+# Print out field separators and update variables that depend on them. Only
+# print if p is true. Call with sep="" and p=0 to print out the last field
+# separator.
+function update_fieldsep(sep, p) {
+ # Count braces
+ sep_tmp = sep;
+ gsub(quote_regexp "|" comment_regexp, "", sep_tmp);
+ while (1)
+ {
+ if (sub("[^{}()]*\\{", "", sep_tmp)) {
+ brace_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\}", "", sep_tmp)) {
+ brace_nesting--;
+ if (brace_nesting < 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+ if (sub("[^{}()]*\\(", "", sep_tmp)) {
+ paren_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\)", "", sep_tmp)) {
+ paren_nesting--;
+ if (paren_nesting < 0) {
+ print "Unbalanced parenthesis!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+
+ break;
+ }
+
+ if (last_fs_print)
+ printf("%s", last_fs) > outputfile;
+ last_fs = sep;
+ last_fs_print = p;
+}
+
+# Shifts the fields down by n positions. Calls next if there are no more. If p
+# is true then print out field separators.
+function shift_fields(n, p) {
+ do {
+ if (match($0, FS) > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), p);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ } else {
+ update_fieldsep("", 0);
+ print "" > outputfile;
+ next;
+ }
+ } while (--n > 0);
+}
+
+# Shifts and prints the first n fields.
+function print_fields(n) {
+ do {
+ update_fieldsep("", 0);
+ printf("%s", $1) > outputfile;
+ shift_fields(1, 1);
+ } while (--n > 0);
+}
+
+{
+ combine_backslashes();
+}
+
+# Print leading FS
+{
+ if (match($0, "^(" FS ")+") > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), 1);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ }
+}
+
+# Parse the line.
+{
+ while (NF > 0) {
+ if ($1 == "struct" && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_struct &&
+ brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "srcu_struct" &&
+ $0 ~ "^struct(" FS ")+srcu_struct(" FS ")+\\{") {
+ inside_srcu_struct = 1;
+ print_fields(2);
+ continue;
+ }
+ if (inside_srcu_struct && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_struct = 0;
+ update_fieldsep("", 0);
+ for (name in rcu_batches)
+ print "extern struct rcu_batch " name ";" > outputfile;
+ }
+
+ if (inside_srcu_struct && $1 == "struct" && $2 == "rcu_batch") {
+ # Move rcu_batches outside of the struct.
+ rcu_batches[$3] = "";
+ shift_fields(3, 1);
+ sub(/;[[:space:]]*$/, "", last_fs);
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_init_def &&
+ $1 == "#define" && $2 == "__SRCU_STRUCT_INIT") {
+ inside_srcu_init_def = 1;
+ srcu_init_param_name = $3;
+ in_macro = 1;
+ print_fields(3);
+ continue;
+ }
+ if (inside_srcu_init_def && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_init_def = 0;
+ in_macro = 0;
+ continue;
+ }
+
+ if (inside_srcu_init_def && brace_nesting == 1 &&
+ paren_nesting == 0 && last_fs ~ /\.[[:space:]]*$/ &&
+ $1 ~ /^[[:alnum:]_]+$/) {
+ name = $1;
+ if (name in rcu_batches) {
+ # Remove the dot.
+ sub(/\.[[:space:]]*$/, "", last_fs);
+
+ old_record = $0;
+ do
+ shift_fields(1, 0);
+ while (last_fs !~ /,/ || paren_nesting > 0);
+ end_loc = length(old_record) - length($0);
+ end_loc += index(last_fs, ",") - length(last_fs);
+
+ last_fs = substr(last_fs, index(last_fs, ",") + 1);
+ last_fs_print = 1;
+
+ match(old_record, "^"name"("FS")+=");
+ start_loc = RSTART + RLENGTH;
+
+ len = end_loc - start_loc;
+ initializer = substr(old_record, start_loc, len);
+ gsub(srcu_init_param_name "\\.", "", initializer);
+ rcu_batches[name] = initializer;
+ continue;
+ }
+ }
+
+ # Don't include a nonexistent file
+ if (!in_macro && $1 == "#include" && /^#include[[:space:]]+"rcu\.h"/) {
+ update_fieldsep("", 0);
+ next;
+ }
+
+ # Ignore most preprocessor stuff.
+ if (!in_macro && $1 ~ /#/) {
+ break;
+ }
+
+ if (brace_nesting > 0 && $1 ~ "^[[:alnum:]_]+$" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (brace_nesting > 0 &&
+ $0 ~ "^[[:alnum:]_]+[[:space:]]*(\\.|->)[[:space:]]*[[:alnum:]_]+" &&
+ $2 in rcu_batches) {
+ # Make uses of rcu_batches global. Somewhat unreliable.
+ shift_fields(1, 0);
+ print_fields(1);
+ continue;
+ }
+
+ if ($1 == "static" && NF < 3) {
+ read_line();
+ continue;
+ }
+ if ($1 == "static" && ($2 == "bool" && $3 == "try_check_zero" ||
+ $2 == "void" && $3 == "srcu_flip")) {
+ shift_fields(1, 1);
+ print_fields(2);
+ continue;
+ }
+
+ # Distinguish between read-side and write-side memory barriers.
+ if ($1 == "smp_mb" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (match($0, /^smp_mb[[:space:]();\/*]*[[:alnum:]]/)) {
+ barrier_letter = substr($0, RLENGTH, 1);
+ if (barrier_letter ~ /A|D/)
+ new_barrier_name = "sync_smp_mb";
+ else if (barrier_letter ~ /B|C/)
+ new_barrier_name = "rs_smp_mb";
+ else {
+ print "Unrecognized memory barrier." > "/dev/null";
+ exit 1;
+ }
+
+ shift_fields(1, 1);
+ printf("%s", new_barrier_name) > outputfile;
+ continue;
+ }
+
+ # Skip definition of rcu_synchronize, since it is already
+ # defined in misc.h. Only present in old versions of srcu.
+ if (brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "rcu_synchronize" &&
+ $0 ~ "^struct(" FS ")+rcu_synchronize(" FS ")+\\{") {
+ shift_fields(2, 0);
+ while (brace_nesting) {
+ if (NF < 2)
+ read_line();
+ shift_fields(1, 0);
+ }
+ }
+
+ # Skip definition of wakeme_after_rcu for the same reason
+ if (brace_nesting == 0 && $1 == "static" && $2 == "void" &&
+ $3 == "wakeme_after_rcu") {
+ while (NF < 5)
+ read_line();
+ shift_fields(3, 0);
+ do {
+ while (NF < 3)
+ read_line();
+ shift_fields(1, 0);
+ } while (paren_nesting || brace_nesting);
+ }
+
+ if ($1 ~ /^(unsigned|long)$/ && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ # Give srcu_batches_completed the correct type for old SRCU.
+ if (brace_nesting == 0 && $1 == "long" &&
+ $2 == "srcu_batches_completed") {
+ update_fieldsep("", 0);
+ printf("unsigned ") > outputfile;
+ print_fields(2);
+ continue;
+ }
+ if (brace_nesting == 0 && $1 == "unsigned" && $2 == "long" &&
+ $3 == "srcu_batches_completed") {
+ print_fields(3);
+ continue;
+ }
+
+ # Just print out the input code by default.
+ print_fields(1);
+ }
+ update_fieldsep("", 0);
+ print > outputfile;
+ next;
+}
+
+END {
+ update_fieldsep("", 0);
+
+ if (brace_nesting != 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+
+ # Define the rcu_batches
+ for (name in rcu_batches)
+ print "struct rcu_batch " name " = " rcu_batches[name] ";" > c_output;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
new file mode 100644
index 000000000000..a64955447995
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
@@ -0,0 +1,16 @@
+#ifndef ASSUME_H
+#define ASSUME_H
+
+/* Provide an assumption macro that can be disabled for gcc. */
+#ifdef RUN
+#define assume(x) \
+ do { \
+ /* Evaluate x to suppress warnings. */ \
+ (void) (x); \
+ } while (0)
+
+#else
+#define assume(x) __CPROVER_assume(x)
+#endif
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
new file mode 100644
index 000000000000..6687acc08e6d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
@@ -0,0 +1,41 @@
+#ifndef BARRIERS_H
+#define BARRIERS_H
+
+#define barrier() __asm__ __volatile__("" : : : "memory")
+
+#ifdef RUN
+#define smp_mb() __sync_synchronize()
+#define smp_mb__after_unlock_lock() __sync_synchronize()
+#else
+/*
+ * Copied from CBMC's implementation of __sync_synchronize(), which
+ * seems to be disabled by default.
+ */
+#define smp_mb() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#define smp_mb__after_unlock_lock() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#endif
+
+/*
+ * Allow memory barriers to be disabled in either the read or write side
+ * of SRCU individually.
+ */
+
+#ifndef NO_SYNC_SMP_MB
+#define sync_smp_mb() smp_mb()
+#else
+#define sync_smp_mb() do {} while (0)
+#endif
+
+#ifndef NO_READ_SIDE_SMP_MB
+#define rs_smp_mb() smp_mb()
+#else
+#define rs_smp_mb() do {} while (0)
+#endif
+
+#define ACCESS_ONCE(x) (*(volatile typeof(x) *) &(x))
+#define READ_ONCE(x) ACCESS_ONCE(x)
+#define WRITE_ONCE(x, val) (ACCESS_ONCE(x) = (val))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
new file mode 100644
index 000000000000..2a80e91f78e7
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
@@ -0,0 +1,13 @@
+#ifndef BUG_ON_H
+#define BUG_ON_H
+
+#include <assert.h>
+
+#define BUG() assert(0)
+#define BUG_ON(x) assert(!(x))
+
+/* Does it make sense to treat warnings as errors? */
+#define WARN() BUG()
+#define WARN_ON(x) (BUG_ON(x), false)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
new file mode 100644
index 000000000000..29eb5d2697ed
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
@@ -0,0 +1,13 @@
+#include <config.h>
+
+/* Include all source files. */
+
+#include "include_srcu.c"
+
+#include "preempt.c"
+#include "misc.c"
+
+/* Used by test.c files */
+#include <pthread.h>
+#include <stdlib.h>
+#include <linux/srcu.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
new file mode 100644
index 000000000000..a60038aeea7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
@@ -0,0 +1,27 @@
+/* "Cheater" definitions based on restricted Kconfig choices. */
+
+#undef CONFIG_TINY_RCU
+#undef __CHECKER__
+#undef CONFIG_DEBUG_LOCK_ALLOC
+#undef CONFIG_DEBUG_OBJECTS_RCU_HEAD
+#undef CONFIG_HOTPLUG_CPU
+#undef CONFIG_MODULES
+#undef CONFIG_NO_HZ_FULL_SYSIDLE
+#undef CONFIG_PREEMPT_COUNT
+#undef CONFIG_PREEMPT_RCU
+#undef CONFIG_PROVE_RCU
+#undef CONFIG_RCU_NOCB_CPU
+#undef CONFIG_RCU_NOCB_CPU_ALL
+#undef CONFIG_RCU_STALL_COMMON
+#undef CONFIG_RCU_TRACE
+#undef CONFIG_RCU_USER_QS
+#undef CONFIG_TASKS_RCU
+#define CONFIG_TREE_RCU
+
+#define CONFIG_GENERIC_ATOMIC64
+
+#if NR_CPUS > 1
+#define CONFIG_SMP
+#else
+#undef CONFIG_SMP
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
new file mode 100644
index 000000000000..5ec582a53018
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
@@ -0,0 +1,31 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#define synchronize_srcu(sp) synchronize_srcu_original(sp)
+#endif
+
+#include <srcu.c>
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#undef synchronize_srcu
+
+#include "simple_sync_srcu.c"
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
new file mode 100644
index 000000000000..3aad63917858
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
@@ -0,0 +1,33 @@
+#ifndef INT_TYPEDEFS_H
+#define INT_TYPEDEFS_H
+
+#include <inttypes.h>
+
+typedef int8_t s8;
+typedef uint8_t u8;
+typedef int16_t s16;
+typedef uint16_t u16;
+typedef int32_t s32;
+typedef uint32_t u32;
+typedef int64_t s64;
+typedef uint64_t u64;
+
+typedef int8_t __s8;
+typedef uint8_t __u8;
+typedef int16_t __s16;
+typedef uint16_t __u16;
+typedef int32_t __s32;
+typedef uint32_t __u32;
+typedef int64_t __s64;
+typedef uint64_t __u64;
+
+#define S8_C(x) INT8_C(x)
+#define U8_C(x) UINT8_C(x)
+#define S16_C(x) INT16_C(x)
+#define U16_C(x) UINT16_C(x)
+#define S32_C(x) INT32_C(x)
+#define U32_C(x) UINT32_C(x)
+#define S64_C(x) INT64_C(x)
+#define U64_C(x) UINT64_C(x)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
new file mode 100644
index 000000000000..356004665576
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
@@ -0,0 +1,220 @@
+#ifndef LOCKS_H
+#define LOCKS_H
+
+#include <limits.h>
+#include <pthread.h>
+#include <stdbool.h>
+
+#include "assume.h"
+#include "bug_on.h"
+#include "preempt.h"
+
+int nondet_int(void);
+
+#define __acquire(x)
+#define __acquires(x)
+#define __release(x)
+#define __releases(x)
+
+/* Only use one lock mechanism. Select which one. */
+#ifdef PTHREAD_LOCK
+struct lock_impl {
+ pthread_mutex_t mutex;
+};
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_lock(&lock->mutex));
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_unlock(&lock->mutex));
+}
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+ int err = pthread_mutex_trylock(&lock->mutex);
+
+ if (!err)
+ return true;
+ else if (err == EBUSY)
+ return false;
+ BUG();
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ pthread_mutex_init(&lock->mutex, NULL);
+}
+
+#define LOCK_IMPL_INITIALIZER {.mutex = PTHREAD_MUTEX_INITIALIZER}
+
+#else /* !defined(PTHREAD_LOCK) */
+/* Spinlock that assumes that it always gets the lock immediately. */
+
+struct lock_impl {
+ bool locked;
+};
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+#ifdef RUN
+ /* TODO: Should this be a test and set? */
+ return __sync_bool_compare_and_swap(&lock->locked, false, true);
+#else
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = true;
+ __CPROVER_atomic_end();
+
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RRfence", "RWfence");
+
+ return !old_locked;
+#endif
+}
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ /*
+ * CBMC doesn't support busy waiting, so just assume that the
+ * lock is available.
+ */
+ assume(lock_impl_trylock(lock));
+
+ /*
+ * If the lock was already held by this thread then the assumption
+ * is unsatisfiable (deadlock).
+ */
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+#ifdef RUN
+ BUG_ON(!__sync_bool_compare_and_swap(&lock->locked, true, false));
+#else
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RWfence", "WWfence");
+
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = false;
+ __CPROVER_atomic_end();
+
+ BUG_ON(!old_locked);
+#endif
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ lock->locked = false;
+}
+
+#define LOCK_IMPL_INITIALIZER {.locked = false}
+
+#endif /* !defined(PTHREAD_LOCK) */
+
+/*
+ * Implement spinlocks using the lock mechanism. Wrap the lock to prevent mixing
+ * locks of different types.
+ */
+typedef struct {
+ struct lock_impl internal_lock;
+} spinlock_t;
+
+#define SPIN_LOCK_UNLOCKED {.internal_lock = LOCK_IMPL_INITIALIZER}
+#define __SPIN_LOCK_UNLOCKED(x) SPIN_LOCK_UNLOCKED
+#define DEFINE_SPINLOCK(x) spinlock_t x = SPIN_LOCK_UNLOCKED
+
+static inline void spin_lock_init(spinlock_t *lock)
+{
+ lock_impl_init(&lock->internal_lock);
+}
+
+static inline void spin_lock(spinlock_t *lock)
+{
+ /*
+ * Spin locks also need to be removed in order to eliminate all
+ * memory barriers. They are only used by the write side anyway.
+ */
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ lock_impl_lock(&lock->internal_lock);
+#endif
+}
+
+static inline void spin_unlock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ lock_impl_unlock(&lock->internal_lock);
+ preempt_enable();
+#endif
+}
+
+/* Don't bother with interrupts */
+#define spin_lock_irq(lock) spin_lock(lock)
+#define spin_unlock_irq(lock) spin_unlock(lock)
+#define spin_lock_irqsave(lock, flags) spin_lock(lock)
+#define spin_unlock_irqrestore(lock, flags) spin_unlock(lock)
+
+/*
+ * This is supposed to return an int, but I think that a bool should work as
+ * well.
+ */
+static inline bool spin_trylock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ return lock_impl_trylock(&lock->internal_lock);
+#else
+ return true;
+#endif
+}
+
+struct completion {
+ /* Hopefuly this won't overflow. */
+ unsigned int count;
+};
+
+#define COMPLETION_INITIALIZER(x) {.count = 0}
+#define DECLARE_COMPLETION(x) struct completion x = COMPLETION_INITIALIZER(x)
+#define DECLARE_COMPLETION_ONSTACK(x) DECLARE_COMPLETION(x)
+
+static inline void init_completion(struct completion *c)
+{
+ c->count = 0;
+}
+
+static inline void wait_for_completion(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_sub(&c->count, 1);
+
+ assume(prev_count);
+}
+
+static inline void complete(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_add(&c->count, 1);
+
+ BUG_ON(prev_count == UINT_MAX);
+}
+
+/* This function probably isn't very useful for CBMC. */
+static inline bool try_wait_for_completion(struct completion *c)
+{
+ BUG();
+}
+
+static inline bool completion_done(struct completion *c)
+{
+ return c->count;
+}
+
+/* TODO: Implement complete_all */
+static inline void complete_all(struct completion *c)
+{
+ BUG();
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
new file mode 100644
index 000000000000..ca892e3b2351
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
@@ -0,0 +1,11 @@
+#include <config.h>
+
+#include "misc.h"
+#include "bug_on.h"
+
+struct rcu_head;
+
+void wakeme_after_rcu(struct rcu_head *head)
+{
+ BUG();
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
new file mode 100644
index 000000000000..aca50030f954
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
@@ -0,0 +1,58 @@
+#ifndef MISC_H
+#define MISC_H
+
+#include "assume.h"
+#include "int_typedefs.h"
+#include "locks.h"
+
+#include <linux/types.h>
+
+/* Probably won't need to deal with bottom halves. */
+static inline void local_bh_disable(void) {}
+static inline void local_bh_enable(void) {}
+
+#define MODULE_ALIAS(X)
+#define module_param(...)
+#define EXPORT_SYMBOL_GPL(x)
+
+#define container_of(ptr, type, member) ({ \
+ const typeof(((type *)0)->member) *__mptr = (ptr); \
+ (type *)((char *)__mptr - offsetof(type, member)); \
+})
+
+#ifndef USE_SIMPLE_SYNC_SRCU
+/* Abuse udelay to make sure that busy loops terminate. */
+#define udelay(x) assume(0)
+
+#else
+
+/* The simple custom synchronize_srcu is ok with try_check_zero failing. */
+#define udelay(x) do { } while (0)
+#endif
+
+#define trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
+ do { } while (0)
+
+#define notrace
+
+/* Avoid including rcupdate.h */
+struct rcu_synchronize {
+ struct rcu_head head;
+ struct completion completion;
+};
+
+void wakeme_after_rcu(struct rcu_head *head);
+
+#define rcu_lock_acquire(a) do { } while (0)
+#define rcu_lock_release(a) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
+#define RCU_LOCKDEP_WARN(c, s) do { } while (0)
+
+/* Let CBMC non-deterministically choose switch between normal and expedited. */
+bool rcu_gp_is_normal(void);
+bool rcu_gp_is_expedited(void);
+
+/* Do the same for old versions of rcu. */
+#define rcu_expedited (rcu_gp_is_expedited())
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
new file mode 100644
index 000000000000..3de5a49de49b
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
@@ -0,0 +1,92 @@
+#ifndef PERCPU_H
+#define PERCPU_H
+
+#include <stddef.h>
+#include "bug_on.h"
+#include "preempt.h"
+
+#define __percpu
+
+/* Maximum size of any percpu data. */
+#define PERCPU_OFFSET (4 * sizeof(long))
+
+/* Ignore alignment, as CBMC doesn't care about false sharing. */
+#define alloc_percpu(type) __alloc_percpu(sizeof(type), 1)
+
+static inline void *__alloc_percpu(size_t size, size_t align)
+{
+ BUG();
+ return NULL;
+}
+
+static inline void free_percpu(void *ptr)
+{
+ BUG();
+}
+
+#define per_cpu_ptr(ptr, cpu) \
+ ((typeof(ptr)) ((char *) (ptr) + PERCPU_OFFSET * cpu))
+
+#define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1)
+#define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1)
+#define __this_cpu_sub(pcp, n) __this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+#define this_cpu_inc(pcp) this_cpu_add(pcp, 1)
+#define this_cpu_dec(pcp) this_cpu_sub(pcp, 1)
+#define this_cpu_sub(pcp, n) this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+/* Make CBMC use atomics to work around bug. */
+#ifdef RUN
+#define THIS_CPU_ADD_HELPER(ptr, x) (*(ptr) += (x))
+#else
+/*
+ * Split the atomic into a read and a write so that it has the least
+ * possible ordering.
+ */
+#define THIS_CPU_ADD_HELPER(ptr, x) \
+ do { \
+ typeof(ptr) this_cpu_add_helper_ptr = (ptr); \
+ typeof(ptr) this_cpu_add_helper_x = (x); \
+ typeof(*ptr) this_cpu_add_helper_temp; \
+ __CPROVER_atomic_begin(); \
+ this_cpu_add_helper_temp = *(this_cpu_add_helper_ptr); \
+ __CPROVER_atomic_end(); \
+ this_cpu_add_helper_temp += this_cpu_add_helper_x; \
+ __CPROVER_atomic_begin(); \
+ *(this_cpu_add_helper_ptr) = this_cpu_add_helper_temp; \
+ __CPROVER_atomic_end(); \
+ } while (0)
+#endif
+
+/*
+ * For some reason CBMC needs an atomic operation even though this is percpu
+ * data.
+ */
+#define __this_cpu_add(pcp, n) \
+ do { \
+ BUG_ON(preemptible()); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), thread_cpu_id), \
+ (typeof(pcp)) (n)); \
+ } while (0)
+
+#define this_cpu_add(pcp, n) \
+ do { \
+ int this_cpu_add_impl_cpu = get_cpu(); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), this_cpu_add_impl_cpu), \
+ (typeof(pcp)) (n)); \
+ put_cpu(); \
+ } while (0)
+
+/*
+ * This will cause a compiler warning because of the cast from char[][] to
+ * type*. This will cause a compile time error if type is too big.
+ */
+#define DEFINE_PER_CPU(type, name) \
+ char name[NR_CPUS][PERCPU_OFFSET]; \
+ typedef char percpu_too_big_##name \
+ [sizeof(type) > PERCPU_OFFSET ? -1 : 1]
+
+#define for_each_possible_cpu(cpu) \
+ for ((cpu) = 0; (cpu) < NR_CPUS; ++(cpu))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
new file mode 100644
index 000000000000..4f1b068e9b7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
@@ -0,0 +1,78 @@
+#include <config.h>
+
+#include "preempt.h"
+
+#include "assume.h"
+#include "locks.h"
+
+/* Support NR_CPUS of at most 64 */
+#define CPU_PREEMPTION_LOCKS_INIT0 LOCK_IMPL_INITIALIZER
+#define CPU_PREEMPTION_LOCKS_INIT1 \
+ CPU_PREEMPTION_LOCKS_INIT0, CPU_PREEMPTION_LOCKS_INIT0
+#define CPU_PREEMPTION_LOCKS_INIT2 \
+ CPU_PREEMPTION_LOCKS_INIT1, CPU_PREEMPTION_LOCKS_INIT1
+#define CPU_PREEMPTION_LOCKS_INIT3 \
+ CPU_PREEMPTION_LOCKS_INIT2, CPU_PREEMPTION_LOCKS_INIT2
+#define CPU_PREEMPTION_LOCKS_INIT4 \
+ CPU_PREEMPTION_LOCKS_INIT3, CPU_PREEMPTION_LOCKS_INIT3
+#define CPU_PREEMPTION_LOCKS_INIT5 \
+ CPU_PREEMPTION_LOCKS_INIT4, CPU_PREEMPTION_LOCKS_INIT4
+
+/*
+ * Simulate disabling preemption by locking a particular cpu. NR_CPUS
+ * should be the actual number of cpus, not just the maximum.
+ */
+struct lock_impl cpu_preemption_locks[NR_CPUS] = {
+ CPU_PREEMPTION_LOCKS_INIT0
+#if (NR_CPUS - 1) & 1
+ , CPU_PREEMPTION_LOCKS_INIT0
+#endif
+#if (NR_CPUS - 1) & 2
+ , CPU_PREEMPTION_LOCKS_INIT1
+#endif
+#if (NR_CPUS - 1) & 4
+ , CPU_PREEMPTION_LOCKS_INIT2
+#endif
+#if (NR_CPUS - 1) & 8
+ , CPU_PREEMPTION_LOCKS_INIT3
+#endif
+#if (NR_CPUS - 1) & 16
+ , CPU_PREEMPTION_LOCKS_INIT4
+#endif
+#if (NR_CPUS - 1) & 32
+ , CPU_PREEMPTION_LOCKS_INIT5
+#endif
+};
+
+#undef CPU_PREEMPTION_LOCKS_INIT0
+#undef CPU_PREEMPTION_LOCKS_INIT1
+#undef CPU_PREEMPTION_LOCKS_INIT2
+#undef CPU_PREEMPTION_LOCKS_INIT3
+#undef CPU_PREEMPTION_LOCKS_INIT4
+#undef CPU_PREEMPTION_LOCKS_INIT5
+
+__thread int thread_cpu_id;
+__thread int preempt_disable_count;
+
+void preempt_disable(void)
+{
+ BUG_ON(preempt_disable_count < 0 || preempt_disable_count == INT_MAX);
+
+ if (preempt_disable_count++)
+ return;
+
+ thread_cpu_id = nondet_int();
+ assume(thread_cpu_id >= 0);
+ assume(thread_cpu_id < NR_CPUS);
+ lock_impl_lock(&cpu_preemption_locks[thread_cpu_id]);
+}
+
+void preempt_enable(void)
+{
+ BUG_ON(preempt_disable_count < 1);
+
+ if (--preempt_disable_count)
+ return;
+
+ lock_impl_unlock(&cpu_preemption_locks[thread_cpu_id]);
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
new file mode 100644
index 000000000000..2f95ee0e4dd5
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
@@ -0,0 +1,58 @@
+#ifndef PREEMPT_H
+#define PREEMPT_H
+
+#include <stdbool.h>
+
+#include "bug_on.h"
+
+/* This flag contains garbage if preempt_disable_count is 0. */
+extern __thread int thread_cpu_id;
+
+/* Support recursive preemption disabling. */
+extern __thread int preempt_disable_count;
+
+void preempt_disable(void);
+void preempt_enable(void);
+
+static inline void preempt_disable_notrace(void)
+{
+ preempt_disable();
+}
+
+static inline void preempt_enable_no_resched(void)
+{
+ preempt_enable();
+}
+
+static inline void preempt_enable_notrace(void)
+{
+ preempt_enable();
+}
+
+static inline int preempt_count(void)
+{
+ return preempt_disable_count;
+}
+
+static inline bool preemptible(void)
+{
+ return !preempt_count();
+}
+
+static inline int get_cpu(void)
+{
+ preempt_disable();
+ return thread_cpu_id;
+}
+
+static inline void put_cpu(void)
+{
+ preempt_enable();
+}
+
+static inline void might_sleep(void)
+{
+ BUG_ON(preempt_disable_count);
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
new file mode 100644
index 000000000000..ac9cbc62b411
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
@@ -0,0 +1,50 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#include <linux/srcu.h>
+
+/* Functions needed from modify_srcu.c */
+bool try_check_zero(struct srcu_struct *sp, int idx, int trycount);
+void srcu_flip(struct srcu_struct *sp);
+
+/* Simpler implementation of synchronize_srcu that ignores batching. */
+void synchronize_srcu(struct srcu_struct *sp)
+{
+ int idx;
+ /*
+ * This code assumes that try_check_zero will succeed anyway,
+ * so there is no point in multiple tries.
+ */
+ const int trycount = 1;
+
+ might_sleep();
+
+ /* Ignore the lock, as multiple writers aren't working yet anyway. */
+
+ idx = 1 ^ (sp->completed & 1);
+
+ /* For comments see srcu_advance_batches. */
+
+ assume(try_check_zero(sp, idx, trycount));
+
+ srcu_flip(sp);
+
+ assume(try_check_zero(sp, idx^1, trycount));
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
new file mode 100644
index 000000000000..e58c8dfd3e90
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
@@ -0,0 +1,102 @@
+#ifndef WORKQUEUES_H
+#define WORKQUEUES_H
+
+#include <stdbool.h>
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "int_typedefs.h"
+
+#include <linux/types.h>
+
+/* Stub workqueue implementation. */
+
+struct work_struct;
+typedef void (*work_func_t)(struct work_struct *work);
+void delayed_work_timer_fn(unsigned long __data);
+
+struct work_struct {
+/* atomic_long_t data; */
+ unsigned long data;
+
+ struct list_head entry;
+ work_func_t func;
+#ifdef CONFIG_LOCKDEP
+ struct lockdep_map lockdep_map;
+#endif
+};
+
+struct timer_list {
+ struct hlist_node entry;
+ unsigned long expires;
+ void (*function)(unsigned long);
+ unsigned long data;
+ u32 flags;
+ int slack;
+};
+
+struct delayed_work {
+ struct work_struct work;
+ struct timer_list timer;
+
+ /* target workqueue and CPU ->timer uses to queue ->work */
+ struct workqueue_struct *wq;
+ int cpu;
+};
+
+
+static inline bool schedule_work(struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool schedule_work_on(int cpu, struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_work(struct workqueue_struct *wq,
+ struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_delayed_work(struct workqueue_struct *wq,
+ struct delayed_work *dwork,
+ unsigned long delay)
+{
+ BUG();
+ return true;
+}
+
+#define INIT_WORK(w, f) \
+ do { \
+ (w)->data = 0; \
+ (w)->func = (f); \
+ } while (0)
+
+#define INIT_DELAYED_WORK(w, f) INIT_WORK(&(w)->work, (f))
+
+#define __WORK_INITIALIZER(n, f) { \
+ .data = 0, \
+ .entry = { &(n).entry, &(n).entry }, \
+ .func = f \
+ }
+
+/* Don't bother initializing timer. */
+#define __DELAYED_WORK_INITIALIZER(n, f, tflags) { \
+ .work = __WORK_INITIALIZER((n).work, (f)), \
+ }
+
+#define DECLARE_WORK(n, f) \
+ struct workqueue_struct n = __WORK_INITIALIZER
+
+#define DECLARE_DELAYED_WORK(n, f) \
+ struct delayed_work n = __DELAYED_WORK_INITIALIZER(n, f, 0)
+
+#define system_power_efficient_wq ((struct workqueue_struct *) NULL)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
new file mode 100644
index 000000000000..f47cb2045f13
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
@@ -0,0 +1 @@
+*.out
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
new file mode 100644
index 000000000000..3a3aee149225
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
@@ -0,0 +1,11 @@
+CBMC_FLAGS = -I../.. -I../../src -I../../include -I../../empty_includes -32 -pointer-check -mm pso
+
+all:
+ for i in ./*.pass; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-pass $$i > $$i.out 2>&1 ; \
+ done
+ for i in ./*.fail; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-fail $$i > $$i.out 2>&1 ; \
+ done
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
new file mode 100644
index 000000000000..40c8075919d1
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DASSERT_END"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
new file mode 100644
index 000000000000..ada5baf0b60d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
new file mode 100644
index 000000000000..8fe00c8db466
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_2"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
new file mode 100644
index 000000000000..612ed6772844
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_3"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
new file mode 100644
index 000000000000..470b1105a112
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
@@ -0,0 +1,72 @@
+#include <src/combined_source.c>
+
+int x;
+int y;
+
+int __unbuffered_tpr_x;
+int __unbuffered_tpr_y;
+
+DEFINE_SRCU(ss);
+
+void rcu_reader(void)
+{
+ int idx;
+
+#ifndef FORCE_FAILURE_3
+ idx = srcu_read_lock(&ss);
+#endif
+ might_sleep();
+
+ __unbuffered_tpr_y = READ_ONCE(y);
+#ifdef FORCE_FAILURE
+ srcu_read_unlock(&ss, idx);
+ idx = srcu_read_lock(&ss);
+#endif
+ WRITE_ONCE(x, 1);
+
+#ifndef FORCE_FAILURE_3
+ srcu_read_unlock(&ss, idx);
+#endif
+ might_sleep();
+}
+
+void *thread_update(void *arg)
+{
+ WRITE_ONCE(y, 1);
+#ifndef FORCE_FAILURE_2
+ synchronize_srcu(&ss);
+#endif
+ might_sleep();
+ __unbuffered_tpr_x = READ_ONCE(x);
+
+ return NULL;
+}
+
+void *thread_process_reader(void *arg)
+{
+ rcu_reader();
+
+ return NULL;
+}
+
+int main(int argc, char *argv[])
+{
+ pthread_t tu;
+ pthread_t tpr;
+
+ if (pthread_create(&tu, NULL, thread_update, NULL))
+ abort();
+ if (pthread_create(&tpr, NULL, thread_process_reader, NULL))
+ abort();
+ if (pthread_join(tu, NULL))
+ abort();
+ if (pthread_join(tpr, NULL))
+ abort();
+ assert(__unbuffered_tpr_y != 0 || __unbuffered_tpr_x != 0);
+
+#ifdef ASSERT_END
+ assert(0);
+#endif
+
+ return 0;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
new file mode 100755
index 000000000000..d1545972a0fa
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+
+# This script expects a mode (either --should-pass or --should-fail) followed by
+# an input file. The script uses the following environment variables. The test C
+# source file is expected to be named test.c in the directory containing the
+# input file.
+#
+# CBMC: The command to run CBMC. Default: cbmc
+# CBMC_FLAGS: Additional flags to pass to CBMC
+# NR_CPUS: Number of cpus to run tests with. Default specified by the test
+# SYNC_SRCU_MODE: Choose implementation of synchronize_srcu. Defaults to simple.
+# kernel: Version included in the linux kernel source.
+# simple: Use try_check_zero directly.
+#
+# The input file is a script that is sourced by this file. It can define any of
+# the following variables to configure the test.
+#
+# test_cbmc_options: Extra options to pass to CBMC.
+# min_cpus_fail: Minimum number of CPUs (NR_CPUS) for verification to fail.
+# The test is expected to pass if it is run with fewer. (Only
+# useful for .fail files)
+# default_cpus: Quantity of CPUs to use for the test, if not specified on the
+# command line. Default: Larger of 2 and MIN_CPUS_FAIL.
+
+set -e
+
+if test "$#" -ne 2; then
+ echo "Expected one option followed by an input file" 1>&2
+ exit 99
+fi
+
+if test "x$1" = "x--should-pass"; then
+ should_pass="yes"
+elif test "x$1" = "x--should-fail"; then
+ should_pass="no"
+else
+ echo "Unrecognized argument '$1'" 1>&2
+
+ # Exit code 99 indicates a hard error.
+ exit 99
+fi
+
+CBMC=${CBMC:-cbmc}
+
+SYNC_SRCU_MODE=${SYNC_SRCU_MODE:-simple}
+
+case ${SYNC_SRCU_MODE} in
+kernel) sync_srcu_mode_flags="" ;;
+simple) sync_srcu_mode_flags="-DUSE_SIMPLE_SYNC_SRCU" ;;
+
+*)
+ echo "Unrecognized argument '${SYNC_SRCU_MODE}'" 1>&2
+ exit 99
+ ;;
+esac
+
+min_cpus_fail=1
+
+c_file=`dirname "$2"`/test.c
+
+# Source the input file.
+. $2
+
+if test ${min_cpus_fail} -gt 2; then
+ default_default_cpus=${min_cpus_fail}
+else
+ default_default_cpus=2
+fi
+default_cpus=${default_cpus:-${default_default_cpus}}
+cpus=${NR_CPUS:-${default_cpus}}
+
+# Check if there are two few cpus to make the test fail.
+if test $cpus -lt ${min_cpus_fail:-0}; then
+ should_pass="yes"
+fi
+
+cbmc_opts="-DNR_CPUS=${cpus} ${sync_srcu_mode_flags} ${test_cbmc_options} ${CBMC_FLAGS}"
+
+echo "Running CBMC: ${CBMC} ${cbmc_opts} ${c_file}"
+if ${CBMC} ${cbmc_opts} "${c_file}"; then
+ # Verification successful. Make sure that it was supposed to verify.
+ test "x${should_pass}" = xyes
+else
+ cbmc_exit_status=$?
+
+ # An exit status of 10 indicates a failed verification.
+ # (see cbmc_parse_optionst::do_bmc in the CBMC source code)
+ if test ${cbmc_exit_status} -eq 10 && test "x${should_pass}" = xno; then
+ :
+ else
+ echo "CBMC returned ${cbmc_exit_status} exit status" 1>&2
+
+ # Parse errors have exit status 6. Any other type of error
+ # should be considered a hard error.
+ if test ${cbmc_exit_status} -ne 6 && \
+ test ${cbmc_exit_status} -ne 10; then
+ exit 99
+ else
+ exit 1
+ fi
+ fi
+fi
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 19:45 ` Paul E. McKenney
@ 2017-01-16 6:56 ` Ingo Molnar
0 siblings, 0 replies; 43+ messages in thread
From: Ingo Molnar @ 2017-01-16 6:56 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani, will.deacon, boqun.feng,
linuxppc-dev, Peter Zijlstra
* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Jan 15, 2017 at 10:40:58AM +0100, Ingo Molnar wrote:
> >
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> >
> > > > [sounds of rummaging around in the Git tree]
> > > >
> > > > I found this commit of yours from ancient history (more than a year ago!):
> > > >
> > > > commit 12d560f4ea87030667438a169912380be00cea4b
> > > > Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > Date: Tue Jul 14 18:35:23 2015 -0700
> > > >
> > > > rcu,locking: Privatize smp_mb__after_unlock_lock()
> > > >
> > > > RCU is the only thing that uses smp_mb__after_unlock_lock(), and is
> > > > likely the only thing that ever will use it, so this commit makes this
> > > > macro private to RCU.
> > > >
> > > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > Cc: Will Deacon <will.deacon@arm.com>
> > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > > > Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>
> > > >
> > > > So I concur and I'm fine with your patch - or with the status quo code as well.
> > >
> > > I already have the patch queued, so how about I keep it if I get an ack
> > > from the powerpc guys and drop it otherwise?
> >
> > Yeah, sounds good! Your patch made me look up 'RelAcq' so it has documentation
> > value as well ;-)
>
> ;-) ;-) ;-)
>
> Looking forward, my guess would be that if some other code needs
> smp_mb__after_unlock_lock() or if some other architecture needs
> non-smb_mb() special handling, I should consider making it work the
> same as smp_mb__after_atomic() and friends. Does that seem like a
> reasonable thought?
Yeah, absolutely - it's just that the pattern triggered the 'this looks a bit too
specialized' response in me, but after seeing the details (again ...) I agree that
this time is different!
Thanks,
Ingo
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-14 21:41 ` Paul E. McKenney
2017-01-15 7:11 ` Ingo Molnar
@ 2017-01-23 8:12 ` Michael Ellerman
2017-01-24 2:45 ` Paul E. McKenney
1 sibling, 1 reply; 43+ messages in thread
From: Michael Ellerman @ 2017-01-23 8:12 UTC (permalink / raw)
To: paulmck, Ingo Molnar
Cc: boqun.feng, bobby.prani, peterz, fweisbec, dvhart, jiangshanlai,
linux-kernel, rostedt, josh, dhowells, edumazet,
mathieu.desnoyers, oleg, dipankar, will.deacon, akpm,
linuxppc-dev, tglx
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> writes:
> On Sat, Jan 14, 2017 at 11:54:17AM -0800, Paul E. McKenney wrote:
>> On Sat, Jan 14, 2017 at 10:35:50AM +0100, Ingo Molnar wrote:
>> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> [ . . . ]
>
>> > > + */
>> > > +#ifdef CONFIG_PPC
>> > > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
>> > > +#else /* #ifdef CONFIG_PPC */
>> > > +#define smp_mb__after_unlock_lock() do { } while (0)
>> > > +#endif /* #else #ifdef CONFIG_PPC */
>> >
>> > Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
>> > #ifdefs into generic headers is generally frowned upon.
>> >
>> > The canonical approach would be either to define a helper Kconfig variable that
>> > can be set by PPC (but other architectures don't need to set it), or to expose a
>> > suitable macro (function) for architectures to define in their barrier.h arch
>> > header file.
>>
>> Very well, I will add a separate commit for this. 4.11 OK?
>
> Does the patch below seem reasonable?
>
> Thanx, Paul
>
> ------------------------------------------------------------------------
>
> commit 271c0601237c41a279f975563e13837bace0df03
> Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Date: Sat Jan 14 13:32:50 2017 -0800
>
> rcu: Make arch select smp_mb__after_unlock_lock() strength
>
> The definition of smp_mb__after_unlock_lock() is currently smp_mb()
> for CONFIG_PPC and a no-op otherwise. It would be better to instead
> provide an architecture-selectable Kconfig option, and select the
> strength of smp_mb__after_unlock_lock() based on that option. This
> commit therefore creates CONFIG_ARCH_WEAK_RELACQ, has PPC select it,
> and bases the definition of smp_mb__after_unlock_lock() on this new
> CONFIG_ARCH_WEAK_RELACQ Kconfig option.
>
> Reported-by: Ingo Molnar <mingo@kernel.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: <linuxppc-dev@lists.ozlabs.org>
Personally I'd call it ARCH_WEAK_RELEASE_ACQUIRE, which is longer but
clearer I think. But it's not a big deal, so which ever you prefer.
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
cheers
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
@ 2017-01-23 8:38 ` Lance Roy
2017-01-23 19:12 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-23 8:38 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Sun, 15 Jan 2017 14:42:34 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> If a process invokes synchronize_srcu(), is delayed just the right amount
> of time, and thus does not sleep when waiting for the grace period to
> complete, there is no ordering between the end of the grace period and
> the code following the synchronize_srcu(). Similarly, there can be a
> lack of ordering between the end of the SRCU grace period and callback
> invocation.
>
> This commit adds the necessary ordering.
>
> Reported-by: Lance Roy <ldr709@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> include/linux/rcupdate.h | 12 ++++++++++++
> kernel/rcu/srcu.c | 5 +++++
> kernel/rcu/tree.h | 12 ------------
> 3 files changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 01f71e1d2e94..6ade6a52d9d4 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -1161,5 +1161,17 @@ do { \
> ftrace_dump(oops_dump_mode); \
> } while (0)
>
> +/*
> + * Place this after a lock-acquisition primitive to guarantee that
> + * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
> + * if the UNLOCK and LOCK are executed by the same CPU or if the
> + * UNLOCK and LOCK operate on the same lock variable.
> + */
> +#ifdef CONFIG_PPC
> +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for
> lock. */ +#else /* #ifdef CONFIG_PPC */
> +#define smp_mb__after_unlock_lock() do { } while (0)
> +#endif /* #else #ifdef CONFIG_PPC */
> +
>
> #endif /* __LINUX_RCUPDATE_H */
> diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> index ddabf5fbf562..f2abfbae258c 100644
> --- a/kernel/rcu/srcu.c
> +++ b/kernel/rcu/srcu.c
> @@ -359,6 +359,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head
> *head, head->next = NULL;
> head->func = func;
> spin_lock_irqsave(&sp->queue_lock, flags);
> + smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
> rcu_batch_queue(&sp->batch_queue, head);
> if (!sp->running) {
> sp->running = true;
> @@ -392,6 +393,7 @@ static void __synchronize_srcu(struct srcu_struct *sp,
> int trycount) head->next = NULL;
> head->func = wakeme_after_rcu;
> spin_lock_irq(&sp->queue_lock);
> + smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
> if (!sp->running) {
> /* steal the processing owner */
> sp->running = true;
> @@ -413,6 +415,8 @@ static void __synchronize_srcu(struct srcu_struct *sp,
> int trycount)
> if (!done)
> wait_for_completion(&rcu.completion);
> +
> + smp_mb(); /* Caller's later accesses after GP. */
I think that this memory barrier is only necessary when done == false, as
otherwise srcu_advance_batches() should provide sufficient memory ordering.
> }
>
> /**
> @@ -587,6 +591,7 @@ static void srcu_invoke_callbacks(struct srcu_struct *sp)
> int i;
> struct rcu_head *head;
>
> + smp_mb(); /* Callback accesses after GP. */
Shouldn't srcu_advance_batches() have already run all necessary memory barriers?
> for (i = 0; i < SRCU_CALLBACK_BATCH; i++) {
> head = rcu_batch_dequeue(&sp->batch_done);
> if (!head)
> diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
> index fe98dd24adf8..abcc25bdcb29 100644
> --- a/kernel/rcu/tree.h
> +++ b/kernel/rcu/tree.h
> @@ -688,18 +688,6 @@ static inline void rcu_nocb_q_lengths(struct rcu_data
> *rdp, long *ql, long *qll) #endif /* #ifdef CONFIG_RCU_TRACE */
>
> /*
> - * Place this after a lock-acquisition primitive to guarantee that
> - * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
> - * if the UNLOCK and LOCK are executed by the same CPU or if the
> - * UNLOCK and LOCK operate on the same lock variable.
> - */
> -#ifdef CONFIG_PPC
> -#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for
> lock. */ -#else /* #ifdef CONFIG_PPC */
> -#define smp_mb__after_unlock_lock() do { } while (0)
> -#endif /* #else #ifdef CONFIG_PPC */
> -
> -/*
> * Wrappers for the rcu_node::lock acquire and release.
> *
> * Because the rcu_nodes form a tree, the tree traversal locking will observe
Thanks,
Lance
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-23 8:38 ` Lance Roy
@ 2017-01-23 19:12 ` Paul E. McKenney
2017-01-23 20:06 ` Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-23 19:12 UTC (permalink / raw)
To: Lance Roy
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, Jan 23, 2017 at 12:38:29AM -0800, Lance Roy wrote:
> On Sun, 15 Jan 2017 14:42:34 -0800
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
>
> > If a process invokes synchronize_srcu(), is delayed just the right amount
> > of time, and thus does not sleep when waiting for the grace period to
> > complete, there is no ordering between the end of the grace period and
> > the code following the synchronize_srcu(). Similarly, there can be a
> > lack of ordering between the end of the SRCU grace period and callback
> > invocation.
> >
> > This commit adds the necessary ordering.
> >
> > Reported-by: Lance Roy <ldr709@gmail.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> > include/linux/rcupdate.h | 12 ++++++++++++
> > kernel/rcu/srcu.c | 5 +++++
> > kernel/rcu/tree.h | 12 ------------
> > 3 files changed, 17 insertions(+), 12 deletions(-)
> >
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index 01f71e1d2e94..6ade6a52d9d4 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -1161,5 +1161,17 @@ do { \
> > ftrace_dump(oops_dump_mode); \
> > } while (0)
> >
> > +/*
> > + * Place this after a lock-acquisition primitive to guarantee that
> > + * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
> > + * if the UNLOCK and LOCK are executed by the same CPU or if the
> > + * UNLOCK and LOCK operate on the same lock variable.
> > + */
> > +#ifdef CONFIG_PPC
> > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for
> > lock. */ +#else /* #ifdef CONFIG_PPC */
> > +#define smp_mb__after_unlock_lock() do { } while (0)
> > +#endif /* #else #ifdef CONFIG_PPC */
> > +
> >
> > #endif /* __LINUX_RCUPDATE_H */
> > diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> > index ddabf5fbf562..f2abfbae258c 100644
> > --- a/kernel/rcu/srcu.c
> > +++ b/kernel/rcu/srcu.c
> > @@ -359,6 +359,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head
> > *head, head->next = NULL;
> > head->func = func;
> > spin_lock_irqsave(&sp->queue_lock, flags);
> > + smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
> > rcu_batch_queue(&sp->batch_queue, head);
> > if (!sp->running) {
> > sp->running = true;
> > @@ -392,6 +393,7 @@ static void __synchronize_srcu(struct srcu_struct *sp,
> > int trycount) head->next = NULL;
> > head->func = wakeme_after_rcu;
> > spin_lock_irq(&sp->queue_lock);
> > + smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
> > if (!sp->running) {
> > /* steal the processing owner */
> > sp->running = true;
> > @@ -413,6 +415,8 @@ static void __synchronize_srcu(struct srcu_struct *sp,
> > int trycount)
> > if (!done)
> > wait_for_completion(&rcu.completion);
> > +
> > + smp_mb(); /* Caller's later accesses after GP. */
>
> I think that this memory barrier is only necessary when done == false, as
> otherwise srcu_advance_batches() should provide sufficient memory ordering.
Let me make sure that I understand your rationale here.
The idea is that although srcu_readers_active_idx_check() did a full
memory barrier, this might have happened on some other CPU, which
would not have provided ordering to the current CPU in the race case
where current CPU didn't actually sleep. (This can happen where the
current task is preempted, and then is resumed just as the grace period
completes.)
Or are you concerned about some other sequence of events?
(I have moved the smp_mb() inside the "if (!done)" in the meantime.)
> > }
> >
> > /**
> > @@ -587,6 +591,7 @@ static void srcu_invoke_callbacks(struct srcu_struct *sp)
> > int i;
> > struct rcu_head *head;
> >
> > + smp_mb(); /* Callback accesses after GP. */
>
> Shouldn't srcu_advance_batches() have already run all necessary memory barriers?
It does look that way:
o process_srcu() is the only thing that invokes srcu_invoke_callbacks().
o process_srcu() invokes srcu_advance_batches() immediately before
srcu_invoke_callbacks(), so any memory barriers invoked from
srcu_advance_batches() affect process_srcu() (unlike the earlier
example where srcu_advance_batches() might be executed in the
context of some other task).
o srcu_advance_batches() unconditionally invokes try_check_zero(),
which in turn unconditionally invokes srcu_readers_active_idx_check(),
which in turn invokes smp_mb().
This smp_mb() precedes a successful check that all pre-existing
readers are done, otherwise srcu_advance_batches() won't have
returned (or won't have advanced the callbacks, which in turn
will prevent them from being invoked).
I have removed this memory barrier and added a comment.
> > for (i = 0; i < SRCU_CALLBACK_BATCH; i++) {
> > head = rcu_batch_dequeue(&sp->batch_done);
> > if (!head)
> > diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
> > index fe98dd24adf8..abcc25bdcb29 100644
> > --- a/kernel/rcu/tree.h
> > +++ b/kernel/rcu/tree.h
> > @@ -688,18 +688,6 @@ static inline void rcu_nocb_q_lengths(struct rcu_data
> > *rdp, long *ql, long *qll) #endif /* #ifdef CONFIG_RCU_TRACE */
> >
> > /*
> > - * Place this after a lock-acquisition primitive to guarantee that
> > - * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
> > - * if the UNLOCK and LOCK are executed by the same CPU or if the
> > - * UNLOCK and LOCK operate on the same lock variable.
> > - */
> > -#ifdef CONFIG_PPC
> > -#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for
> > lock. */ -#else /* #ifdef CONFIG_PPC */
> > -#define smp_mb__after_unlock_lock() do { } while (0)
> > -#endif /* #else #ifdef CONFIG_PPC */
> > -
> > -/*
> > * Wrappers for the rcu_node::lock acquire and release.
> > *
> > * Because the rcu_nodes form a tree, the tree traversal locking will observe
And thank you for your review and comments!!!
Thanx, Paul
> Thanks,
> Lance
>
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-23 19:12 ` Paul E. McKenney
@ 2017-01-23 20:06 ` Lance Roy
0 siblings, 0 replies; 43+ messages in thread
From: Lance Roy @ 2017-01-23 20:06 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, 23 Jan 2017 11:12:07 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> On Mon, Jan 23, 2017 at 12:38:29AM -0800, Lance Roy wrote:
> > On Sun, 15 Jan 2017 14:42:34 -0800
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> >
> > > @@ -413,6 +415,8 @@ static void __synchronize_srcu(struct srcu_struct *sp,
> > > int trycount)
> > > if (!done)
> > > wait_for_completion(&rcu.completion);
> > > +
> > > + smp_mb(); /* Caller's later accesses after GP. */
> >
> > I think that this memory barrier is only necessary when done == false, as
> > otherwise srcu_advance_batches() should provide sufficient memory
> > ordering.
>
> Let me make sure that I understand your rationale here.
>
> The idea is that although srcu_readers_active_idx_check() did a full
> memory barrier, this might have happened on some other CPU, which
> would not have provided ordering to the current CPU in the race case
> where current CPU didn't actually sleep. (This can happen where the
> current task is preempted, and then is resumed just as the grace period
> completes.)
>
> Or are you concerned about some other sequence of events?
Yes, the problem only occurs when the only full memory barrier is executed on a
different CPU.
> (I have moved the smp_mb() inside the "if (!done)" in the meantime.)
Thanks.
> > > @@ -587,6 +591,7 @@ static void srcu_invoke_callbacks(struct srcu_struct
> > > *sp) int i;
> > > struct rcu_head *head;
> > >
> > > + smp_mb(); /* Callback accesses after GP. */
> >
> > Shouldn't srcu_advance_batches() have already run all necessary memory
> > barriers?
>
> It does look that way:
>
> o process_srcu() is the only thing that invokes
> srcu_invoke_callbacks().
>
> o process_srcu() invokes srcu_advance_batches() immediately before
> srcu_invoke_callbacks(), so any memory barriers invoked from
> srcu_advance_batches() affect process_srcu() (unlike the earlier
> example where srcu_advance_batches() might be executed in the
> context of some other task).
>
> o srcu_advance_batches() unconditionally invokes try_check_zero(),
> which in turn unconditionally invokes srcu_readers_active_idx_check(),
> which in turn invokes smp_mb().
>
> This smp_mb() precedes a successful check that all pre-existing
> readers are done, otherwise srcu_advance_batches() won't have
> returned (or won't have advanced the callbacks, which in turn
> will prevent them from being invoked).
>
> I have removed this memory barrier and added a comment.
>
> And thank you for your review and comments!!!
Thanks,
Lance
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts Paul E. McKenney
@ 2017-01-23 20:17 ` Lance Roy
2017-01-23 20:17 ` [PATCH] SRCU: More efficient " Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-23 20:17 UTC (permalink / raw)
To: linux-kernel
Cc: Paul E. McKenney, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani, Lance Roy
Here is a more recent version of the patch. It has more accurate comments.
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH] SRCU: More efficient reader counts.
2017-01-23 20:17 ` Lance Roy
@ 2017-01-23 20:17 ` Lance Roy
2017-01-23 20:35 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-23 20:17 UTC (permalink / raw)
To: linux-kernel
Cc: Paul E. McKenney, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani, Lance Roy
SRCU uses two per-cpu counters: a nesting counter to count the number of
active critical sections, and a sequence counter to ensure that the nesting
counters don't change while they are being added together in
srcu_readers_active_idx_check().
This patch instead uses per-cpu lock and unlock counters. Because the both
counters only increase and srcu_readers_active_idx_check() reads the unlock
counter before the lock counter, this achieves the same end without having
to increment two different counters in srcu_read_lock(). This also saves a
smp_mb() in srcu_readers_active_idx_check().
Possible bug: There is no guarantee that the lock counter won't overflow
during srcu_readers_active_idx_check(), as there are no memory barriers
around srcu_flip() (see comment in srcu_readers_active_idx_check() for
details). However, this problem was already present before this patch.
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lance Roy <ldr709@gmail.com>
---
include/linux/srcu.h | 4 +-
kernel/rcu/rcutorture.c | 20 ++++++++-
kernel/rcu/srcu.c | 116 ++++++++++++++++++------------------------------
3 files changed, 63 insertions(+), 77 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index dc8eb63..0caea34 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -34,8 +34,8 @@
#include <linux/workqueue.h>
struct srcu_struct_array {
- unsigned long c[2];
- unsigned long seq[2];
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
};
struct rcu_batch {
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 87c5122..c3f25d1 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -564,10 +564,26 @@ static void srcu_torture_stats(void)
pr_alert("%s%s per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
+ unsigned long l0, l1;
+ unsigned long u0, u1;
long c0, c1;
+ struct srcu_struct_array *counts =
+ per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
- c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
+ u0 = counts->unlock_count[!idx];
+ u1 = counts->unlock_count[idx];
+
+ /*
+ * Make sure that a lock is always counted if the corresponding
+ * unlock is counted.
+ */
+ smp_rmb();
+
+ l0 = counts->lock_count[!idx];
+ l1 = counts->lock_count[idx];
+
+ c0 = (long)(l0 - u0);
+ c1 = (long)(l1 - u1);
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 9b9cdd5..38e9aae 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -141,34 +141,38 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
- * Returns approximate total of the readers' ->seq[] values for the
+ * Returns approximate total of the readers' ->lock_count[] values for the
* rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
+ struct srcu_struct_array *cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ t = READ_ONCE(cpu_counts->lock_count[idx]);
sum += t;
}
return sum;
}
/*
- * Returns approximate number of readers active on the specified rank
- * of the per-CPU ->c[] counters.
+ * Returns approximate total of the readers' ->unlock_count[] values for the
+ * rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
+ struct srcu_struct_array *cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ t = READ_ONCE(cpu_counts->unlock_count[idx]);
sum += t;
}
return sum;
@@ -176,79 +180,42 @@ static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
/*
* Return true if the number of pre-existing readers is determined to
- * be stably zero. An example unstable zero can occur if the call
- * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
- * but due to task migration, sees the corresponding __srcu_read_unlock()
- * decrement. This can happen because srcu_readers_active_idx() takes
- * time to sum the array, and might in fact be interrupted or preempted
- * partway through the summation.
+ * be zero.
*/
static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
{
- unsigned long seq;
+ unsigned long unlocks;
- seq = srcu_readers_seq_idx(sp, idx);
+ unlocks = srcu_readers_unlock_idx(sp, idx);
/*
- * The following smp_mb() A pairs with the smp_mb() B located in
- * __srcu_read_lock(). This pairing ensures that if an
- * __srcu_read_lock() increments its counter after the summation
- * in srcu_readers_active_idx(), then the corresponding SRCU read-side
- * critical section will see any changes made prior to the start
- * of the current SRCU grace period.
+ * Make sure that a lock is always counted if the corresponding unlock
+ * is counted. Needs to be a smp_mb() as the read side may contain a
+ * read from a variable that is written to before the synchronize_srcu()
+ * in the write side. In this case smp_mb()s A and B act like the store
+ * buffering pattern.
*
- * Also, if the above call to srcu_readers_seq_idx() saw the
- * increment of ->seq[], then the call to srcu_readers_active_idx()
- * must see the increment of ->c[].
+ * This smp_mb() also pairs with smp_mb() C to prevent writes after the
+ * synchronize_srcu() from being executed before the grace period ends.
*/
smp_mb(); /* A */
/*
- * Note that srcu_readers_active_idx() can incorrectly return
- * zero even though there is a pre-existing reader throughout.
- * To see this, suppose that task A is in a very long SRCU
- * read-side critical section that started on CPU 0, and that
- * no other reader exists, so that the sum of the counters
- * is equal to one. Then suppose that task B starts executing
- * srcu_readers_active_idx(), summing up to CPU 1, and then that
- * task C starts reading on CPU 0, so that its increment is not
- * summed, but finishes reading on CPU 2, so that its decrement
- * -is- summed. Then when task B completes its sum, it will
- * incorrectly get zero, despite the fact that task A has been
- * in its SRCU read-side critical section the whole time.
- *
- * We therefore do a validation step should srcu_readers_active_idx()
- * return zero.
- */
- if (srcu_readers_active_idx(sp, idx) != 0)
- return false;
-
- /*
- * The remainder of this function is the validation step.
- * The following smp_mb() D pairs with the smp_mb() C in
- * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
- * by srcu_readers_active_idx() above, then any destructive
- * operation performed after the grace period will happen after
- * the corresponding SRCU read-side critical section.
+ * If the locks are the same as the unlocks, then there must of have
+ * been no readers on this index at some time in between. This does not
+ * mean that there are no more readers, as one could have read the
+ * current index but have incremented the lock counter yet.
*
- * Note that there can be at most NR_CPUS worth of readers using
- * the old index, which is not enough to overflow even a 32-bit
- * integer. (Yes, this does mean that systems having more than
- * a billion or so CPUs need to be 64-bit systems.) Therefore,
- * the sum of the ->seq[] counters cannot possibly overflow.
- * Therefore, the only way that the return values of the two
- * calls to srcu_readers_seq_idx() can be equal is if there were
- * no increments of the corresponding rank of ->seq[] counts
- * in the interim. But the missed-increment scenario laid out
- * above includes an increment of the ->seq[] counter by
- * the corresponding __srcu_read_lock(). Therefore, if this
- * scenario occurs, the return values from the two calls to
- * srcu_readers_seq_idx() will differ, and thus the validation
- * step below suffices.
+ * Possible bug: There is no guarantee that there haven't been ULONG_MAX
+ * increments of ->lock_count[] since the unlocks were counted, meaning
+ * that this could return true even if there are still active readers.
+ * Since there are no memory barriers around srcu_flip(), the CPU is not
+ * required to increment ->completed before running
+ * srcu_readers_unlock_idx(), which means that there could be an
+ * arbitrarily large number of critical sections that execute after
+ * srcu_readers_unlock_idx() but use the old value of ->completed.
*/
- smp_mb(); /* D */
-
- return srcu_readers_seq_idx(sp, idx) == seq;
+ return srcu_readers_lock_idx(sp, idx) == unlocks;
}
/**
@@ -266,8 +233,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
unsigned long sum = 0;
for_each_possible_cpu(cpu) {
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
+ struct srcu_struct_array *cpu_counts =
+ per_cpu_ptr(sp->per_cpu_ref, cpu);
+ sum += READ_ONCE(cpu_counts->lock_count[0]);
+ sum += READ_ONCE(cpu_counts->lock_count[1]);
+ sum -= READ_ONCE(cpu_counts->unlock_count[0]);
+ sum -= READ_ONCE(cpu_counts->unlock_count[1]);
}
return sum;
}
@@ -298,9 +269,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
int idx;
idx = READ_ONCE(sp->completed) & 0x1;
- __this_cpu_inc(sp->per_cpu_ref->c[idx]);
+ __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
smp_mb(); /* B */ /* Avoid leaking the critical section. */
- __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
return idx;
}
EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -314,7 +284,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
void __srcu_read_unlock(struct srcu_struct *sp, int idx)
{
smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_dec(sp->per_cpu_ref->c[idx]);
+ this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
}
EXPORT_SYMBOL_GPL(__srcu_read_unlock);
@@ -349,7 +319,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
/*
* Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->c[] and ->seq[] arrays. This allows
+ * use the other rank of the ->(un)lock_count[] arrays. This allows
* us to wait for pre-existing readers in a starvation-free manner.
*/
static void srcu_flip(struct srcu_struct *sp)
--
2.9.0
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH] SRCU: More efficient reader counts.
2017-01-23 20:17 ` [PATCH] SRCU: More efficient " Lance Roy
@ 2017-01-23 20:35 ` Paul E. McKenney
2017-01-23 21:33 ` Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-23 20:35 UTC (permalink / raw)
To: Lance Roy
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, Jan 23, 2017 at 12:17:25PM -0800, Lance Roy wrote:
> SRCU uses two per-cpu counters: a nesting counter to count the number of
> active critical sections, and a sequence counter to ensure that the nesting
> counters don't change while they are being added together in
> srcu_readers_active_idx_check().
>
> This patch instead uses per-cpu lock and unlock counters. Because the both
> counters only increase and srcu_readers_active_idx_check() reads the unlock
> counter before the lock counter, this achieves the same end without having
> to increment two different counters in srcu_read_lock(). This also saves a
> smp_mb() in srcu_readers_active_idx_check().
>
> Possible bug: There is no guarantee that the lock counter won't overflow
> during srcu_readers_active_idx_check(), as there are no memory barriers
> around srcu_flip() (see comment in srcu_readers_active_idx_check() for
> details). However, this problem was already present before this patch.
>
> Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Signed-off-by: Lance Roy <ldr709@gmail.com>
In general, the comment update looks good, but this patch undoes my
application of review feedback to your original patch. Could you please
submit the comment update as a separate patch on top of branch rcu/next
of my -rcu tree?
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
Of course, if any of the changes from the review feedback are in any
way problematic, please do let me know.
Thanx, Paul
> ---
> include/linux/srcu.h | 4 +-
> kernel/rcu/rcutorture.c | 20 ++++++++-
> kernel/rcu/srcu.c | 116 ++++++++++++++++++------------------------------
> 3 files changed, 63 insertions(+), 77 deletions(-)
>
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index dc8eb63..0caea34 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -34,8 +34,8 @@
> #include <linux/workqueue.h>
>
> struct srcu_struct_array {
> - unsigned long c[2];
> - unsigned long seq[2];
> + unsigned long lock_count[2];
> + unsigned long unlock_count[2];
> };
>
> struct rcu_batch {
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 87c5122..c3f25d1 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -564,10 +564,26 @@ static void srcu_torture_stats(void)
> pr_alert("%s%s per-CPU(idx=%d):",
> torture_type, TORTURE_FLAG, idx);
> for_each_possible_cpu(cpu) {
> + unsigned long l0, l1;
> + unsigned long u0, u1;
> long c0, c1;
> + struct srcu_struct_array *counts =
> + per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
>
> - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> + u0 = counts->unlock_count[!idx];
> + u1 = counts->unlock_count[idx];
> +
> + /*
> + * Make sure that a lock is always counted if the corresponding
> + * unlock is counted.
> + */
> + smp_rmb();
> +
> + l0 = counts->lock_count[!idx];
> + l1 = counts->lock_count[idx];
> +
> + c0 = (long)(l0 - u0);
> + c1 = (long)(l1 - u1);
> pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
> }
> pr_cont("\n");
> diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> index 9b9cdd5..38e9aae 100644
> --- a/kernel/rcu/srcu.c
> +++ b/kernel/rcu/srcu.c
> @@ -141,34 +141,38 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
> #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
>
> /*
> - * Returns approximate total of the readers' ->seq[] values for the
> + * Returns approximate total of the readers' ->lock_count[] values for the
> * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> + struct srcu_struct_array *cpu_counts =
> + per_cpu_ptr(sp->per_cpu_ref, cpu);
> + t = READ_ONCE(cpu_counts->lock_count[idx]);
> sum += t;
> }
> return sum;
> }
>
> /*
> - * Returns approximate number of readers active on the specified rank
> - * of the per-CPU ->c[] counters.
> + * Returns approximate total of the readers' ->unlock_count[] values for the
> + * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> + struct srcu_struct_array *cpu_counts =
> + per_cpu_ptr(sp->per_cpu_ref, cpu);
> + t = READ_ONCE(cpu_counts->unlock_count[idx]);
> sum += t;
> }
> return sum;
> @@ -176,79 +180,42 @@ static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
>
> /*
> * Return true if the number of pre-existing readers is determined to
> - * be stably zero. An example unstable zero can occur if the call
> - * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
> - * but due to task migration, sees the corresponding __srcu_read_unlock()
> - * decrement. This can happen because srcu_readers_active_idx() takes
> - * time to sum the array, and might in fact be interrupted or preempted
> - * partway through the summation.
> + * be zero.
> */
> static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
> {
> - unsigned long seq;
> + unsigned long unlocks;
>
> - seq = srcu_readers_seq_idx(sp, idx);
> + unlocks = srcu_readers_unlock_idx(sp, idx);
>
> /*
> - * The following smp_mb() A pairs with the smp_mb() B located in
> - * __srcu_read_lock(). This pairing ensures that if an
> - * __srcu_read_lock() increments its counter after the summation
> - * in srcu_readers_active_idx(), then the corresponding SRCU read-side
> - * critical section will see any changes made prior to the start
> - * of the current SRCU grace period.
> + * Make sure that a lock is always counted if the corresponding unlock
> + * is counted. Needs to be a smp_mb() as the read side may contain a
> + * read from a variable that is written to before the synchronize_srcu()
> + * in the write side. In this case smp_mb()s A and B act like the store
> + * buffering pattern.
> *
> - * Also, if the above call to srcu_readers_seq_idx() saw the
> - * increment of ->seq[], then the call to srcu_readers_active_idx()
> - * must see the increment of ->c[].
> + * This smp_mb() also pairs with smp_mb() C to prevent writes after the
> + * synchronize_srcu() from being executed before the grace period ends.
> */
> smp_mb(); /* A */
>
> /*
> - * Note that srcu_readers_active_idx() can incorrectly return
> - * zero even though there is a pre-existing reader throughout.
> - * To see this, suppose that task A is in a very long SRCU
> - * read-side critical section that started on CPU 0, and that
> - * no other reader exists, so that the sum of the counters
> - * is equal to one. Then suppose that task B starts executing
> - * srcu_readers_active_idx(), summing up to CPU 1, and then that
> - * task C starts reading on CPU 0, so that its increment is not
> - * summed, but finishes reading on CPU 2, so that its decrement
> - * -is- summed. Then when task B completes its sum, it will
> - * incorrectly get zero, despite the fact that task A has been
> - * in its SRCU read-side critical section the whole time.
> - *
> - * We therefore do a validation step should srcu_readers_active_idx()
> - * return zero.
> - */
> - if (srcu_readers_active_idx(sp, idx) != 0)
> - return false;
> -
> - /*
> - * The remainder of this function is the validation step.
> - * The following smp_mb() D pairs with the smp_mb() C in
> - * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
> - * by srcu_readers_active_idx() above, then any destructive
> - * operation performed after the grace period will happen after
> - * the corresponding SRCU read-side critical section.
> + * If the locks are the same as the unlocks, then there must of have
> + * been no readers on this index at some time in between. This does not
> + * mean that there are no more readers, as one could have read the
> + * current index but have incremented the lock counter yet.
> *
> - * Note that there can be at most NR_CPUS worth of readers using
> - * the old index, which is not enough to overflow even a 32-bit
> - * integer. (Yes, this does mean that systems having more than
> - * a billion or so CPUs need to be 64-bit systems.) Therefore,
> - * the sum of the ->seq[] counters cannot possibly overflow.
> - * Therefore, the only way that the return values of the two
> - * calls to srcu_readers_seq_idx() can be equal is if there were
> - * no increments of the corresponding rank of ->seq[] counts
> - * in the interim. But the missed-increment scenario laid out
> - * above includes an increment of the ->seq[] counter by
> - * the corresponding __srcu_read_lock(). Therefore, if this
> - * scenario occurs, the return values from the two calls to
> - * srcu_readers_seq_idx() will differ, and thus the validation
> - * step below suffices.
> + * Possible bug: There is no guarantee that there haven't been ULONG_MAX
> + * increments of ->lock_count[] since the unlocks were counted, meaning
> + * that this could return true even if there are still active readers.
> + * Since there are no memory barriers around srcu_flip(), the CPU is not
> + * required to increment ->completed before running
> + * srcu_readers_unlock_idx(), which means that there could be an
> + * arbitrarily large number of critical sections that execute after
> + * srcu_readers_unlock_idx() but use the old value of ->completed.
> */
> - smp_mb(); /* D */
> -
> - return srcu_readers_seq_idx(sp, idx) == seq;
> + return srcu_readers_lock_idx(sp, idx) == unlocks;
> }
>
> /**
> @@ -266,8 +233,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
> unsigned long sum = 0;
>
> for_each_possible_cpu(cpu) {
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
> + struct srcu_struct_array *cpu_counts =
> + per_cpu_ptr(sp->per_cpu_ref, cpu);
> + sum += READ_ONCE(cpu_counts->lock_count[0]);
> + sum += READ_ONCE(cpu_counts->lock_count[1]);
> + sum -= READ_ONCE(cpu_counts->unlock_count[0]);
> + sum -= READ_ONCE(cpu_counts->unlock_count[1]);
> }
> return sum;
> }
> @@ -298,9 +269,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
> int idx;
>
> idx = READ_ONCE(sp->completed) & 0x1;
> - __this_cpu_inc(sp->per_cpu_ref->c[idx]);
> + __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
> smp_mb(); /* B */ /* Avoid leaking the critical section. */
> - __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
> return idx;
> }
> EXPORT_SYMBOL_GPL(__srcu_read_lock);
> @@ -314,7 +284,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
> void __srcu_read_unlock(struct srcu_struct *sp, int idx)
> {
> smp_mb(); /* C */ /* Avoid leaking the critical section. */
> - this_cpu_dec(sp->per_cpu_ref->c[idx]);
> + this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
> }
> EXPORT_SYMBOL_GPL(__srcu_read_unlock);
>
> @@ -349,7 +319,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
>
> /*
> * Increment the ->completed counter so that future SRCU readers will
> - * use the other rank of the ->c[] and ->seq[] arrays. This allows
> + * use the other rank of the ->(un)lock_count[] arrays. This allows
> * us to wait for pre-existing readers in a starvation-free manner.
> */
> static void srcu_flip(struct srcu_struct *sp)
> --
> 2.9.0
>
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH] SRCU: More efficient reader counts.
2017-01-23 20:35 ` Paul E. McKenney
@ 2017-01-23 21:33 ` Lance Roy
2017-01-23 21:35 ` [PATCH] srcu: Implement more-efficient " Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-23 21:33 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, 23 Jan 2017 12:35:08 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> On Mon, Jan 23, 2017 at 12:17:25PM -0800, Lance Roy wrote:
> > SRCU uses two per-cpu counters: a nesting counter to count the number of
> > active critical sections, and a sequence counter to ensure that the nesting
> > counters don't change while they are being added together in
> > srcu_readers_active_idx_check().
> >
> > This patch instead uses per-cpu lock and unlock counters. Because the both
> > counters only increase and srcu_readers_active_idx_check() reads the unlock
> > counter before the lock counter, this achieves the same end without having
> > to increment two different counters in srcu_read_lock(). This also saves a
> > smp_mb() in srcu_readers_active_idx_check().
> >
> > Possible bug: There is no guarantee that the lock counter won't overflow
> > during srcu_readers_active_idx_check(), as there are no memory barriers
> > around srcu_flip() (see comment in srcu_readers_active_idx_check() for
> > details). However, this problem was already present before this patch.
> >
> > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Signed-off-by: Lance Roy <ldr709@gmail.com>
>
> In general, the comment update looks good, but this patch undoes my
> application of review feedback to your original patch. Could you please
> submit the comment update as a separate patch on top of branch rcu/next
> of my -rcu tree?
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
I have added the review feedback to the new patch. I will send the new version.
If you prefer that the comment changes are in a separate patch I can change it,
but I thought it would be better not to have an incorrect statement in the
commit message.
> Of course, if any of the changes from the review feedback are in any
> way problematic, please do let me know.
I don't see any problems.
Thanks,
Lance
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH] srcu: Implement more-efficient reader counts
2017-01-23 21:33 ` Lance Roy
@ 2017-01-23 21:35 ` Lance Roy
2017-01-24 0:42 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-23 21:35 UTC (permalink / raw)
To: linux-kernel
Cc: Paul E. McKenney, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani, Lance Roy
SRCU uses two per-cpu counters: a nesting counter to count the number of
active critical sections, and a sequence counter to ensure that the nesting
counters don't change while they are being added together in
srcu_readers_active_idx_check().
This patch instead uses per-cpu lock and unlock counters. Because both
counters only increase and srcu_readers_active_idx_check() reads the unlock
counter before the lock counter, this achieves the same end without having
to increment two different counters in srcu_read_lock(). This also saves a
smp_mb() in srcu_readers_active_idx_check().
Possible bug: There is no guarantee that the lock counter won't overflow
during srcu_readers_active_idx_check(), as there are no memory barriers
around srcu_flip() (see comment in srcu_readers_active_idx_check() for
details). However, this problem was already present before this patch.
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lance Roy <ldr709@gmail.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
include/linux/srcu.h | 10 ++--
kernel/rcu/rcutorture.c | 19 +++++++-
kernel/rcu/srcu.c | 122 +++++++++++++++++-------------------------------
3 files changed, 66 insertions(+), 85 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index dc8eb63..a598cf3 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -33,9 +33,9 @@
#include <linux/rcupdate.h>
#include <linux/workqueue.h>
-struct srcu_struct_array {
- unsigned long c[2];
- unsigned long seq[2];
+struct srcu_array {
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
};
struct rcu_batch {
@@ -46,7 +46,7 @@ struct rcu_batch {
struct srcu_struct {
unsigned long completed;
- struct srcu_struct_array __percpu *per_cpu_ref;
+ struct srcu_array __percpu *per_cpu_ref;
spinlock_t queue_lock; /* protect ->batch_queue, ->running */
bool running;
/* callbacks just queued */
@@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
* See include/linux/percpu-defs.h for the rules on per-CPU variables.
*/
#define __DEFINE_SRCU(name, is_static) \
- static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
+ static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 87c5122..d81345b 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
pr_alert("%s%s per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
+ unsigned long l0, l1;
+ unsigned long u0, u1;
long c0, c1;
+ struct srcu_array *counts = per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
- c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
+ u0 = counts->unlock_count[!idx];
+ u1 = counts->unlock_count[idx];
+
+ /*
+ * Make sure that a lock is always counted if the corresponding
+ * unlock is counted.
+ */
+ smp_rmb();
+
+ l0 = counts->lock_count[!idx];
+ l1 = counts->lock_count[idx];
+
+ c0 = l0 - u0;
+ c1 = l1 - u1;
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 9b9cdd5..c9a0015 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
rcu_batch_init(&sp->batch_check1);
rcu_batch_init(&sp->batch_done);
INIT_DELAYED_WORK(&sp->work, process_srcu);
- sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
+ sp->per_cpu_ref = alloc_percpu(struct srcu_array);
return sp->per_cpu_ref ? 0 : -ENOMEM;
}
@@ -141,114 +141,77 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
- * Returns approximate total of the readers' ->seq[] values for the
+ * Returns approximate total of the readers' ->lock_count[] values for the
* rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[idx]);
}
return sum;
}
/*
- * Returns approximate number of readers active on the specified rank
- * of the per-CPU ->c[] counters.
+ * Returns approximate total of the readers' ->unlock_count[] values for the
+ * rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->unlock_count[idx]);
}
return sum;
}
/*
* Return true if the number of pre-existing readers is determined to
- * be stably zero. An example unstable zero can occur if the call
- * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
- * but due to task migration, sees the corresponding __srcu_read_unlock()
- * decrement. This can happen because srcu_readers_active_idx() takes
- * time to sum the array, and might in fact be interrupted or preempted
- * partway through the summation.
+ * be zero.
*/
static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
{
- unsigned long seq;
+ unsigned long unlocks;
- seq = srcu_readers_seq_idx(sp, idx);
+ unlocks = srcu_readers_unlock_idx(sp, idx);
/*
- * The following smp_mb() A pairs with the smp_mb() B located in
- * __srcu_read_lock(). This pairing ensures that if an
- * __srcu_read_lock() increments its counter after the summation
- * in srcu_readers_active_idx(), then the corresponding SRCU read-side
- * critical section will see any changes made prior to the start
- * of the current SRCU grace period.
+ * Make sure that a lock is always counted if the corresponding unlock
+ * is counted. Needs to be a smp_mb() as the read side may contain a
+ * read from a variable that is written to before the synchronize_srcu()
+ * in the write side. In this case smp_mb()s A and B act like the store
+ * buffering pattern.
*
- * Also, if the above call to srcu_readers_seq_idx() saw the
- * increment of ->seq[], then the call to srcu_readers_active_idx()
- * must see the increment of ->c[].
+ * This smp_mb() also pairs with smp_mb() C to prevent accesses after the
+ * synchronize_srcu() from being executed before the grace period ends.
*/
smp_mb(); /* A */
/*
- * Note that srcu_readers_active_idx() can incorrectly return
- * zero even though there is a pre-existing reader throughout.
- * To see this, suppose that task A is in a very long SRCU
- * read-side critical section that started on CPU 0, and that
- * no other reader exists, so that the sum of the counters
- * is equal to one. Then suppose that task B starts executing
- * srcu_readers_active_idx(), summing up to CPU 1, and then that
- * task C starts reading on CPU 0, so that its increment is not
- * summed, but finishes reading on CPU 2, so that its decrement
- * -is- summed. Then when task B completes its sum, it will
- * incorrectly get zero, despite the fact that task A has been
- * in its SRCU read-side critical section the whole time.
+ * If the locks are the same as the unlocks, then there must have
+ * been no readers on this index at some time in between. This does not
+ * mean that there are no more readers, as one could have read the
+ * current index but not have incremented the lock counter yet.
*
- * We therefore do a validation step should srcu_readers_active_idx()
- * return zero.
+ * Possible bug: There is no guarantee that there haven't been ULONG_MAX
+ * increments of ->lock_count[] since the unlocks were counted, meaning
+ * that this could return true even if there are still active readers.
+ * Since there are no memory barriers around srcu_flip(), the CPU is not
+ * required to increment ->completed before running
+ * srcu_readers_unlock_idx(), which means that there could be an
+ * arbitrarily large number of critical sections that execute after
+ * srcu_readers_unlock_idx() but use the old value of ->completed.
*/
- if (srcu_readers_active_idx(sp, idx) != 0)
- return false;
-
- /*
- * The remainder of this function is the validation step.
- * The following smp_mb() D pairs with the smp_mb() C in
- * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
- * by srcu_readers_active_idx() above, then any destructive
- * operation performed after the grace period will happen after
- * the corresponding SRCU read-side critical section.
- *
- * Note that there can be at most NR_CPUS worth of readers using
- * the old index, which is not enough to overflow even a 32-bit
- * integer. (Yes, this does mean that systems having more than
- * a billion or so CPUs need to be 64-bit systems.) Therefore,
- * the sum of the ->seq[] counters cannot possibly overflow.
- * Therefore, the only way that the return values of the two
- * calls to srcu_readers_seq_idx() can be equal is if there were
- * no increments of the corresponding rank of ->seq[] counts
- * in the interim. But the missed-increment scenario laid out
- * above includes an increment of the ->seq[] counter by
- * the corresponding __srcu_read_lock(). Therefore, if this
- * scenario occurs, the return values from the two calls to
- * srcu_readers_seq_idx() will differ, and thus the validation
- * step below suffices.
- */
- smp_mb(); /* D */
-
- return srcu_readers_seq_idx(sp, idx) == seq;
+ return srcu_readers_lock_idx(sp, idx) == unlocks;
}
/**
@@ -266,8 +229,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
unsigned long sum = 0;
for_each_possible_cpu(cpu) {
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[0]);
+ sum += READ_ONCE(cpuc->lock_count[1]);
+ sum -= READ_ONCE(cpuc->unlock_count[0]);
+ sum -= READ_ONCE(cpuc->unlock_count[1]);
}
return sum;
}
@@ -298,9 +265,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
int idx;
idx = READ_ONCE(sp->completed) & 0x1;
- __this_cpu_inc(sp->per_cpu_ref->c[idx]);
+ __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
smp_mb(); /* B */ /* Avoid leaking the critical section. */
- __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
return idx;
}
EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -314,7 +280,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
void __srcu_read_unlock(struct srcu_struct *sp, int idx)
{
smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_dec(sp->per_cpu_ref->c[idx]);
+ this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
}
EXPORT_SYMBOL_GPL(__srcu_read_unlock);
@@ -349,7 +315,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
/*
* Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->c[] and ->seq[] arrays. This allows
+ * use the other rank of the ->(un)lock_count[] arrays. This allows
* us to wait for pre-existing readers in a starvation-free manner.
*/
static void srcu_flip(struct srcu_struct *sp)
--
2.9.0
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH] srcu: Implement more-efficient reader counts
2017-01-23 21:35 ` [PATCH] srcu: Implement more-efficient " Lance Roy
@ 2017-01-24 0:42 ` Paul E. McKenney
2017-01-24 0:53 ` Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 0:42 UTC (permalink / raw)
To: Lance Roy
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, Jan 23, 2017 at 01:35:18PM -0800, Lance Roy wrote:
> SRCU uses two per-cpu counters: a nesting counter to count the number of
> active critical sections, and a sequence counter to ensure that the nesting
> counters don't change while they are being added together in
> srcu_readers_active_idx_check().
>
> This patch instead uses per-cpu lock and unlock counters. Because both
> counters only increase and srcu_readers_active_idx_check() reads the unlock
> counter before the lock counter, this achieves the same end without having
> to increment two different counters in srcu_read_lock(). This also saves a
> smp_mb() in srcu_readers_active_idx_check().
>
> Possible bug: There is no guarantee that the lock counter won't overflow
> during srcu_readers_active_idx_check(), as there are no memory barriers
> around srcu_flip() (see comment in srcu_readers_active_idx_check() for
> details). However, this problem was already present before this patch.
>
> Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Signed-off-by: Lance Roy <ldr709@gmail.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
OK, this only has differences only in the comments, so I can reasonably
substitute it, even this near the next merge window.
But let's talk about the potential overflow. If I understand correctly,
for this to happen, there needs to be ULONG_MAX/2 or thereabouts
srcu_read_lock() calls without matching srcu_read_unlock() calls.
Let's focus on 32-bit systems for the moment.
Lockdep allows at most 48 locks held at a given time by any single task,
and RCU does not pass in a non-NULL nest_lock -- if you acquire more than
that, a lockdep kernel will splat. Each task has at least one 4K page of
kernel stack, so there can be at most 1,048,576 tasks (actually quite a
bit fewer due to the size of the task_struct and so on). This allows
at most 50,331,648 unmatched srcu_read_lock() calls in the system,
which is not sufficient to cause overflow.
Or am I missing something here?
Thanx, Paul
> ---
> include/linux/srcu.h | 10 ++--
> kernel/rcu/rcutorture.c | 19 +++++++-
> kernel/rcu/srcu.c | 122 +++++++++++++++++-------------------------------
> 3 files changed, 66 insertions(+), 85 deletions(-)
>
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index dc8eb63..a598cf3 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -33,9 +33,9 @@
> #include <linux/rcupdate.h>
> #include <linux/workqueue.h>
>
> -struct srcu_struct_array {
> - unsigned long c[2];
> - unsigned long seq[2];
> +struct srcu_array {
> + unsigned long lock_count[2];
> + unsigned long unlock_count[2];
> };
>
> struct rcu_batch {
> @@ -46,7 +46,7 @@ struct rcu_batch {
>
> struct srcu_struct {
> unsigned long completed;
> - struct srcu_struct_array __percpu *per_cpu_ref;
> + struct srcu_array __percpu *per_cpu_ref;
> spinlock_t queue_lock; /* protect ->batch_queue, ->running */
> bool running;
> /* callbacks just queued */
> @@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
> * See include/linux/percpu-defs.h for the rules on per-CPU variables.
> */
> #define __DEFINE_SRCU(name, is_static) \
> - static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
> + static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
> is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
> #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
> #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 87c5122..d81345b 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
> pr_alert("%s%s per-CPU(idx=%d):",
> torture_type, TORTURE_FLAG, idx);
> for_each_possible_cpu(cpu) {
> + unsigned long l0, l1;
> + unsigned long u0, u1;
> long c0, c1;
> + struct srcu_array *counts = per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
>
> - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> + u0 = counts->unlock_count[!idx];
> + u1 = counts->unlock_count[idx];
> +
> + /*
> + * Make sure that a lock is always counted if the corresponding
> + * unlock is counted.
> + */
> + smp_rmb();
> +
> + l0 = counts->lock_count[!idx];
> + l1 = counts->lock_count[idx];
> +
> + c0 = l0 - u0;
> + c1 = l1 - u1;
> pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
> }
> pr_cont("\n");
> diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> index 9b9cdd5..c9a0015 100644
> --- a/kernel/rcu/srcu.c
> +++ b/kernel/rcu/srcu.c
> @@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
> rcu_batch_init(&sp->batch_check1);
> rcu_batch_init(&sp->batch_done);
> INIT_DELAYED_WORK(&sp->work, process_srcu);
> - sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
> + sp->per_cpu_ref = alloc_percpu(struct srcu_array);
> return sp->per_cpu_ref ? 0 : -ENOMEM;
> }
>
> @@ -141,114 +141,77 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
> #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
>
> /*
> - * Returns approximate total of the readers' ->seq[] values for the
> + * Returns approximate total of the readers' ->lock_count[] values for the
> * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> - unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> - sum += t;
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->lock_count[idx]);
> }
> return sum;
> }
>
> /*
> - * Returns approximate number of readers active on the specified rank
> - * of the per-CPU ->c[] counters.
> + * Returns approximate total of the readers' ->unlock_count[] values for the
> + * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> - unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> - sum += t;
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->unlock_count[idx]);
> }
> return sum;
> }
>
> /*
> * Return true if the number of pre-existing readers is determined to
> - * be stably zero. An example unstable zero can occur if the call
> - * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
> - * but due to task migration, sees the corresponding __srcu_read_unlock()
> - * decrement. This can happen because srcu_readers_active_idx() takes
> - * time to sum the array, and might in fact be interrupted or preempted
> - * partway through the summation.
> + * be zero.
> */
> static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
> {
> - unsigned long seq;
> + unsigned long unlocks;
>
> - seq = srcu_readers_seq_idx(sp, idx);
> + unlocks = srcu_readers_unlock_idx(sp, idx);
>
> /*
> - * The following smp_mb() A pairs with the smp_mb() B located in
> - * __srcu_read_lock(). This pairing ensures that if an
> - * __srcu_read_lock() increments its counter after the summation
> - * in srcu_readers_active_idx(), then the corresponding SRCU read-side
> - * critical section will see any changes made prior to the start
> - * of the current SRCU grace period.
> + * Make sure that a lock is always counted if the corresponding unlock
> + * is counted. Needs to be a smp_mb() as the read side may contain a
> + * read from a variable that is written to before the synchronize_srcu()
> + * in the write side. In this case smp_mb()s A and B act like the store
> + * buffering pattern.
> *
> - * Also, if the above call to srcu_readers_seq_idx() saw the
> - * increment of ->seq[], then the call to srcu_readers_active_idx()
> - * must see the increment of ->c[].
> + * This smp_mb() also pairs with smp_mb() C to prevent accesses after the
> + * synchronize_srcu() from being executed before the grace period ends.
> */
> smp_mb(); /* A */
>
> /*
> - * Note that srcu_readers_active_idx() can incorrectly return
> - * zero even though there is a pre-existing reader throughout.
> - * To see this, suppose that task A is in a very long SRCU
> - * read-side critical section that started on CPU 0, and that
> - * no other reader exists, so that the sum of the counters
> - * is equal to one. Then suppose that task B starts executing
> - * srcu_readers_active_idx(), summing up to CPU 1, and then that
> - * task C starts reading on CPU 0, so that its increment is not
> - * summed, but finishes reading on CPU 2, so that its decrement
> - * -is- summed. Then when task B completes its sum, it will
> - * incorrectly get zero, despite the fact that task A has been
> - * in its SRCU read-side critical section the whole time.
> + * If the locks are the same as the unlocks, then there must have
> + * been no readers on this index at some time in between. This does not
> + * mean that there are no more readers, as one could have read the
> + * current index but not have incremented the lock counter yet.
> *
> - * We therefore do a validation step should srcu_readers_active_idx()
> - * return zero.
> + * Possible bug: There is no guarantee that there haven't been ULONG_MAX
> + * increments of ->lock_count[] since the unlocks were counted, meaning
> + * that this could return true even if there are still active readers.
> + * Since there are no memory barriers around srcu_flip(), the CPU is not
> + * required to increment ->completed before running
> + * srcu_readers_unlock_idx(), which means that there could be an
> + * arbitrarily large number of critical sections that execute after
> + * srcu_readers_unlock_idx() but use the old value of ->completed.
> */
> - if (srcu_readers_active_idx(sp, idx) != 0)
> - return false;
> -
> - /*
> - * The remainder of this function is the validation step.
> - * The following smp_mb() D pairs with the smp_mb() C in
> - * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
> - * by srcu_readers_active_idx() above, then any destructive
> - * operation performed after the grace period will happen after
> - * the corresponding SRCU read-side critical section.
> - *
> - * Note that there can be at most NR_CPUS worth of readers using
> - * the old index, which is not enough to overflow even a 32-bit
> - * integer. (Yes, this does mean that systems having more than
> - * a billion or so CPUs need to be 64-bit systems.) Therefore,
> - * the sum of the ->seq[] counters cannot possibly overflow.
> - * Therefore, the only way that the return values of the two
> - * calls to srcu_readers_seq_idx() can be equal is if there were
> - * no increments of the corresponding rank of ->seq[] counts
> - * in the interim. But the missed-increment scenario laid out
> - * above includes an increment of the ->seq[] counter by
> - * the corresponding __srcu_read_lock(). Therefore, if this
> - * scenario occurs, the return values from the two calls to
> - * srcu_readers_seq_idx() will differ, and thus the validation
> - * step below suffices.
> - */
> - smp_mb(); /* D */
> -
> - return srcu_readers_seq_idx(sp, idx) == seq;
> + return srcu_readers_lock_idx(sp, idx) == unlocks;
> }
>
> /**
> @@ -266,8 +229,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
> unsigned long sum = 0;
>
> for_each_possible_cpu(cpu) {
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->lock_count[0]);
> + sum += READ_ONCE(cpuc->lock_count[1]);
> + sum -= READ_ONCE(cpuc->unlock_count[0]);
> + sum -= READ_ONCE(cpuc->unlock_count[1]);
> }
> return sum;
> }
> @@ -298,9 +265,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
> int idx;
>
> idx = READ_ONCE(sp->completed) & 0x1;
> - __this_cpu_inc(sp->per_cpu_ref->c[idx]);
> + __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
> smp_mb(); /* B */ /* Avoid leaking the critical section. */
> - __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
> return idx;
> }
> EXPORT_SYMBOL_GPL(__srcu_read_lock);
> @@ -314,7 +280,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
> void __srcu_read_unlock(struct srcu_struct *sp, int idx)
> {
> smp_mb(); /* C */ /* Avoid leaking the critical section. */
> - this_cpu_dec(sp->per_cpu_ref->c[idx]);
> + this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
> }
> EXPORT_SYMBOL_GPL(__srcu_read_unlock);
>
> @@ -349,7 +315,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
>
> /*
> * Increment the ->completed counter so that future SRCU readers will
> - * use the other rank of the ->c[] and ->seq[] arrays. This allows
> + * use the other rank of the ->(un)lock_count[] arrays. This allows
> * us to wait for pre-existing readers in a starvation-free manner.
> */
> static void srcu_flip(struct srcu_struct *sp)
> --
> 2.9.0
>
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH] srcu: Implement more-efficient reader counts
2017-01-24 0:42 ` Paul E. McKenney
@ 2017-01-24 0:53 ` Lance Roy
2017-01-24 1:57 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-24 0:53 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, 23 Jan 2017 16:42:52 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> On Mon, Jan 23, 2017 at 01:35:18PM -0800, Lance Roy wrote:
> > SRCU uses two per-cpu counters: a nesting counter to count the number of
> > active critical sections, and a sequence counter to ensure that the nesting
> > counters don't change while they are being added together in
> > srcu_readers_active_idx_check().
> >
> > This patch instead uses per-cpu lock and unlock counters. Because both
> > counters only increase and srcu_readers_active_idx_check() reads the unlock
> > counter before the lock counter, this achieves the same end without having
> > to increment two different counters in srcu_read_lock(). This also saves a
> > smp_mb() in srcu_readers_active_idx_check().
> >
> > Possible bug: There is no guarantee that the lock counter won't overflow
> > during srcu_readers_active_idx_check(), as there are no memory barriers
> > around srcu_flip() (see comment in srcu_readers_active_idx_check() for
> > details). However, this problem was already present before this patch.
> >
> > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Signed-off-by: Lance Roy <ldr709@gmail.com>
> > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
>
> OK, this only has differences only in the comments, so I can reasonably
> substitute it, even this near the next merge window.
>
> But let's talk about the potential overflow. If I understand correctly,
> for this to happen, there needs to be ULONG_MAX/2 or thereabouts
> srcu_read_lock() calls without matching srcu_read_unlock() calls.
> Let's focus on 32-bit systems for the moment.
>
> Lockdep allows at most 48 locks held at a given time by any single task,
> and RCU does not pass in a non-NULL nest_lock -- if you acquire more than
> that, a lockdep kernel will splat. Each task has at least one 4K page of
> kernel stack, so there can be at most 1,048,576 tasks (actually quite a
> bit fewer due to the size of the task_struct and so on). This allows
> at most 50,331,648 unmatched srcu_read_lock() calls in the system,
> which is not sufficient to cause overflow.
>
> Or am I missing something here?
>
> Thanx, Paul
It isn't the nest that is the problem. Here is a previous email I wrote
describing the problem:
The trouble is that disabling preemption is not enough to ensure that there
is at most one srcu_read_lock() call per CPU that missed the srcu_flip().
Define a reader to be an SRCU lock+unlock pair. A reader is called active if it
has incremented ->lock_count[] but hasn't incremented ->unlock_count[] yet, and
completed if it has incremented ->unlock_count[]. I think that we only want to
limit the number of active readers and the number of CPUs. In particular, I
don't think there should be a limit on the rate of completion of read side
critical section.
The goal of srcu_readers_active_idx_check() is to verify that there were zero
active readers on the inactive index at some time during its execution. To do
this, it totals the unlock counts, executes a memory barrier, totals the lock
counts, and takes the difference. This difference counts the readers that are
active when srcu_readers_lock_idx() gets to their CPU, plus the readers that
completed after srcu_readers_unlock_idx() and before srcu_readers_lock_idx().
If the true (infinite precision) value of the difference is zero, then there
were no active readers at some point while srcu_readers_lock_idx() is running.
However, the difference is only stored in a long, so there is a potential for
overflow if too many readers complete during srcu_readers_active_idx_check().
For example, let's say there are three threads, each running on their own CPU:
int data, flag;
struct srcu_struct *sp = /* ... */;
Thread 0:
data = 1;
synchronize_srcu(sp);
flag = 1;
Thread 1:
int data_r, flag_r;
int idx = srcu_read_lock(sp);
data_r = data;
flag_r = flag;
srcu_read_unlock(sp, idx);
BUG_ON(flag_r == 1 && data_r == 0);
Thread 2:
while (true) {
int idx = srcu_read_lock(sp);
srcu_read_unlock(sp, idx);
}
Let's take the following execution order. Thread 1 increments
the CPU 1 version of sp->lock_count[0], sets idx to zero, and loads data (0)
into data_r. Thread 0 then sets data to be 1, verifies that there are no
readers on index 1, and increments sp->completed, but the CPU actually doesn't
preform the last operation, putting it off until the next memory barrier. Thread
0 then calls srcu_readers_active_idx_check() on index 0, which runs
srcu_readers_unlock_idx() (returning 0). Right after srcu_readers_unlock_idx()
completes, thread 2 starts running. Since Thread 0 hasn't actually incremented
sp->completed in any way that is visible to thread 2, srcu_read_lock() will
still return 0. Thread 2 can then run for ULONG_MAX iterations, setting
the CPU 2 version of sp->unlock_count[0] to ULONG_MAX. CPU 0 then finally gets
around to incrementing sp->completed, runs its memory barrier, and then reads
the lock counts: 1, 0, ULONG_MAX. The total of ULONG_MAX+1 will overflow to 0
and compare equal with earlier unlock count. Thread 0 will then think that the
grace period is over and set flag to one. Thread 1 can then read flag (1) into
flag_r and run srcu_read_unlock(). The BUG_ON statement will then fail.
Although ULONG_MAX readers completed during srcu_readers_active_idx_check(),
there were at most 2 active readers at any time, so this program doesn't run
into any limit.
I hope that was clear enough.
Thanks,
Lance
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH] srcu: Implement more-efficient reader counts
2017-01-24 0:53 ` Lance Roy
@ 2017-01-24 1:57 ` Paul E. McKenney
2017-01-24 3:26 ` Lance Roy
0 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 1:57 UTC (permalink / raw)
To: Lance Roy
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Mon, Jan 23, 2017 at 04:53:34PM -0800, Lance Roy wrote:
> On Mon, 23 Jan 2017 16:42:52 -0800
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
>
> > On Mon, Jan 23, 2017 at 01:35:18PM -0800, Lance Roy wrote:
> > > SRCU uses two per-cpu counters: a nesting counter to count the number of
> > > active critical sections, and a sequence counter to ensure that the nesting
> > > counters don't change while they are being added together in
> > > srcu_readers_active_idx_check().
> > >
> > > This patch instead uses per-cpu lock and unlock counters. Because both
> > > counters only increase and srcu_readers_active_idx_check() reads the unlock
> > > counter before the lock counter, this achieves the same end without having
> > > to increment two different counters in srcu_read_lock(). This also saves a
> > > smp_mb() in srcu_readers_active_idx_check().
> > >
> > > Possible bug: There is no guarantee that the lock counter won't overflow
> > > during srcu_readers_active_idx_check(), as there are no memory barriers
> > > around srcu_flip() (see comment in srcu_readers_active_idx_check() for
> > > details). However, this problem was already present before this patch.
> > >
> > > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > Signed-off-by: Lance Roy <ldr709@gmail.com>
> > > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > > Cc: Peter Zijlstra <peterz@infradead.org>
> >
> > OK, this only has differences only in the comments, so I can reasonably
> > substitute it, even this near the next merge window.
> >
> > But let's talk about the potential overflow. If I understand correctly,
> > for this to happen, there needs to be ULONG_MAX/2 or thereabouts
> > srcu_read_lock() calls without matching srcu_read_unlock() calls.
> > Let's focus on 32-bit systems for the moment.
> >
> > Lockdep allows at most 48 locks held at a given time by any single task,
> > and RCU does not pass in a non-NULL nest_lock -- if you acquire more than
> > that, a lockdep kernel will splat. Each task has at least one 4K page of
> > kernel stack, so there can be at most 1,048,576 tasks (actually quite a
> > bit fewer due to the size of the task_struct and so on). This allows
> > at most 50,331,648 unmatched srcu_read_lock() calls in the system,
> > which is not sufficient to cause overflow.
> >
> > Or am I missing something here?
> >
> > Thanx, Paul
>
> It isn't the nest that is the problem. Here is a previous email I wrote
> describing the problem:
>
>
> The trouble is that disabling preemption is not enough to ensure that there
> is at most one srcu_read_lock() call per CPU that missed the srcu_flip().
>
> Define a reader to be an SRCU lock+unlock pair. A reader is called active if it
> has incremented ->lock_count[] but hasn't incremented ->unlock_count[] yet, and
> completed if it has incremented ->unlock_count[]. I think that we only want to
> limit the number of active readers and the number of CPUs. In particular, I
> don't think there should be a limit on the rate of completion of read side
> critical section.
>
> The goal of srcu_readers_active_idx_check() is to verify that there were zero
> active readers on the inactive index at some time during its execution. To do
> this, it totals the unlock counts, executes a memory barrier, totals the lock
> counts, and takes the difference. This difference counts the readers that are
> active when srcu_readers_lock_idx() gets to their CPU, plus the readers that
> completed after srcu_readers_unlock_idx() and before srcu_readers_lock_idx().
> If the true (infinite precision) value of the difference is zero, then there
> were no active readers at some point while srcu_readers_lock_idx() is running.
> However, the difference is only stored in a long, so there is a potential for
> overflow if too many readers complete during srcu_readers_active_idx_check().
>
> For example, let's say there are three threads, each running on their own CPU:
>
> int data, flag;
> struct srcu_struct *sp = /* ... */;
>
> Thread 0:
> data = 1;
> synchronize_srcu(sp);
> flag = 1;
>
> Thread 1:
> int data_r, flag_r;
> int idx = srcu_read_lock(sp);
> data_r = data;
> flag_r = flag;
> srcu_read_unlock(sp, idx);
> BUG_ON(flag_r == 1 && data_r == 0);
>
> Thread 2:
> while (true) {
> int idx = srcu_read_lock(sp);
> srcu_read_unlock(sp, idx);
> }
>
> Let's take the following execution order. Thread 1 increments
> the CPU 1 version of sp->lock_count[0], sets idx to zero, and loads data (0)
> into data_r. Thread 0 then sets data to be 1, verifies that there are no
> readers on index 1, and increments sp->completed, but the CPU actually doesn't
> preform the last operation, putting it off until the next memory barrier. Thread
> 0 then calls srcu_readers_active_idx_check() on index 0, which runs
> srcu_readers_unlock_idx() (returning 0). Right after srcu_readers_unlock_idx()
> completes, thread 2 starts running. Since Thread 0 hasn't actually incremented
> sp->completed in any way that is visible to thread 2, srcu_read_lock() will
> still return 0. Thread 2 can then run for ULONG_MAX iterations, setting
> the CPU 2 version of sp->unlock_count[0] to ULONG_MAX. CPU 0 then finally gets
> around to incrementing sp->completed, runs its memory barrier, and then reads
> the lock counts: 1, 0, ULONG_MAX. The total of ULONG_MAX+1 will overflow to 0
> and compare equal with earlier unlock count. Thread 0 will then think that the
> grace period is over and set flag to one. Thread 1 can then read flag (1) into
> flag_r and run srcu_read_unlock(). The BUG_ON statement will then fail.
>
> Although ULONG_MAX readers completed during srcu_readers_active_idx_check(),
> there were at most 2 active readers at any time, so this program doesn't run
> into any limit.
>
> I hope that was clear enough.
Yeah, we did have this same conversation awhile back, didn't we?
Back then, did I think to ask if this could be minimized or even prevented
by adding memory barriers appropriately? ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering
2017-01-23 8:12 ` Michael Ellerman
@ 2017-01-24 2:45 ` Paul E. McKenney
0 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 2:45 UTC (permalink / raw)
To: Michael Ellerman
Cc: Ingo Molnar, boqun.feng, bobby.prani, peterz, fweisbec, dvhart,
jiangshanlai, linux-kernel, rostedt, josh, dhowells, edumazet,
mathieu.desnoyers, oleg, dipankar, will.deacon, akpm,
linuxppc-dev, tglx
On Mon, Jan 23, 2017 at 07:12:03PM +1100, Michael Ellerman wrote:
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> writes:
>
> > On Sat, Jan 14, 2017 at 11:54:17AM -0800, Paul E. McKenney wrote:
> >> On Sat, Jan 14, 2017 at 10:35:50AM +0100, Ingo Molnar wrote:
> >> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> >
> > [ . . . ]
> >
> >> > > + */
> >> > > +#ifdef CONFIG_PPC
> >> > > +#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> >> > > +#else /* #ifdef CONFIG_PPC */
> >> > > +#define smp_mb__after_unlock_lock() do { } while (0)
> >> > > +#endif /* #else #ifdef CONFIG_PPC */
> >> >
> >> > Yeah, so I realize that this was pre-existing code, but putting CONFIG_$ARCH
> >> > #ifdefs into generic headers is generally frowned upon.
> >> >
> >> > The canonical approach would be either to define a helper Kconfig variable that
> >> > can be set by PPC (but other architectures don't need to set it), or to expose a
> >> > suitable macro (function) for architectures to define in their barrier.h arch
> >> > header file.
> >>
> >> Very well, I will add a separate commit for this. 4.11 OK?
> >
> > Does the patch below seem reasonable?
> >
> > Thanx, Paul
> >
> > ------------------------------------------------------------------------
> >
> > commit 271c0601237c41a279f975563e13837bace0df03
> > Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Date: Sat Jan 14 13:32:50 2017 -0800
> >
> > rcu: Make arch select smp_mb__after_unlock_lock() strength
> >
> > The definition of smp_mb__after_unlock_lock() is currently smp_mb()
> > for CONFIG_PPC and a no-op otherwise. It would be better to instead
> > provide an architecture-selectable Kconfig option, and select the
> > strength of smp_mb__after_unlock_lock() based on that option. This
> > commit therefore creates CONFIG_ARCH_WEAK_RELACQ, has PPC select it,
> > and bases the definition of smp_mb__after_unlock_lock() on this new
> > CONFIG_ARCH_WEAK_RELACQ Kconfig option.
> >
> > Reported-by: Ingo Molnar <mingo@kernel.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
> > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > Cc: Paul Mackerras <paulus@samba.org>
> > Cc: Michael Ellerman <mpe@ellerman.id.au>
> > Cc: <linuxppc-dev@lists.ozlabs.org>
>
> Personally I'd call it ARCH_WEAK_RELEASE_ACQUIRE, which is longer but
> clearer I think. But it's not a big deal, so which ever you prefer.
ARCH_WEAK_RELEASE_ACQUIRE it is!
> Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Applied, thank you!
Thanx, Paul
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH] srcu: Implement more-efficient reader counts
2017-01-24 1:57 ` Paul E. McKenney
@ 2017-01-24 3:26 ` Lance Roy
2017-01-24 17:07 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-24 3:26 UTC (permalink / raw)
To: Paul E. McKenney, bobby.prani
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg
> Yeah, we did have this same conversation awhile back, didn't we?
>
> Back then, did I think to ask if this could be minimized or even prevented
> by adding memory barriers appropriately? ;-)
>
> Thanx, Paul
Yes, it can be fixed by adding a memory barrier after incrementing ->completed
inside srcu_flip(). The upper limit on NR_CPUS turns out to be more complicated
than this, as it needs to deal with highly nested read side critical sections
mixed with the critical section loops, but only the one memory barrier should
be necessary.
Thanks,
Lance
Begin forwarded message:
Date: Fri, 18 Nov 2016 16:54:42 -0800
From: Lance Roy <ldr709@gmail.com>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, jiangshanlai@gmail.com,
dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org,
rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com,
dvhart@linux.intel.com, fweisbec@gmail.com, oleg@redhat.com,
bobby.prani@gmail.com Subject: Re: [RFC PATCH] SRCU: More efficient reader
counts.
On Fri, 18 Nov 2016 15:13:45 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> On Fri, Nov 18, 2016 at 12:33:00PM -0800, Lance Roy wrote:
> > The trouble is that disabling preemption is not enough to ensure that there
> > is at most one srcu_read_lock() call per CPU that missed the srcu_flip().
> >
> > Define a reader to be an SRCU lock+unlock pair. A reader is called active
> > if it has incremented ->lock_count[] but hasn't incremented
> > ->unlock_count[] yet, and completed if it has incremented
> > ->unlock_count[]. I think that we only want to limit the number of active
> > readers and the number of CPUs. In particular, I don't think there should
> > be a limit on the rate of completion of read side critical section.
> >
> > The goal of srcu_readers_active_idx_check() is to verify that there were
> > zero active readers on the inactive index at some time during its
> > execution. To do this, it totals the unlock counts, executes a memory
> > barrier, totals the lock counts, and takes the difference. This difference
> > counts the readers that are active when srcu_readers_lock_idx() gets to
> > their CPU, plus the readers that completed after srcu_readers_unlock_idx()
> > and before srcu_readers_lock_idx(). If the true (infinite precision) value
> > of the difference is zero, then there were no active readers at some point
> > while srcu_readers_lock_idx() is running. However, the difference is only
> > stored in a long, so there is a potential for overflow if too many readers
> > complete during srcu_readers_active_idx_check().
> >
> > For example, let's say there are three threads, each running on their own
> > CPU:
> >
> > int data, flag;
> > struct srcu_struct *sp = /* ... */;
> >
> > Thread 0:
> > data = 1;
> > synchronize_srcu(sp);
> > flag = 1;
> >
> > Thread 1:
> > int data_r, flag_r;
> > int idx = srcu_read_lock(sp);
> > data_r = data;
> > flag_r = flag;
> > srcu_read_unlock(sp, idx);
> > BUG_ON(flag_r == 1 && data_r == 0);
> >
> > Thread 2:
> > while (true) {
> > int idx = srcu_read_lock(sp);
> > srcu_read_unlock(sp, idx);
> > }
> >
> > Let's take the following execution order. Thread 1 increments
> > the CPU 1 version of sp->lock_count[0], sets idx to zero, and loads data (0)
> > into data_r. Thread 0 then sets data to be 1, verifies that there are no
> > readers on index 1, and increments sp->completed, but the CPU actually
> > doesn't preform the last operation, putting it off until the next memory
> > barrier. Thread 0 then calls srcu_readers_active_idx_check() on index 0,
> > which runs srcu_readers_unlock_idx() (returning 0). Right after
> > srcu_readers_unlock_idx() completes, thread 2 starts running. Since Thread
> > 0 hasn't actually incremented sp->completed in any way that is visible to
> > thread 2, srcu_read_lock() will still return 0. Thread 2 can then run for
> > ULONG_MAX iterations, setting the CPU 2 version of sp->unlock_count[0] to
> > ULONG_MAX. CPU 0 then finally gets around to incrementing sp->completed,
> > runs its memory barrier, and then reads the lock counts: 1, 0, ULONG_MAX.
> > The total of ULONG_MAX+1 will overflow to 0 and compare equal with earlier
> > unlock count. Thread 0 will then think that the grace period is over and
> > set flag to one. Thread 1 can then read flag (1) into flag_r and run
> > srcu_read_unlock(). The BUG_ON statement will then fail.
> >
> > Although ULONG_MAX readers completed during srcu_readers_active_idx_check(),
> > there were at most 2 active readers at any time, so this program doesn't run
> > into any limit.
> >
> > I hope that was clear enough.
>
> Indeed it is!
>
> So adding a full memory barrier immediately after the srcu_flip() should
> prevent this, because if the updater failed to see an unlock increment,
> the second following lock for that CPU/task would be guaranteed to see
> the flip. Or am I still missing something?
Yes, adding a full memory barrier after srcu_flip() prevents this problem.
> Is there a sequence of events that requires a full memory barrier
> before the srcu_flip()?
I am now unsure if a memory barrier before srcu_flip() is necessary. I thought
that it would be needed to prevent the CPU from preforming the increment early,
but I have just noticed that srcu_advance_batches() will return early if the
first try_check_zero() fails, creating a control dependency. I think that this
control dependency should be enough to prevent the problem from occurring.
One interesting thing about this version is that there is only an address
dependency between the load of ->completed and the increment of ->load_count[].
This means that without an smp_read_barrier_depends() between the two, a reader
could use the new value of ->completed to increment ->load_count[], before
actually reading ->completed. (I was surprised that this ordering doesn't come
for free, as it seems like violating it would require speculative writes.) It
doesn't matter much if the dependency barrier is actually there, because
leaving it out just means at most 1 more spurious increment per CPU.
I think that with the added synchronization in srcu_flip() there would be at
most 3 * NR_CPUS readers that start and complete inside one
srcu_readers_active_idx_check(). With the read side dependency barrier this
would drop to 2 * NR_CPUS.
Thanks,
Lance
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH] srcu: Implement more-efficient reader counts
2017-01-24 3:26 ` Lance Roy
@ 2017-01-24 17:07 ` Paul E. McKenney
0 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 17:07 UTC (permalink / raw)
To: Lance Roy
Cc: bobby.prani, linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg
On Mon, Jan 23, 2017 at 07:26:45PM -0800, Lance Roy wrote:
> > Yeah, we did have this same conversation awhile back, didn't we?
> >
> > Back then, did I think to ask if this could be minimized or even prevented
> > by adding memory barriers appropriately? ;-)
> >
> > Thanx, Paul
>
> Yes, it can be fixed by adding a memory barrier after incrementing ->completed
> inside srcu_flip(). The upper limit on NR_CPUS turns out to be more complicated
> than this, as it needs to deal with highly nested read side critical sections
> mixed with the critical section loops, but only the one memory barrier should
> be necessary.
Something like this, then?
Thanx, Paul
------------------------------------------------------------------------
commit 35be9e413dde662fc9661352e595105ac4b0b167
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Tue Jan 24 08:51:34 2017 -0800
srcu: Reduce probability of SRCU ->unlock_count[] counter overflow
Because there are no memory barriers between the srcu_flip() ->completed
increment and the summation of the read-side ->unlock_count[] counters,
both the compiler and the CPU can reorder the summation with the
->completed increment. If the updater is preempted long enough during
this process, the read-side counters could overflow, resulting in a
too-short grace period.
This commit therefore adds a memory barrier just after the ->completed
increment, ensuring that if the summation misses an increment of
->unlock_count[] from __srcu_read_unlock(), the next __srcu_read_lock()
will see the new value of ->completed, thus bounding the number of
->unlock_count[] increments that can be missed to NR_CPUS. The actual
overflow computation is more complex due to the possibility of nesting
of __srcu_read_lock().
Reported-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index d3378ceb9762..aefe3ab20a6a 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -337,7 +337,16 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
*/
static void srcu_flip(struct srcu_struct *sp)
{
- sp->completed++;
+ WRITE_ONCE(sp->completed, sp->completed + 1);
+
+ /*
+ * Ensure that if the updater misses an __srcu_read_unlock()
+ * increment, that task's next __srcu_read_lock() will see the
+ * above counter update. Note that both this memory barrier
+ * and the one in srcu_readers_active_idx_check() provide the
+ * guarantee for __srcu_read_lock().
+ */
+ smp_mb(); /* D */ /* Pairs with C. */
}
/*
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
` (2 preceding siblings ...)
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
@ 2017-01-24 22:00 ` Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts Paul E. McKenney
` (3 more replies)
3 siblings, 4 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 22:00 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani
Hello!
This series provides v3 updates to SRCU:
1. This is a rewrite of the algorithm simplifying reader-count
tracking. Algorithm courtesy of Mathieu Desnoyers, implementation
courtesy of Lance Roy.
2. Force full grace-period ordering in SRCU.
3. Add CBMC-based formal verification for SRCU, courtesy of Lance Roy.
Updates since v2:
o Fix memory-barrier problems noted by Lance Roy.
o Add memory barrier to lower probability of counter overflow,
also noted by Lance Roy.
Updates since v1:
o Applied Ingo Molnar feedback.
o Fix some checkpatch issues.
Thanx, Paul
------------------------------------------------------------------------
include/linux/rcupdate.h | 12
include/linux/srcu.h | 10
kernel/rcu/rcutorture.c | 19
kernel/rcu/srcu.c | 144 +--
kernel/rcu/tree.h | 12
tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h | 155 ++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 ++++++++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h | 16
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c | 13
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 27
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h | 220 +++++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h | 92 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c | 78 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h | 58 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile | 11
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail | 1
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c | 72 +
tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh | 102 ++
34 files changed, 1679 insertions(+), 100 deletions(-)
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
@ 2017-01-24 22:00 ` Paul E. McKenney
2017-01-25 18:17 ` Lance Roy
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 2/4] srcu: Force full grace-period ordering Paul E. McKenney
` (2 subsequent siblings)
3 siblings, 1 reply; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 22:00 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
SRCU uses two per-cpu counters: a nesting counter to count the number of
active critical sections, and a sequence counter to ensure that the nesting
counters don't change while they are being added together in
srcu_readers_active_idx_check().
This patch instead uses per-cpu lock and unlock counters. Because both
counters only increase and srcu_readers_active_idx_check() reads the unlock
counter before the lock counter, this achieves the same end without having
to increment two different counters in srcu_read_lock(). This also saves a
smp_mb() in srcu_readers_active_idx_check().
A possible problem with this patch is that it can only handle
ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
handle up to ULONG_MAX.
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
include/linux/srcu.h | 10 ++--
kernel/rcu/rcutorture.c | 19 +++++++-
kernel/rcu/srcu.c | 123 ++++++++++++++++++------------------------------
3 files changed, 67 insertions(+), 85 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index dc8eb63c6568..a598cf3ac70c 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -33,9 +33,9 @@
#include <linux/rcupdate.h>
#include <linux/workqueue.h>
-struct srcu_struct_array {
- unsigned long c[2];
- unsigned long seq[2];
+struct srcu_array {
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
};
struct rcu_batch {
@@ -46,7 +46,7 @@ struct rcu_batch {
struct srcu_struct {
unsigned long completed;
- struct srcu_struct_array __percpu *per_cpu_ref;
+ struct srcu_array __percpu *per_cpu_ref;
spinlock_t queue_lock; /* protect ->batch_queue, ->running */
bool running;
/* callbacks just queued */
@@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
* See include/linux/percpu-defs.h for the rules on per-CPU variables.
*/
#define __DEFINE_SRCU(name, is_static) \
- static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
+ static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 87c51225ceec..d81345be730e 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
pr_alert("%s%s per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
+ unsigned long l0, l1;
+ unsigned long u0, u1;
long c0, c1;
+ struct srcu_array *counts = per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
- c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
- c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
+ u0 = counts->unlock_count[!idx];
+ u1 = counts->unlock_count[idx];
+
+ /*
+ * Make sure that a lock is always counted if the corresponding
+ * unlock is counted.
+ */
+ smp_rmb();
+
+ l0 = counts->lock_count[!idx];
+ l1 = counts->lock_count[idx];
+
+ c0 = l0 - u0;
+ c1 = l1 - u1;
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 9b9cdd549caa..ddabf5fbf562 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
rcu_batch_init(&sp->batch_check1);
rcu_batch_init(&sp->batch_done);
INIT_DELAYED_WORK(&sp->work, process_srcu);
- sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
+ sp->per_cpu_ref = alloc_percpu(struct srcu_array);
return sp->per_cpu_ref ? 0 : -ENOMEM;
}
@@ -141,114 +141,78 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
- * Returns approximate total of the readers' ->seq[] values for the
+ * Returns approximate total of the readers' ->lock_count[] values for the
* rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[idx]);
}
return sum;
}
/*
- * Returns approximate number of readers active on the specified rank
- * of the per-CPU ->c[] counters.
+ * Returns approximate total of the readers' ->unlock_count[] values for the
+ * rank of per-CPU counters specified by idx.
*/
-static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
+static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
{
int cpu;
unsigned long sum = 0;
- unsigned long t;
for_each_possible_cpu(cpu) {
- t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
- sum += t;
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->unlock_count[idx]);
}
return sum;
}
/*
* Return true if the number of pre-existing readers is determined to
- * be stably zero. An example unstable zero can occur if the call
- * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
- * but due to task migration, sees the corresponding __srcu_read_unlock()
- * decrement. This can happen because srcu_readers_active_idx() takes
- * time to sum the array, and might in fact be interrupted or preempted
- * partway through the summation.
+ * be zero.
*/
static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
{
- unsigned long seq;
+ unsigned long unlocks;
- seq = srcu_readers_seq_idx(sp, idx);
+ unlocks = srcu_readers_unlock_idx(sp, idx);
/*
- * The following smp_mb() A pairs with the smp_mb() B located in
- * __srcu_read_lock(). This pairing ensures that if an
- * __srcu_read_lock() increments its counter after the summation
- * in srcu_readers_active_idx(), then the corresponding SRCU read-side
- * critical section will see any changes made prior to the start
- * of the current SRCU grace period.
+ * Make sure that a lock is always counted if the corresponding unlock
+ * is counted. Needs to be a smp_mb() as the read side may contain a
+ * read from a variable that is written to before the synchronize_srcu()
+ * in the write side. In this case smp_mb()s A and B act like the store
+ * buffering pattern.
*
- * Also, if the above call to srcu_readers_seq_idx() saw the
- * increment of ->seq[], then the call to srcu_readers_active_idx()
- * must see the increment of ->c[].
+ * This smp_mb() also pairs with smp_mb() C to prevent writes after the
+ * synchronize_srcu() from being executed before the grace period ends.
*/
smp_mb(); /* A */
/*
- * Note that srcu_readers_active_idx() can incorrectly return
- * zero even though there is a pre-existing reader throughout.
- * To see this, suppose that task A is in a very long SRCU
- * read-side critical section that started on CPU 0, and that
- * no other reader exists, so that the sum of the counters
- * is equal to one. Then suppose that task B starts executing
- * srcu_readers_active_idx(), summing up to CPU 1, and then that
- * task C starts reading on CPU 0, so that its increment is not
- * summed, but finishes reading on CPU 2, so that its decrement
- * -is- summed. Then when task B completes its sum, it will
- * incorrectly get zero, despite the fact that task A has been
- * in its SRCU read-side critical section the whole time.
+ * If the locks are the same as the unlocks, then there must have
+ * been no readers on this index at some time in between. This does not
+ * mean that there are no more readers, as one could have read the
+ * current index but not have incremented the lock counter yet.
*
- * We therefore do a validation step should srcu_readers_active_idx()
- * return zero.
+ * Note that there can be at most NR_CPUS worth of readers using the old
+ * index that haven't incremented ->lock_count[] yet. Therefore, the
+ * sum of the ->lock_count[]s cannot increment enough times to overflow
+ * and end up equal the sum of the ->unlock_count[]s, as long as there
+ * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this does
+ * mean that systems having more than a billion or so CPUs need to be
+ * 64-bit systems.) Therefore, the only way that the return values of
+ * the two calls to srcu_readers_(un)lock_idx() can be equal is if there
+ * are no active readers using this index.
*/
- if (srcu_readers_active_idx(sp, idx) != 0)
- return false;
-
- /*
- * The remainder of this function is the validation step.
- * The following smp_mb() D pairs with the smp_mb() C in
- * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
- * by srcu_readers_active_idx() above, then any destructive
- * operation performed after the grace period will happen after
- * the corresponding SRCU read-side critical section.
- *
- * Note that there can be at most NR_CPUS worth of readers using
- * the old index, which is not enough to overflow even a 32-bit
- * integer. (Yes, this does mean that systems having more than
- * a billion or so CPUs need to be 64-bit systems.) Therefore,
- * the sum of the ->seq[] counters cannot possibly overflow.
- * Therefore, the only way that the return values of the two
- * calls to srcu_readers_seq_idx() can be equal is if there were
- * no increments of the corresponding rank of ->seq[] counts
- * in the interim. But the missed-increment scenario laid out
- * above includes an increment of the ->seq[] counter by
- * the corresponding __srcu_read_lock(). Therefore, if this
- * scenario occurs, the return values from the two calls to
- * srcu_readers_seq_idx() will differ, and thus the validation
- * step below suffices.
- */
- smp_mb(); /* D */
-
- return srcu_readers_seq_idx(sp, idx) == seq;
+ return srcu_readers_lock_idx(sp, idx) == unlocks;
}
/**
@@ -266,8 +230,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
unsigned long sum = 0;
for_each_possible_cpu(cpu) {
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
- sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
+ struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
+
+ sum += READ_ONCE(cpuc->lock_count[0]);
+ sum += READ_ONCE(cpuc->lock_count[1]);
+ sum -= READ_ONCE(cpuc->unlock_count[0]);
+ sum -= READ_ONCE(cpuc->unlock_count[1]);
}
return sum;
}
@@ -298,9 +266,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
int idx;
idx = READ_ONCE(sp->completed) & 0x1;
- __this_cpu_inc(sp->per_cpu_ref->c[idx]);
+ __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
smp_mb(); /* B */ /* Avoid leaking the critical section. */
- __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
return idx;
}
EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -314,7 +281,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
void __srcu_read_unlock(struct srcu_struct *sp, int idx)
{
smp_mb(); /* C */ /* Avoid leaking the critical section. */
- this_cpu_dec(sp->per_cpu_ref->c[idx]);
+ this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
}
EXPORT_SYMBOL_GPL(__srcu_read_unlock);
@@ -349,7 +316,7 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
/*
* Increment the ->completed counter so that future SRCU readers will
- * use the other rank of the ->c[] and ->seq[] arrays. This allows
+ * use the other rank of the ->(un)lock_count[] arrays. This allows
* us to wait for pre-existing readers in a starvation-free manner.
*/
static void srcu_flip(struct srcu_struct *sp)
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 tip/core/rcu 2/4] srcu: Force full grace-period ordering
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts Paul E. McKenney
@ 2017-01-24 22:00 ` Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 3/4] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 4/4] srcu: Reduce probability of SRCU ->unlock_count[] counter overflow Paul E. McKenney
3 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 22:00 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Paul E. McKenney
If a process invokes synchronize_srcu(), is delayed just the right amount
of time, and thus does not sleep when waiting for the grace period to
complete, there is no ordering between the end of the grace period and
the code following the synchronize_srcu(). Similarly, there can be a
lack of ordering between the end of the SRCU grace period and callback
invocation.
This commit adds the necessary ordering.
Reported-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
[ paulmck: Further smp_mb() adjustment per email with Lance Roy. ]
---
include/linux/rcupdate.h | 12 ++++++++++++
kernel/rcu/srcu.c | 10 ++++++++--
kernel/rcu/tree.h | 12 ------------
3 files changed, 20 insertions(+), 14 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 01f71e1d2e94..6ade6a52d9d4 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1161,5 +1161,17 @@ do { \
ftrace_dump(oops_dump_mode); \
} while (0)
+/*
+ * Place this after a lock-acquisition primitive to guarantee that
+ * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
+ * if the UNLOCK and LOCK are executed by the same CPU or if the
+ * UNLOCK and LOCK operate on the same lock variable.
+ */
+#ifdef CONFIG_PPC
+#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
+#else /* #ifdef CONFIG_PPC */
+#define smp_mb__after_unlock_lock() do { } while (0)
+#endif /* #else #ifdef CONFIG_PPC */
+
#endif /* __LINUX_RCUPDATE_H */
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index ddabf5fbf562..fcd07eda95a3 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -359,6 +359,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
head->next = NULL;
head->func = func;
spin_lock_irqsave(&sp->queue_lock, flags);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
rcu_batch_queue(&sp->batch_queue, head);
if (!sp->running) {
sp->running = true;
@@ -392,6 +393,7 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
head->next = NULL;
head->func = wakeme_after_rcu;
spin_lock_irq(&sp->queue_lock);
+ smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
if (!sp->running) {
/* steal the processing owner */
sp->running = true;
@@ -411,8 +413,11 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
spin_unlock_irq(&sp->queue_lock);
}
- if (!done)
+ if (!done) {
wait_for_completion(&rcu.completion);
+ smp_mb(); /* Caller's later accesses after GP. */
+ }
+
}
/**
@@ -580,7 +585,8 @@ static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
/*
* Invoke a limited number of SRCU callbacks that have passed through
* their grace period. If there are more to do, SRCU will reschedule
- * the workqueue.
+ * the workqueue. Note that needed memory barriers have been executed
+ * in this task's context by srcu_readers_active_idx_check().
*/
static void srcu_invoke_callbacks(struct srcu_struct *sp)
{
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index fe98dd24adf8..abcc25bdcb29 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -688,18 +688,6 @@ static inline void rcu_nocb_q_lengths(struct rcu_data *rdp, long *ql, long *qll)
#endif /* #ifdef CONFIG_RCU_TRACE */
/*
- * Place this after a lock-acquisition primitive to guarantee that
- * an UNLOCK+LOCK pair act as a full barrier. This guarantee applies
- * if the UNLOCK and LOCK are executed by the same CPU or if the
- * UNLOCK and LOCK operate on the same lock variable.
- */
-#ifdef CONFIG_PPC
-#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
-#else /* #ifdef CONFIG_PPC */
-#define smp_mb__after_unlock_lock() do { } while (0)
-#endif /* #else #ifdef CONFIG_PPC */
-
-/*
* Wrappers for the rcu_node::lock acquire and release.
*
* Because the rcu_nodes form a tree, the tree traversal locking will observe
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 tip/core/rcu 3/4] rcutorture: Add CBMC-based formal verification for SRCU
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 2/4] srcu: Force full grace-period ordering Paul E. McKenney
@ 2017-01-24 22:00 ` Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 4/4] srcu: Reduce probability of SRCU ->unlock_count[] counter overflow Paul E. McKenney
3 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 22:00 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Lance Roy, Paul E. McKenney
From: Lance Roy <ldr709@gmail.com>
This commit creates a formal/srcu-cbmc directory containing scripts that
pull SRCU in from the source code, filter it to remove things that CBMC
cannot handle, and run a series of verifications on it. This has a number
of shortcomings:
1. It does not yet hook into the upper-level self-test Makefiles.
2. It tests only a single scenario, store buffering.
3. There is no gcc-based syntax-error prefiltering.
Nevertheless, it does fully verify a piece of SRCU under a moderately
weak memory model (PSO).
Signed-off-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
.../rcutorture/formal/srcu-cbmc/.gitignore | 1 +
.../selftests/rcutorture/formal/srcu-cbmc/Makefile | 16 +
.../formal/srcu-cbmc/empty_includes/linux/delay.h | 0
.../formal/srcu-cbmc/empty_includes/linux/export.h | 0
.../formal/srcu-cbmc/empty_includes/linux/mutex.h | 0
.../formal/srcu-cbmc/empty_includes/linux/percpu.h | 0
.../srcu-cbmc/empty_includes/linux/preempt.h | 0
.../srcu-cbmc/empty_includes/linux/rcupdate.h | 0
.../formal/srcu-cbmc/empty_includes/linux/sched.h | 0
.../formal/srcu-cbmc/empty_includes/linux/smp.h | 0
.../srcu-cbmc/empty_includes/linux/workqueue.h | 0
.../srcu-cbmc/empty_includes/uapi/linux/types.h | 0
.../formal/srcu-cbmc/include/linux/.gitignore | 1 +
.../formal/srcu-cbmc/include/linux/kconfig.h | 1 +
.../formal/srcu-cbmc/include/linux/types.h | 155 +++++++++
.../rcutorture/formal/srcu-cbmc/modify_srcu.awk | 375 +++++++++++++++++++++
.../rcutorture/formal/srcu-cbmc/src/assume.h | 16 +
.../rcutorture/formal/srcu-cbmc/src/barriers.h | 41 +++
.../rcutorture/formal/srcu-cbmc/src/bug_on.h | 13 +
.../formal/srcu-cbmc/src/combined_source.c | 13 +
.../rcutorture/formal/srcu-cbmc/src/config.h | 27 ++
.../rcutorture/formal/srcu-cbmc/src/include_srcu.c | 31 ++
.../rcutorture/formal/srcu-cbmc/src/int_typedefs.h | 33 ++
.../rcutorture/formal/srcu-cbmc/src/locks.h | 220 ++++++++++++
.../rcutorture/formal/srcu-cbmc/src/misc.c | 11 +
.../rcutorture/formal/srcu-cbmc/src/misc.h | 58 ++++
.../rcutorture/formal/srcu-cbmc/src/percpu.h | 92 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.c | 78 +++++
.../rcutorture/formal/srcu-cbmc/src/preempt.h | 58 ++++
.../formal/srcu-cbmc/src/simple_sync_srcu.c | 50 +++
.../rcutorture/formal/srcu-cbmc/src/workqueues.h | 102 ++++++
.../srcu-cbmc/tests/store_buffering/.gitignore | 1 +
.../srcu-cbmc/tests/store_buffering/Makefile | 11 +
.../tests/store_buffering/assert_end.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force2.fail | 1 +
.../srcu-cbmc/tests/store_buffering/force3.fail | 1 +
.../srcu-cbmc/tests/store_buffering/main.pass | 0
.../formal/srcu-cbmc/tests/store_buffering/test.c | 72 ++++
.../formal/srcu-cbmc/tests/test_script.sh | 102 ++++++
40 files changed, 1582 insertions(+)
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
create mode 100644 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
create mode 100755 tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
new file mode 100644
index 000000000000..712a3d41a325
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/.gitignore
@@ -0,0 +1 @@
+srcu.c
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
new file mode 100644
index 000000000000..16b01559fa55
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/Makefile
@@ -0,0 +1,16 @@
+all: srcu.c store_buffering
+
+LINUX_SOURCE = ../../../../../..
+
+modified_srcu_input = $(LINUX_SOURCE)/include/linux/srcu.h \
+ $(LINUX_SOURCE)/kernel/rcu/srcu.c
+
+modified_srcu_output = include/linux/srcu.h srcu.c
+
+include/linux/srcu.h: srcu.c
+
+srcu.c: modify_srcu.awk Makefile $(modified_srcu_input)
+ awk -f modify_srcu.awk $(modified_srcu_input) $(modified_srcu_output)
+
+store_buffering:
+ @cd tests/store_buffering; make
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/delay.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/export.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/mutex.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/percpu.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/preempt.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/rcupdate.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/sched.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/smp.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/linux/workqueue.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/empty_includes/uapi/linux/types.h
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
new file mode 100644
index 000000000000..1d016e66980a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/.gitignore
@@ -0,0 +1 @@
+srcu.h
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
new file mode 100644
index 000000000000..f2860dd1b407
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/kconfig.h
@@ -0,0 +1 @@
+#include <LINUX_SOURCE/linux/kconfig.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
new file mode 100644
index 000000000000..4a3d538fef12
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/include/linux/types.h
@@ -0,0 +1,155 @@
+/*
+ * This header has been modifies to remove definitions of types that
+ * are defined in standard userspace headers or are problematic for some
+ * other reason.
+ */
+
+#ifndef _LINUX_TYPES_H
+#define _LINUX_TYPES_H
+
+#define __EXPORTED_HEADERS__
+#include <uapi/linux/types.h>
+
+#ifndef __ASSEMBLY__
+
+#define DECLARE_BITMAP(name, bits) \
+ unsigned long name[BITS_TO_LONGS(bits)]
+
+typedef __u32 __kernel_dev_t;
+
+/* bsd */
+typedef unsigned char u_char;
+typedef unsigned short u_short;
+typedef unsigned int u_int;
+typedef unsigned long u_long;
+
+/* sysv */
+typedef unsigned char unchar;
+typedef unsigned short ushort;
+typedef unsigned int uint;
+typedef unsigned long ulong;
+
+#ifndef __BIT_TYPES_DEFINED__
+#define __BIT_TYPES_DEFINED__
+
+typedef __u8 u_int8_t;
+typedef __s8 int8_t;
+typedef __u16 u_int16_t;
+typedef __s16 int16_t;
+typedef __u32 u_int32_t;
+typedef __s32 int32_t;
+
+#endif /* !(__BIT_TYPES_DEFINED__) */
+
+typedef __u8 uint8_t;
+typedef __u16 uint16_t;
+typedef __u32 uint32_t;
+
+/* this is a special 64bit data type that is 8-byte aligned */
+#define aligned_u64 __u64 __attribute__((aligned(8)))
+#define aligned_be64 __be64 __attribute__((aligned(8)))
+#define aligned_le64 __le64 __attribute__((aligned(8)))
+
+/**
+ * The type used for indexing onto a disc or disc partition.
+ *
+ * Linux always considers sectors to be 512 bytes long independently
+ * of the devices real block size.
+ *
+ * blkcnt_t is the type of the inode's block count.
+ */
+#ifdef CONFIG_LBDAF
+typedef u64 sector_t;
+#else
+typedef unsigned long sector_t;
+#endif
+
+/*
+ * The type of an index into the pagecache.
+ */
+#define pgoff_t unsigned long
+
+/*
+ * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * by the DMA API.
+ *
+ * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
+ * bits wide. Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
+ * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
+ * so they don't care about the size of the actual bus addresses.
+ */
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+typedef u64 dma_addr_t;
+#else
+typedef u32 dma_addr_t;
+#endif
+
+#ifdef CONFIG_PHYS_ADDR_T_64BIT
+typedef u64 phys_addr_t;
+#else
+typedef u32 phys_addr_t;
+#endif
+
+typedef phys_addr_t resource_size_t;
+
+/*
+ * This type is the placeholder for a hardware interrupt number. It has to be
+ * big enough to enclose whatever representation is used by a given platform.
+ */
+typedef unsigned long irq_hw_number_t;
+
+typedef struct {
+ int counter;
+} atomic_t;
+
+#ifdef CONFIG_64BIT
+typedef struct {
+ long counter;
+} atomic64_t;
+#endif
+
+struct list_head {
+ struct list_head *next, *prev;
+};
+
+struct hlist_head {
+ struct hlist_node *first;
+};
+
+struct hlist_node {
+ struct hlist_node *next, **pprev;
+};
+
+/**
+ * struct callback_head - callback structure for use with RCU and task_work
+ * @next: next update requests in a list
+ * @func: actual update function to call after the grace period.
+ *
+ * The struct is aligned to size of pointer. On most architectures it happens
+ * naturally due ABI requirements, but some architectures (like CRIS) have
+ * weird ABI and we need to ask it explicitly.
+ *
+ * The alignment is required to guarantee that bits 0 and 1 of @next will be
+ * clear under normal conditions -- as long as we use call_rcu(),
+ * call_rcu_bh(), call_rcu_sched(), or call_srcu() to queue callback.
+ *
+ * This guarantee is important for few reasons:
+ * - future call_rcu_lazy() will make use of lower bits in the pointer;
+ * - the structure shares storage spacer in struct page with @compound_head,
+ * which encode PageTail() in bit 0. The guarantee is needed to avoid
+ * false-positive PageTail().
+ */
+struct callback_head {
+ struct callback_head *next;
+ void (*func)(struct callback_head *head);
+} __attribute__((aligned(sizeof(void *))));
+#define rcu_head callback_head
+
+typedef void (*rcu_callback_t)(struct rcu_head *head);
+typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
+
+/* clocksource cycle base type */
+typedef u64 cycle_t;
+
+#endif /* __ASSEMBLY__ */
+#endif /* _LINUX_TYPES_H */
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
new file mode 100755
index 000000000000..8ff89043d0a9
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/modify_srcu.awk
@@ -0,0 +1,375 @@
+#!/bin/awk -f
+
+# Modify SRCU for formal verification. The first argument should be srcu.h and
+# the second should be srcu.c. Outputs modified srcu.h and srcu.c into the
+# current directory.
+
+BEGIN {
+ if (ARGC != 5) {
+ print "Usange: input.h input.c output.h output.c" > "/dev/stderr";
+ exit 1;
+ }
+ h_output = ARGV[3];
+ c_output = ARGV[4];
+ ARGC = 3;
+
+ # Tokenize using FS and not RS as FS supports regular expressions. Each
+ # record is one line of source, except that backslashed lines are
+ # combined. Comments are treated as field separators, as are quotes.
+ quote_regexp="\"([^\\\\\"]|\\\\.)*\"";
+ comment_regexp="\\/\\*([^*]|\\*+[^*/])*\\*\\/|\\/\\/.*(\n|$)";
+ FS="([ \\\\\t\n\v\f;,.=(){}+*/<>&|^-]|\\[|\\]|" comment_regexp "|" quote_regexp ")+";
+
+ inside_srcu_struct = 0;
+ inside_srcu_init_def = 0;
+ srcu_init_param_name = "";
+ in_macro = 0;
+ brace_nesting = 0;
+ paren_nesting = 0;
+
+ # Allow the manipulation of the last field separator after has been
+ # seen.
+ last_fs = "";
+ # Whether the last field separator was intended to be output.
+ last_fs_print = 0;
+
+ # rcu_batches stores the initialization for each instance of struct
+ # rcu_batch
+
+ in_comment = 0;
+
+ outputfile = "";
+}
+
+{
+ prev_outputfile = outputfile;
+ if (FILENAME ~ /\.h$/) {
+ outputfile = h_output;
+ if (FNR != NR) {
+ print "Incorrect file order" > "/dev/stderr";
+ exit 1;
+ }
+ }
+ else
+ outputfile = c_output;
+
+ if (prev_outputfile && outputfile != prev_outputfile) {
+ new_outputfile = outputfile;
+ outputfile = prev_outputfile;
+ update_fieldsep("", 0);
+ outputfile = new_outputfile;
+ }
+}
+
+# Combine the next line into $0.
+function combine_line() {
+ ret = getline next_line;
+ if (ret == 0) {
+ # Don't allow two consecutive getlines at the end of the file
+ if (eof_found) {
+ print "Error: expected more input." > "/dev/stderr";
+ exit 1;
+ } else {
+ eof_found = 1;
+ }
+ } else if (ret == -1) {
+ print "Error reading next line of file" FILENAME > "/dev/stderr";
+ exit 1;
+ }
+ $0 = $0 "\n" next_line;
+}
+
+# Combine backslashed lines and multiline comments.
+function combine_backslashes() {
+ while (/\\$|\/\*([^*]|\*+[^*\/])*\**$/) {
+ combine_line();
+ }
+}
+
+function read_line() {
+ combine_line();
+ combine_backslashes();
+}
+
+# Print out field separators and update variables that depend on them. Only
+# print if p is true. Call with sep="" and p=0 to print out the last field
+# separator.
+function update_fieldsep(sep, p) {
+ # Count braces
+ sep_tmp = sep;
+ gsub(quote_regexp "|" comment_regexp, "", sep_tmp);
+ while (1)
+ {
+ if (sub("[^{}()]*\\{", "", sep_tmp)) {
+ brace_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\}", "", sep_tmp)) {
+ brace_nesting--;
+ if (brace_nesting < 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+ if (sub("[^{}()]*\\(", "", sep_tmp)) {
+ paren_nesting++;
+ continue;
+ }
+ if (sub("[^{}()]*\\)", "", sep_tmp)) {
+ paren_nesting--;
+ if (paren_nesting < 0) {
+ print "Unbalanced parenthesis!" > "/dev/stderr";
+ exit 1;
+ }
+ continue;
+ }
+
+ break;
+ }
+
+ if (last_fs_print)
+ printf("%s", last_fs) > outputfile;
+ last_fs = sep;
+ last_fs_print = p;
+}
+
+# Shifts the fields down by n positions. Calls next if there are no more. If p
+# is true then print out field separators.
+function shift_fields(n, p) {
+ do {
+ if (match($0, FS) > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), p);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ } else {
+ update_fieldsep("", 0);
+ print "" > outputfile;
+ next;
+ }
+ } while (--n > 0);
+}
+
+# Shifts and prints the first n fields.
+function print_fields(n) {
+ do {
+ update_fieldsep("", 0);
+ printf("%s", $1) > outputfile;
+ shift_fields(1, 1);
+ } while (--n > 0);
+}
+
+{
+ combine_backslashes();
+}
+
+# Print leading FS
+{
+ if (match($0, "^(" FS ")+") > 0) {
+ update_fieldsep(substr($0, RSTART, RLENGTH), 1);
+ if (RSTART + RLENGTH <= length())
+ $0 = substr($0, RSTART + RLENGTH);
+ else
+ $0 = "";
+ }
+}
+
+# Parse the line.
+{
+ while (NF > 0) {
+ if ($1 == "struct" && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_struct &&
+ brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "srcu_struct" &&
+ $0 ~ "^struct(" FS ")+srcu_struct(" FS ")+\\{") {
+ inside_srcu_struct = 1;
+ print_fields(2);
+ continue;
+ }
+ if (inside_srcu_struct && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_struct = 0;
+ update_fieldsep("", 0);
+ for (name in rcu_batches)
+ print "extern struct rcu_batch " name ";" > outputfile;
+ }
+
+ if (inside_srcu_struct && $1 == "struct" && $2 == "rcu_batch") {
+ # Move rcu_batches outside of the struct.
+ rcu_batches[$3] = "";
+ shift_fields(3, 1);
+ sub(/;[[:space:]]*$/, "", last_fs);
+ continue;
+ }
+
+ if (FILENAME ~ /\.h$/ && !inside_srcu_init_def &&
+ $1 == "#define" && $2 == "__SRCU_STRUCT_INIT") {
+ inside_srcu_init_def = 1;
+ srcu_init_param_name = $3;
+ in_macro = 1;
+ print_fields(3);
+ continue;
+ }
+ if (inside_srcu_init_def && brace_nesting == 0 &&
+ paren_nesting == 0) {
+ inside_srcu_init_def = 0;
+ in_macro = 0;
+ continue;
+ }
+
+ if (inside_srcu_init_def && brace_nesting == 1 &&
+ paren_nesting == 0 && last_fs ~ /\.[[:space:]]*$/ &&
+ $1 ~ /^[[:alnum:]_]+$/) {
+ name = $1;
+ if (name in rcu_batches) {
+ # Remove the dot.
+ sub(/\.[[:space:]]*$/, "", last_fs);
+
+ old_record = $0;
+ do
+ shift_fields(1, 0);
+ while (last_fs !~ /,/ || paren_nesting > 0);
+ end_loc = length(old_record) - length($0);
+ end_loc += index(last_fs, ",") - length(last_fs);
+
+ last_fs = substr(last_fs, index(last_fs, ",") + 1);
+ last_fs_print = 1;
+
+ match(old_record, "^"name"("FS")+=");
+ start_loc = RSTART + RLENGTH;
+
+ len = end_loc - start_loc;
+ initializer = substr(old_record, start_loc, len);
+ gsub(srcu_init_param_name "\\.", "", initializer);
+ rcu_batches[name] = initializer;
+ continue;
+ }
+ }
+
+ # Don't include a nonexistent file
+ if (!in_macro && $1 == "#include" && /^#include[[:space:]]+"rcu\.h"/) {
+ update_fieldsep("", 0);
+ next;
+ }
+
+ # Ignore most preprocessor stuff.
+ if (!in_macro && $1 ~ /#/) {
+ break;
+ }
+
+ if (brace_nesting > 0 && $1 ~ "^[[:alnum:]_]+$" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (brace_nesting > 0 &&
+ $0 ~ "^[[:alnum:]_]+[[:space:]]*(\\.|->)[[:space:]]*[[:alnum:]_]+" &&
+ $2 in rcu_batches) {
+ # Make uses of rcu_batches global. Somewhat unreliable.
+ shift_fields(1, 0);
+ print_fields(1);
+ continue;
+ }
+
+ if ($1 == "static" && NF < 3) {
+ read_line();
+ continue;
+ }
+ if ($1 == "static" && ($2 == "bool" && $3 == "try_check_zero" ||
+ $2 == "void" && $3 == "srcu_flip")) {
+ shift_fields(1, 1);
+ print_fields(2);
+ continue;
+ }
+
+ # Distinguish between read-side and write-side memory barriers.
+ if ($1 == "smp_mb" && NF < 2) {
+ read_line();
+ continue;
+ }
+ if (match($0, /^smp_mb[[:space:]();\/*]*[[:alnum:]]/)) {
+ barrier_letter = substr($0, RLENGTH, 1);
+ if (barrier_letter ~ /A|D/)
+ new_barrier_name = "sync_smp_mb";
+ else if (barrier_letter ~ /B|C/)
+ new_barrier_name = "rs_smp_mb";
+ else {
+ print "Unrecognized memory barrier." > "/dev/null";
+ exit 1;
+ }
+
+ shift_fields(1, 1);
+ printf("%s", new_barrier_name) > outputfile;
+ continue;
+ }
+
+ # Skip definition of rcu_synchronize, since it is already
+ # defined in misc.h. Only present in old versions of srcu.
+ if (brace_nesting == 0 && paren_nesting == 0 &&
+ $1 == "struct" && $2 == "rcu_synchronize" &&
+ $0 ~ "^struct(" FS ")+rcu_synchronize(" FS ")+\\{") {
+ shift_fields(2, 0);
+ while (brace_nesting) {
+ if (NF < 2)
+ read_line();
+ shift_fields(1, 0);
+ }
+ }
+
+ # Skip definition of wakeme_after_rcu for the same reason
+ if (brace_nesting == 0 && $1 == "static" && $2 == "void" &&
+ $3 == "wakeme_after_rcu") {
+ while (NF < 5)
+ read_line();
+ shift_fields(3, 0);
+ do {
+ while (NF < 3)
+ read_line();
+ shift_fields(1, 0);
+ } while (paren_nesting || brace_nesting);
+ }
+
+ if ($1 ~ /^(unsigned|long)$/ && NF < 3) {
+ read_line();
+ continue;
+ }
+
+ # Give srcu_batches_completed the correct type for old SRCU.
+ if (brace_nesting == 0 && $1 == "long" &&
+ $2 == "srcu_batches_completed") {
+ update_fieldsep("", 0);
+ printf("unsigned ") > outputfile;
+ print_fields(2);
+ continue;
+ }
+ if (brace_nesting == 0 && $1 == "unsigned" && $2 == "long" &&
+ $3 == "srcu_batches_completed") {
+ print_fields(3);
+ continue;
+ }
+
+ # Just print out the input code by default.
+ print_fields(1);
+ }
+ update_fieldsep("", 0);
+ print > outputfile;
+ next;
+}
+
+END {
+ update_fieldsep("", 0);
+
+ if (brace_nesting != 0) {
+ print "Unbalanced braces!" > "/dev/stderr";
+ exit 1;
+ }
+
+ # Define the rcu_batches
+ for (name in rcu_batches)
+ print "struct rcu_batch " name " = " rcu_batches[name] ";" > c_output;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
new file mode 100644
index 000000000000..a64955447995
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/assume.h
@@ -0,0 +1,16 @@
+#ifndef ASSUME_H
+#define ASSUME_H
+
+/* Provide an assumption macro that can be disabled for gcc. */
+#ifdef RUN
+#define assume(x) \
+ do { \
+ /* Evaluate x to suppress warnings. */ \
+ (void) (x); \
+ } while (0)
+
+#else
+#define assume(x) __CPROVER_assume(x)
+#endif
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
new file mode 100644
index 000000000000..6687acc08e6d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/barriers.h
@@ -0,0 +1,41 @@
+#ifndef BARRIERS_H
+#define BARRIERS_H
+
+#define barrier() __asm__ __volatile__("" : : : "memory")
+
+#ifdef RUN
+#define smp_mb() __sync_synchronize()
+#define smp_mb__after_unlock_lock() __sync_synchronize()
+#else
+/*
+ * Copied from CBMC's implementation of __sync_synchronize(), which
+ * seems to be disabled by default.
+ */
+#define smp_mb() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#define smp_mb__after_unlock_lock() __CPROVER_fence("WWfence", "RRfence", "RWfence", "WRfence", \
+ "WWcumul", "RRcumul", "RWcumul", "WRcumul")
+#endif
+
+/*
+ * Allow memory barriers to be disabled in either the read or write side
+ * of SRCU individually.
+ */
+
+#ifndef NO_SYNC_SMP_MB
+#define sync_smp_mb() smp_mb()
+#else
+#define sync_smp_mb() do {} while (0)
+#endif
+
+#ifndef NO_READ_SIDE_SMP_MB
+#define rs_smp_mb() smp_mb()
+#else
+#define rs_smp_mb() do {} while (0)
+#endif
+
+#define ACCESS_ONCE(x) (*(volatile typeof(x) *) &(x))
+#define READ_ONCE(x) ACCESS_ONCE(x)
+#define WRITE_ONCE(x, val) (ACCESS_ONCE(x) = (val))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
new file mode 100644
index 000000000000..2a80e91f78e7
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/bug_on.h
@@ -0,0 +1,13 @@
+#ifndef BUG_ON_H
+#define BUG_ON_H
+
+#include <assert.h>
+
+#define BUG() assert(0)
+#define BUG_ON(x) assert(!(x))
+
+/* Does it make sense to treat warnings as errors? */
+#define WARN() BUG()
+#define WARN_ON(x) (BUG_ON(x), false)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
new file mode 100644
index 000000000000..29eb5d2697ed
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/combined_source.c
@@ -0,0 +1,13 @@
+#include <config.h>
+
+/* Include all source files. */
+
+#include "include_srcu.c"
+
+#include "preempt.c"
+#include "misc.c"
+
+/* Used by test.c files */
+#include <pthread.h>
+#include <stdlib.h>
+#include <linux/srcu.h>
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
new file mode 100644
index 000000000000..a60038aeea7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h
@@ -0,0 +1,27 @@
+/* "Cheater" definitions based on restricted Kconfig choices. */
+
+#undef CONFIG_TINY_RCU
+#undef __CHECKER__
+#undef CONFIG_DEBUG_LOCK_ALLOC
+#undef CONFIG_DEBUG_OBJECTS_RCU_HEAD
+#undef CONFIG_HOTPLUG_CPU
+#undef CONFIG_MODULES
+#undef CONFIG_NO_HZ_FULL_SYSIDLE
+#undef CONFIG_PREEMPT_COUNT
+#undef CONFIG_PREEMPT_RCU
+#undef CONFIG_PROVE_RCU
+#undef CONFIG_RCU_NOCB_CPU
+#undef CONFIG_RCU_NOCB_CPU_ALL
+#undef CONFIG_RCU_STALL_COMMON
+#undef CONFIG_RCU_TRACE
+#undef CONFIG_RCU_USER_QS
+#undef CONFIG_TASKS_RCU
+#define CONFIG_TREE_RCU
+
+#define CONFIG_GENERIC_ATOMIC64
+
+#if NR_CPUS > 1
+#define CONFIG_SMP
+#else
+#undef CONFIG_SMP
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
new file mode 100644
index 000000000000..5ec582a53018
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/include_srcu.c
@@ -0,0 +1,31 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#define synchronize_srcu(sp) synchronize_srcu_original(sp)
+#endif
+
+#include <srcu.c>
+
+#ifdef USE_SIMPLE_SYNC_SRCU
+#undef synchronize_srcu
+
+#include "simple_sync_srcu.c"
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
new file mode 100644
index 000000000000..3aad63917858
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/int_typedefs.h
@@ -0,0 +1,33 @@
+#ifndef INT_TYPEDEFS_H
+#define INT_TYPEDEFS_H
+
+#include <inttypes.h>
+
+typedef int8_t s8;
+typedef uint8_t u8;
+typedef int16_t s16;
+typedef uint16_t u16;
+typedef int32_t s32;
+typedef uint32_t u32;
+typedef int64_t s64;
+typedef uint64_t u64;
+
+typedef int8_t __s8;
+typedef uint8_t __u8;
+typedef int16_t __s16;
+typedef uint16_t __u16;
+typedef int32_t __s32;
+typedef uint32_t __u32;
+typedef int64_t __s64;
+typedef uint64_t __u64;
+
+#define S8_C(x) INT8_C(x)
+#define U8_C(x) UINT8_C(x)
+#define S16_C(x) INT16_C(x)
+#define U16_C(x) UINT16_C(x)
+#define S32_C(x) INT32_C(x)
+#define U32_C(x) UINT32_C(x)
+#define S64_C(x) INT64_C(x)
+#define U64_C(x) UINT64_C(x)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
new file mode 100644
index 000000000000..356004665576
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/locks.h
@@ -0,0 +1,220 @@
+#ifndef LOCKS_H
+#define LOCKS_H
+
+#include <limits.h>
+#include <pthread.h>
+#include <stdbool.h>
+
+#include "assume.h"
+#include "bug_on.h"
+#include "preempt.h"
+
+int nondet_int(void);
+
+#define __acquire(x)
+#define __acquires(x)
+#define __release(x)
+#define __releases(x)
+
+/* Only use one lock mechanism. Select which one. */
+#ifdef PTHREAD_LOCK
+struct lock_impl {
+ pthread_mutex_t mutex;
+};
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_lock(&lock->mutex));
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+ BUG_ON(pthread_mutex_unlock(&lock->mutex));
+}
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+ int err = pthread_mutex_trylock(&lock->mutex);
+
+ if (!err)
+ return true;
+ else if (err == EBUSY)
+ return false;
+ BUG();
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ pthread_mutex_init(&lock->mutex, NULL);
+}
+
+#define LOCK_IMPL_INITIALIZER {.mutex = PTHREAD_MUTEX_INITIALIZER}
+
+#else /* !defined(PTHREAD_LOCK) */
+/* Spinlock that assumes that it always gets the lock immediately. */
+
+struct lock_impl {
+ bool locked;
+};
+
+static inline bool lock_impl_trylock(struct lock_impl *lock)
+{
+#ifdef RUN
+ /* TODO: Should this be a test and set? */
+ return __sync_bool_compare_and_swap(&lock->locked, false, true);
+#else
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = true;
+ __CPROVER_atomic_end();
+
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RRfence", "RWfence");
+
+ return !old_locked;
+#endif
+}
+
+static inline void lock_impl_lock(struct lock_impl *lock)
+{
+ /*
+ * CBMC doesn't support busy waiting, so just assume that the
+ * lock is available.
+ */
+ assume(lock_impl_trylock(lock));
+
+ /*
+ * If the lock was already held by this thread then the assumption
+ * is unsatisfiable (deadlock).
+ */
+}
+
+static inline void lock_impl_unlock(struct lock_impl *lock)
+{
+#ifdef RUN
+ BUG_ON(!__sync_bool_compare_and_swap(&lock->locked, true, false));
+#else
+ /* Minimal barrier to prevent accesses leaking out of lock. */
+ __CPROVER_fence("RWfence", "WWfence");
+
+ __CPROVER_atomic_begin();
+ bool old_locked = lock->locked;
+ lock->locked = false;
+ __CPROVER_atomic_end();
+
+ BUG_ON(!old_locked);
+#endif
+}
+
+static inline void lock_impl_init(struct lock_impl *lock)
+{
+ lock->locked = false;
+}
+
+#define LOCK_IMPL_INITIALIZER {.locked = false}
+
+#endif /* !defined(PTHREAD_LOCK) */
+
+/*
+ * Implement spinlocks using the lock mechanism. Wrap the lock to prevent mixing
+ * locks of different types.
+ */
+typedef struct {
+ struct lock_impl internal_lock;
+} spinlock_t;
+
+#define SPIN_LOCK_UNLOCKED {.internal_lock = LOCK_IMPL_INITIALIZER}
+#define __SPIN_LOCK_UNLOCKED(x) SPIN_LOCK_UNLOCKED
+#define DEFINE_SPINLOCK(x) spinlock_t x = SPIN_LOCK_UNLOCKED
+
+static inline void spin_lock_init(spinlock_t *lock)
+{
+ lock_impl_init(&lock->internal_lock);
+}
+
+static inline void spin_lock(spinlock_t *lock)
+{
+ /*
+ * Spin locks also need to be removed in order to eliminate all
+ * memory barriers. They are only used by the write side anyway.
+ */
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ lock_impl_lock(&lock->internal_lock);
+#endif
+}
+
+static inline void spin_unlock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ lock_impl_unlock(&lock->internal_lock);
+ preempt_enable();
+#endif
+}
+
+/* Don't bother with interrupts */
+#define spin_lock_irq(lock) spin_lock(lock)
+#define spin_unlock_irq(lock) spin_unlock(lock)
+#define spin_lock_irqsave(lock, flags) spin_lock(lock)
+#define spin_unlock_irqrestore(lock, flags) spin_unlock(lock)
+
+/*
+ * This is supposed to return an int, but I think that a bool should work as
+ * well.
+ */
+static inline bool spin_trylock(spinlock_t *lock)
+{
+#ifndef NO_SYNC_SMP_MB
+ preempt_disable();
+ return lock_impl_trylock(&lock->internal_lock);
+#else
+ return true;
+#endif
+}
+
+struct completion {
+ /* Hopefuly this won't overflow. */
+ unsigned int count;
+};
+
+#define COMPLETION_INITIALIZER(x) {.count = 0}
+#define DECLARE_COMPLETION(x) struct completion x = COMPLETION_INITIALIZER(x)
+#define DECLARE_COMPLETION_ONSTACK(x) DECLARE_COMPLETION(x)
+
+static inline void init_completion(struct completion *c)
+{
+ c->count = 0;
+}
+
+static inline void wait_for_completion(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_sub(&c->count, 1);
+
+ assume(prev_count);
+}
+
+static inline void complete(struct completion *c)
+{
+ unsigned int prev_count = __sync_fetch_and_add(&c->count, 1);
+
+ BUG_ON(prev_count == UINT_MAX);
+}
+
+/* This function probably isn't very useful for CBMC. */
+static inline bool try_wait_for_completion(struct completion *c)
+{
+ BUG();
+}
+
+static inline bool completion_done(struct completion *c)
+{
+ return c->count;
+}
+
+/* TODO: Implement complete_all */
+static inline void complete_all(struct completion *c)
+{
+ BUG();
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
new file mode 100644
index 000000000000..ca892e3b2351
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.c
@@ -0,0 +1,11 @@
+#include <config.h>
+
+#include "misc.h"
+#include "bug_on.h"
+
+struct rcu_head;
+
+void wakeme_after_rcu(struct rcu_head *head)
+{
+ BUG();
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
new file mode 100644
index 000000000000..aca50030f954
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/misc.h
@@ -0,0 +1,58 @@
+#ifndef MISC_H
+#define MISC_H
+
+#include "assume.h"
+#include "int_typedefs.h"
+#include "locks.h"
+
+#include <linux/types.h>
+
+/* Probably won't need to deal with bottom halves. */
+static inline void local_bh_disable(void) {}
+static inline void local_bh_enable(void) {}
+
+#define MODULE_ALIAS(X)
+#define module_param(...)
+#define EXPORT_SYMBOL_GPL(x)
+
+#define container_of(ptr, type, member) ({ \
+ const typeof(((type *)0)->member) *__mptr = (ptr); \
+ (type *)((char *)__mptr - offsetof(type, member)); \
+})
+
+#ifndef USE_SIMPLE_SYNC_SRCU
+/* Abuse udelay to make sure that busy loops terminate. */
+#define udelay(x) assume(0)
+
+#else
+
+/* The simple custom synchronize_srcu is ok with try_check_zero failing. */
+#define udelay(x) do { } while (0)
+#endif
+
+#define trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
+ do { } while (0)
+
+#define notrace
+
+/* Avoid including rcupdate.h */
+struct rcu_synchronize {
+ struct rcu_head head;
+ struct completion completion;
+};
+
+void wakeme_after_rcu(struct rcu_head *head);
+
+#define rcu_lock_acquire(a) do { } while (0)
+#define rcu_lock_release(a) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
+#define RCU_LOCKDEP_WARN(c, s) do { } while (0)
+
+/* Let CBMC non-deterministically choose switch between normal and expedited. */
+bool rcu_gp_is_normal(void);
+bool rcu_gp_is_expedited(void);
+
+/* Do the same for old versions of rcu. */
+#define rcu_expedited (rcu_gp_is_expedited())
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
new file mode 100644
index 000000000000..3de5a49de49b
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/percpu.h
@@ -0,0 +1,92 @@
+#ifndef PERCPU_H
+#define PERCPU_H
+
+#include <stddef.h>
+#include "bug_on.h"
+#include "preempt.h"
+
+#define __percpu
+
+/* Maximum size of any percpu data. */
+#define PERCPU_OFFSET (4 * sizeof(long))
+
+/* Ignore alignment, as CBMC doesn't care about false sharing. */
+#define alloc_percpu(type) __alloc_percpu(sizeof(type), 1)
+
+static inline void *__alloc_percpu(size_t size, size_t align)
+{
+ BUG();
+ return NULL;
+}
+
+static inline void free_percpu(void *ptr)
+{
+ BUG();
+}
+
+#define per_cpu_ptr(ptr, cpu) \
+ ((typeof(ptr)) ((char *) (ptr) + PERCPU_OFFSET * cpu))
+
+#define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1)
+#define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1)
+#define __this_cpu_sub(pcp, n) __this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+#define this_cpu_inc(pcp) this_cpu_add(pcp, 1)
+#define this_cpu_dec(pcp) this_cpu_sub(pcp, 1)
+#define this_cpu_sub(pcp, n) this_cpu_add(pcp, -(typeof(pcp)) (n))
+
+/* Make CBMC use atomics to work around bug. */
+#ifdef RUN
+#define THIS_CPU_ADD_HELPER(ptr, x) (*(ptr) += (x))
+#else
+/*
+ * Split the atomic into a read and a write so that it has the least
+ * possible ordering.
+ */
+#define THIS_CPU_ADD_HELPER(ptr, x) \
+ do { \
+ typeof(ptr) this_cpu_add_helper_ptr = (ptr); \
+ typeof(ptr) this_cpu_add_helper_x = (x); \
+ typeof(*ptr) this_cpu_add_helper_temp; \
+ __CPROVER_atomic_begin(); \
+ this_cpu_add_helper_temp = *(this_cpu_add_helper_ptr); \
+ __CPROVER_atomic_end(); \
+ this_cpu_add_helper_temp += this_cpu_add_helper_x; \
+ __CPROVER_atomic_begin(); \
+ *(this_cpu_add_helper_ptr) = this_cpu_add_helper_temp; \
+ __CPROVER_atomic_end(); \
+ } while (0)
+#endif
+
+/*
+ * For some reason CBMC needs an atomic operation even though this is percpu
+ * data.
+ */
+#define __this_cpu_add(pcp, n) \
+ do { \
+ BUG_ON(preemptible()); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), thread_cpu_id), \
+ (typeof(pcp)) (n)); \
+ } while (0)
+
+#define this_cpu_add(pcp, n) \
+ do { \
+ int this_cpu_add_impl_cpu = get_cpu(); \
+ THIS_CPU_ADD_HELPER(per_cpu_ptr(&(pcp), this_cpu_add_impl_cpu), \
+ (typeof(pcp)) (n)); \
+ put_cpu(); \
+ } while (0)
+
+/*
+ * This will cause a compiler warning because of the cast from char[][] to
+ * type*. This will cause a compile time error if type is too big.
+ */
+#define DEFINE_PER_CPU(type, name) \
+ char name[NR_CPUS][PERCPU_OFFSET]; \
+ typedef char percpu_too_big_##name \
+ [sizeof(type) > PERCPU_OFFSET ? -1 : 1]
+
+#define for_each_possible_cpu(cpu) \
+ for ((cpu) = 0; (cpu) < NR_CPUS; ++(cpu))
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
new file mode 100644
index 000000000000..4f1b068e9b7a
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.c
@@ -0,0 +1,78 @@
+#include <config.h>
+
+#include "preempt.h"
+
+#include "assume.h"
+#include "locks.h"
+
+/* Support NR_CPUS of at most 64 */
+#define CPU_PREEMPTION_LOCKS_INIT0 LOCK_IMPL_INITIALIZER
+#define CPU_PREEMPTION_LOCKS_INIT1 \
+ CPU_PREEMPTION_LOCKS_INIT0, CPU_PREEMPTION_LOCKS_INIT0
+#define CPU_PREEMPTION_LOCKS_INIT2 \
+ CPU_PREEMPTION_LOCKS_INIT1, CPU_PREEMPTION_LOCKS_INIT1
+#define CPU_PREEMPTION_LOCKS_INIT3 \
+ CPU_PREEMPTION_LOCKS_INIT2, CPU_PREEMPTION_LOCKS_INIT2
+#define CPU_PREEMPTION_LOCKS_INIT4 \
+ CPU_PREEMPTION_LOCKS_INIT3, CPU_PREEMPTION_LOCKS_INIT3
+#define CPU_PREEMPTION_LOCKS_INIT5 \
+ CPU_PREEMPTION_LOCKS_INIT4, CPU_PREEMPTION_LOCKS_INIT4
+
+/*
+ * Simulate disabling preemption by locking a particular cpu. NR_CPUS
+ * should be the actual number of cpus, not just the maximum.
+ */
+struct lock_impl cpu_preemption_locks[NR_CPUS] = {
+ CPU_PREEMPTION_LOCKS_INIT0
+#if (NR_CPUS - 1) & 1
+ , CPU_PREEMPTION_LOCKS_INIT0
+#endif
+#if (NR_CPUS - 1) & 2
+ , CPU_PREEMPTION_LOCKS_INIT1
+#endif
+#if (NR_CPUS - 1) & 4
+ , CPU_PREEMPTION_LOCKS_INIT2
+#endif
+#if (NR_CPUS - 1) & 8
+ , CPU_PREEMPTION_LOCKS_INIT3
+#endif
+#if (NR_CPUS - 1) & 16
+ , CPU_PREEMPTION_LOCKS_INIT4
+#endif
+#if (NR_CPUS - 1) & 32
+ , CPU_PREEMPTION_LOCKS_INIT5
+#endif
+};
+
+#undef CPU_PREEMPTION_LOCKS_INIT0
+#undef CPU_PREEMPTION_LOCKS_INIT1
+#undef CPU_PREEMPTION_LOCKS_INIT2
+#undef CPU_PREEMPTION_LOCKS_INIT3
+#undef CPU_PREEMPTION_LOCKS_INIT4
+#undef CPU_PREEMPTION_LOCKS_INIT5
+
+__thread int thread_cpu_id;
+__thread int preempt_disable_count;
+
+void preempt_disable(void)
+{
+ BUG_ON(preempt_disable_count < 0 || preempt_disable_count == INT_MAX);
+
+ if (preempt_disable_count++)
+ return;
+
+ thread_cpu_id = nondet_int();
+ assume(thread_cpu_id >= 0);
+ assume(thread_cpu_id < NR_CPUS);
+ lock_impl_lock(&cpu_preemption_locks[thread_cpu_id]);
+}
+
+void preempt_enable(void)
+{
+ BUG_ON(preempt_disable_count < 1);
+
+ if (--preempt_disable_count)
+ return;
+
+ lock_impl_unlock(&cpu_preemption_locks[thread_cpu_id]);
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
new file mode 100644
index 000000000000..2f95ee0e4dd5
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/preempt.h
@@ -0,0 +1,58 @@
+#ifndef PREEMPT_H
+#define PREEMPT_H
+
+#include <stdbool.h>
+
+#include "bug_on.h"
+
+/* This flag contains garbage if preempt_disable_count is 0. */
+extern __thread int thread_cpu_id;
+
+/* Support recursive preemption disabling. */
+extern __thread int preempt_disable_count;
+
+void preempt_disable(void);
+void preempt_enable(void);
+
+static inline void preempt_disable_notrace(void)
+{
+ preempt_disable();
+}
+
+static inline void preempt_enable_no_resched(void)
+{
+ preempt_enable();
+}
+
+static inline void preempt_enable_notrace(void)
+{
+ preempt_enable();
+}
+
+static inline int preempt_count(void)
+{
+ return preempt_disable_count;
+}
+
+static inline bool preemptible(void)
+{
+ return !preempt_count();
+}
+
+static inline int get_cpu(void)
+{
+ preempt_disable();
+ return thread_cpu_id;
+}
+
+static inline void put_cpu(void)
+{
+ preempt_enable();
+}
+
+static inline void might_sleep(void)
+{
+ BUG_ON(preempt_disable_count);
+}
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
new file mode 100644
index 000000000000..ac9cbc62b411
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/simple_sync_srcu.c
@@ -0,0 +1,50 @@
+#include <config.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <pthread.h>
+#include <stddef.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "int_typedefs.h"
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "locks.h"
+#include "misc.h"
+#include "preempt.h"
+#include "percpu.h"
+#include "workqueues.h"
+
+#include <linux/srcu.h>
+
+/* Functions needed from modify_srcu.c */
+bool try_check_zero(struct srcu_struct *sp, int idx, int trycount);
+void srcu_flip(struct srcu_struct *sp);
+
+/* Simpler implementation of synchronize_srcu that ignores batching. */
+void synchronize_srcu(struct srcu_struct *sp)
+{
+ int idx;
+ /*
+ * This code assumes that try_check_zero will succeed anyway,
+ * so there is no point in multiple tries.
+ */
+ const int trycount = 1;
+
+ might_sleep();
+
+ /* Ignore the lock, as multiple writers aren't working yet anyway. */
+
+ idx = 1 ^ (sp->completed & 1);
+
+ /* For comments see srcu_advance_batches. */
+
+ assume(try_check_zero(sp, idx, trycount));
+
+ srcu_flip(sp);
+
+ assume(try_check_zero(sp, idx^1, trycount));
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
new file mode 100644
index 000000000000..e58c8dfd3e90
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/workqueues.h
@@ -0,0 +1,102 @@
+#ifndef WORKQUEUES_H
+#define WORKQUEUES_H
+
+#include <stdbool.h>
+
+#include "barriers.h"
+#include "bug_on.h"
+#include "int_typedefs.h"
+
+#include <linux/types.h>
+
+/* Stub workqueue implementation. */
+
+struct work_struct;
+typedef void (*work_func_t)(struct work_struct *work);
+void delayed_work_timer_fn(unsigned long __data);
+
+struct work_struct {
+/* atomic_long_t data; */
+ unsigned long data;
+
+ struct list_head entry;
+ work_func_t func;
+#ifdef CONFIG_LOCKDEP
+ struct lockdep_map lockdep_map;
+#endif
+};
+
+struct timer_list {
+ struct hlist_node entry;
+ unsigned long expires;
+ void (*function)(unsigned long);
+ unsigned long data;
+ u32 flags;
+ int slack;
+};
+
+struct delayed_work {
+ struct work_struct work;
+ struct timer_list timer;
+
+ /* target workqueue and CPU ->timer uses to queue ->work */
+ struct workqueue_struct *wq;
+ int cpu;
+};
+
+
+static inline bool schedule_work(struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool schedule_work_on(int cpu, struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_work(struct workqueue_struct *wq,
+ struct work_struct *work)
+{
+ BUG();
+ return true;
+}
+
+static inline bool queue_delayed_work(struct workqueue_struct *wq,
+ struct delayed_work *dwork,
+ unsigned long delay)
+{
+ BUG();
+ return true;
+}
+
+#define INIT_WORK(w, f) \
+ do { \
+ (w)->data = 0; \
+ (w)->func = (f); \
+ } while (0)
+
+#define INIT_DELAYED_WORK(w, f) INIT_WORK(&(w)->work, (f))
+
+#define __WORK_INITIALIZER(n, f) { \
+ .data = 0, \
+ .entry = { &(n).entry, &(n).entry }, \
+ .func = f \
+ }
+
+/* Don't bother initializing timer. */
+#define __DELAYED_WORK_INITIALIZER(n, f, tflags) { \
+ .work = __WORK_INITIALIZER((n).work, (f)), \
+ }
+
+#define DECLARE_WORK(n, f) \
+ struct workqueue_struct n = __WORK_INITIALIZER
+
+#define DECLARE_DELAYED_WORK(n, f) \
+ struct delayed_work n = __DELAYED_WORK_INITIALIZER(n, f, 0)
+
+#define system_power_efficient_wq ((struct workqueue_struct *) NULL)
+
+#endif
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
new file mode 100644
index 000000000000..f47cb2045f13
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/.gitignore
@@ -0,0 +1 @@
+*.out
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
new file mode 100644
index 000000000000..3a3aee149225
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/Makefile
@@ -0,0 +1,11 @@
+CBMC_FLAGS = -I../.. -I../../src -I../../include -I../../empty_includes -32 -pointer-check -mm pso
+
+all:
+ for i in ./*.pass; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-pass $$i > $$i.out 2>&1 ; \
+ done
+ for i in ./*.fail; do \
+ echo $$i ; \
+ CBMC_FLAGS="$(CBMC_FLAGS)" sh ../test_script.sh --should-fail $$i > $$i.out 2>&1 ; \
+ done
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
new file mode 100644
index 000000000000..40c8075919d1
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/assert_end.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DASSERT_END"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
new file mode 100644
index 000000000000..ada5baf0b60d
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
new file mode 100644
index 000000000000..8fe00c8db466
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force2.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_2"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
new file mode 100644
index 000000000000..612ed6772844
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/force3.fail
@@ -0,0 +1 @@
+test_cbmc_options="-DFORCE_FAILURE_3"
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/main.pass
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
new file mode 100644
index 000000000000..470b1105a112
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/store_buffering/test.c
@@ -0,0 +1,72 @@
+#include <src/combined_source.c>
+
+int x;
+int y;
+
+int __unbuffered_tpr_x;
+int __unbuffered_tpr_y;
+
+DEFINE_SRCU(ss);
+
+void rcu_reader(void)
+{
+ int idx;
+
+#ifndef FORCE_FAILURE_3
+ idx = srcu_read_lock(&ss);
+#endif
+ might_sleep();
+
+ __unbuffered_tpr_y = READ_ONCE(y);
+#ifdef FORCE_FAILURE
+ srcu_read_unlock(&ss, idx);
+ idx = srcu_read_lock(&ss);
+#endif
+ WRITE_ONCE(x, 1);
+
+#ifndef FORCE_FAILURE_3
+ srcu_read_unlock(&ss, idx);
+#endif
+ might_sleep();
+}
+
+void *thread_update(void *arg)
+{
+ WRITE_ONCE(y, 1);
+#ifndef FORCE_FAILURE_2
+ synchronize_srcu(&ss);
+#endif
+ might_sleep();
+ __unbuffered_tpr_x = READ_ONCE(x);
+
+ return NULL;
+}
+
+void *thread_process_reader(void *arg)
+{
+ rcu_reader();
+
+ return NULL;
+}
+
+int main(int argc, char *argv[])
+{
+ pthread_t tu;
+ pthread_t tpr;
+
+ if (pthread_create(&tu, NULL, thread_update, NULL))
+ abort();
+ if (pthread_create(&tpr, NULL, thread_process_reader, NULL))
+ abort();
+ if (pthread_join(tu, NULL))
+ abort();
+ if (pthread_join(tpr, NULL))
+ abort();
+ assert(__unbuffered_tpr_y != 0 || __unbuffered_tpr_x != 0);
+
+#ifdef ASSERT_END
+ assert(0);
+#endif
+
+ return 0;
+}
diff --git a/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
new file mode 100755
index 000000000000..d1545972a0fa
--- /dev/null
+++ b/tools/testing/selftests/rcutorture/formal/srcu-cbmc/tests/test_script.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+
+# This script expects a mode (either --should-pass or --should-fail) followed by
+# an input file. The script uses the following environment variables. The test C
+# source file is expected to be named test.c in the directory containing the
+# input file.
+#
+# CBMC: The command to run CBMC. Default: cbmc
+# CBMC_FLAGS: Additional flags to pass to CBMC
+# NR_CPUS: Number of cpus to run tests with. Default specified by the test
+# SYNC_SRCU_MODE: Choose implementation of synchronize_srcu. Defaults to simple.
+# kernel: Version included in the linux kernel source.
+# simple: Use try_check_zero directly.
+#
+# The input file is a script that is sourced by this file. It can define any of
+# the following variables to configure the test.
+#
+# test_cbmc_options: Extra options to pass to CBMC.
+# min_cpus_fail: Minimum number of CPUs (NR_CPUS) for verification to fail.
+# The test is expected to pass if it is run with fewer. (Only
+# useful for .fail files)
+# default_cpus: Quantity of CPUs to use for the test, if not specified on the
+# command line. Default: Larger of 2 and MIN_CPUS_FAIL.
+
+set -e
+
+if test "$#" -ne 2; then
+ echo "Expected one option followed by an input file" 1>&2
+ exit 99
+fi
+
+if test "x$1" = "x--should-pass"; then
+ should_pass="yes"
+elif test "x$1" = "x--should-fail"; then
+ should_pass="no"
+else
+ echo "Unrecognized argument '$1'" 1>&2
+
+ # Exit code 99 indicates a hard error.
+ exit 99
+fi
+
+CBMC=${CBMC:-cbmc}
+
+SYNC_SRCU_MODE=${SYNC_SRCU_MODE:-simple}
+
+case ${SYNC_SRCU_MODE} in
+kernel) sync_srcu_mode_flags="" ;;
+simple) sync_srcu_mode_flags="-DUSE_SIMPLE_SYNC_SRCU" ;;
+
+*)
+ echo "Unrecognized argument '${SYNC_SRCU_MODE}'" 1>&2
+ exit 99
+ ;;
+esac
+
+min_cpus_fail=1
+
+c_file=`dirname "$2"`/test.c
+
+# Source the input file.
+. $2
+
+if test ${min_cpus_fail} -gt 2; then
+ default_default_cpus=${min_cpus_fail}
+else
+ default_default_cpus=2
+fi
+default_cpus=${default_cpus:-${default_default_cpus}}
+cpus=${NR_CPUS:-${default_cpus}}
+
+# Check if there are two few cpus to make the test fail.
+if test $cpus -lt ${min_cpus_fail:-0}; then
+ should_pass="yes"
+fi
+
+cbmc_opts="-DNR_CPUS=${cpus} ${sync_srcu_mode_flags} ${test_cbmc_options} ${CBMC_FLAGS}"
+
+echo "Running CBMC: ${CBMC} ${cbmc_opts} ${c_file}"
+if ${CBMC} ${cbmc_opts} "${c_file}"; then
+ # Verification successful. Make sure that it was supposed to verify.
+ test "x${should_pass}" = xyes
+else
+ cbmc_exit_status=$?
+
+ # An exit status of 10 indicates a failed verification.
+ # (see cbmc_parse_optionst::do_bmc in the CBMC source code)
+ if test ${cbmc_exit_status} -eq 10 && test "x${should_pass}" = xno; then
+ :
+ else
+ echo "CBMC returned ${cbmc_exit_status} exit status" 1>&2
+
+ # Parse errors have exit status 6. Any other type of error
+ # should be considered a hard error.
+ if test ${cbmc_exit_status} -ne 6 && \
+ test ${cbmc_exit_status} -ne 10; then
+ exit 99
+ else
+ exit 1
+ fi
+ fi
+fi
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 tip/core/rcu 4/4] srcu: Reduce probability of SRCU ->unlock_count[] counter overflow
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
` (2 preceding siblings ...)
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 3/4] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
@ 2017-01-24 22:00 ` Paul E. McKenney
3 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-24 22:00 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers, josh,
tglx, peterz, rostedt, dhowells, edumazet, dvhart, fweisbec,
oleg, bobby.prani, Paul E. McKenney
Because there are no memory barriers between the srcu_flip() ->completed
increment and the summation of the read-side ->unlock_count[] counters,
both the compiler and the CPU can reorder the summation with the
->completed increment. If the updater is preempted long enough during
this process, the read-side counters could overflow, resulting in a
too-short grace period.
This commit therefore adds a memory barrier just after the ->completed
increment, ensuring that if the summation misses an increment of
->unlock_count[] from __srcu_read_unlock(), the next __srcu_read_lock()
will see the new value of ->completed, thus bounding the number of
->unlock_count[] increments that can be missed to NR_CPUS. The actual
overflow computation is more complex due to the possibility of nesting
of __srcu_read_lock().
Reported-by: Lance Roy <ldr709@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/srcu.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index fcd07eda95a3..5dbd0d4426ff 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -321,7 +321,16 @@ static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
*/
static void srcu_flip(struct srcu_struct *sp)
{
- sp->completed++;
+ WRITE_ONCE(sp->completed, sp->completed + 1);
+
+ /*
+ * Ensure that if the updater misses an __srcu_read_unlock()
+ * increment, that task's next __srcu_read_lock() will see the
+ * above counter update. Note that both this memory barrier
+ * and the one in srcu_readers_active_idx_check() provide the
+ * guarantee for __srcu_read_lock().
+ */
+ smp_mb(); /* D */ /* Pairs with C. */
}
/*
--
2.5.2
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts Paul E. McKenney
@ 2017-01-25 18:17 ` Lance Roy
2017-01-25 21:03 ` Paul E. McKenney
0 siblings, 1 reply; 43+ messages in thread
From: Lance Roy @ 2017-01-25 18:17 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
Could you please use the new patch? The remark about ULONG_MAX - NR_CPUS is
incorrect in this one.
Thanks,
Lance
On Tue, 24 Jan 2017 14:00:26 -0800
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> From: Lance Roy <ldr709@gmail.com>
>
> SRCU uses two per-cpu counters: a nesting counter to count the number of
> active critical sections, and a sequence counter to ensure that the nesting
> counters don't change while they are being added together in
> srcu_readers_active_idx_check().
>
> This patch instead uses per-cpu lock and unlock counters. Because both
> counters only increase and srcu_readers_active_idx_check() reads the unlock
> counter before the lock counter, this achieves the same end without having
> to increment two different counters in srcu_read_lock(). This also saves a
> smp_mb() in srcu_readers_active_idx_check().
>
> A possible problem with this patch is that it can only handle
> ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
> handle up to ULONG_MAX.
>
> Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Signed-off-by: Lance Roy <ldr709@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> ---
> include/linux/srcu.h | 10 ++--
> kernel/rcu/rcutorture.c | 19 +++++++-
> kernel/rcu/srcu.c | 123
> ++++++++++++++++++------------------------------ 3 files changed, 67
> insertions(+), 85 deletions(-)
>
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index dc8eb63c6568..a598cf3ac70c 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -33,9 +33,9 @@
> #include <linux/rcupdate.h>
> #include <linux/workqueue.h>
>
> -struct srcu_struct_array {
> - unsigned long c[2];
> - unsigned long seq[2];
> +struct srcu_array {
> + unsigned long lock_count[2];
> + unsigned long unlock_count[2];
> };
>
> struct rcu_batch {
> @@ -46,7 +46,7 @@ struct rcu_batch {
>
> struct srcu_struct {
> unsigned long completed;
> - struct srcu_struct_array __percpu *per_cpu_ref;
> + struct srcu_array __percpu *per_cpu_ref;
> spinlock_t queue_lock; /* protect ->batch_queue, ->running */
> bool running;
> /* callbacks just queued */
> @@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
> * See include/linux/percpu-defs.h for the rules on per-CPU variables.
> */
> #define __DEFINE_SRCU(name,
> is_static) \
> - static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
> + static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
> is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
> #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static
> */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 87c51225ceec..d81345be730e 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
> pr_alert("%s%s per-CPU(idx=%d):",
> torture_type, TORTURE_FLAG, idx);
> for_each_possible_cpu(cpu) {
> + unsigned long l0, l1;
> + unsigned long u0, u1;
> long c0, c1;
> + struct srcu_array *counts =
> per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
> - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> + u0 = counts->unlock_count[!idx];
> + u1 = counts->unlock_count[idx];
> +
> + /*
> + * Make sure that a lock is always counted if the
> corresponding
> + * unlock is counted.
> + */
> + smp_rmb();
> +
> + l0 = counts->lock_count[!idx];
> + l1 = counts->lock_count[idx];
> +
> + c0 = l0 - u0;
> + c1 = l1 - u1;
> pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
> }
> pr_cont("\n");
> diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> index 9b9cdd549caa..ddabf5fbf562 100644
> --- a/kernel/rcu/srcu.c
> +++ b/kernel/rcu/srcu.c
> @@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
> rcu_batch_init(&sp->batch_check1);
> rcu_batch_init(&sp->batch_done);
> INIT_DELAYED_WORK(&sp->work, process_srcu);
> - sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
> + sp->per_cpu_ref = alloc_percpu(struct srcu_array);
> return sp->per_cpu_ref ? 0 : -ENOMEM;
> }
>
> @@ -141,114 +141,78 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
> #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
>
> /*
> - * Returns approximate total of the readers' ->seq[] values for the
> + * Returns approximate total of the readers' ->lock_count[] values for the
> * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> - unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> - sum += t;
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->lock_count[idx]);
> }
> return sum;
> }
>
> /*
> - * Returns approximate number of readers active on the specified rank
> - * of the per-CPU ->c[] counters.
> + * Returns approximate total of the readers' ->unlock_count[] values for the
> + * rank of per-CPU counters specified by idx.
> */
> -static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
> +static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
> {
> int cpu;
> unsigned long sum = 0;
> - unsigned long t;
>
> for_each_possible_cpu(cpu) {
> - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> - sum += t;
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->unlock_count[idx]);
> }
> return sum;
> }
>
> /*
> * Return true if the number of pre-existing readers is determined to
> - * be stably zero. An example unstable zero can occur if the call
> - * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
> - * but due to task migration, sees the corresponding __srcu_read_unlock()
> - * decrement. This can happen because srcu_readers_active_idx() takes
> - * time to sum the array, and might in fact be interrupted or preempted
> - * partway through the summation.
> + * be zero.
> */
> static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
> {
> - unsigned long seq;
> + unsigned long unlocks;
>
> - seq = srcu_readers_seq_idx(sp, idx);
> + unlocks = srcu_readers_unlock_idx(sp, idx);
>
> /*
> - * The following smp_mb() A pairs with the smp_mb() B located in
> - * __srcu_read_lock(). This pairing ensures that if an
> - * __srcu_read_lock() increments its counter after the summation
> - * in srcu_readers_active_idx(), then the corresponding SRCU
> read-side
> - * critical section will see any changes made prior to the start
> - * of the current SRCU grace period.
> + * Make sure that a lock is always counted if the corresponding
> unlock
> + * is counted. Needs to be a smp_mb() as the read side may contain a
> + * read from a variable that is written to before the
> synchronize_srcu()
> + * in the write side. In this case smp_mb()s A and B act like the
> store
> + * buffering pattern.
> *
> - * Also, if the above call to srcu_readers_seq_idx() saw the
> - * increment of ->seq[], then the call to srcu_readers_active_idx()
> - * must see the increment of ->c[].
> + * This smp_mb() also pairs with smp_mb() C to prevent writes after
> the
> + * synchronize_srcu() from being executed before the grace period
> ends. */
> smp_mb(); /* A */
>
> /*
> - * Note that srcu_readers_active_idx() can incorrectly return
> - * zero even though there is a pre-existing reader throughout.
> - * To see this, suppose that task A is in a very long SRCU
> - * read-side critical section that started on CPU 0, and that
> - * no other reader exists, so that the sum of the counters
> - * is equal to one. Then suppose that task B starts executing
> - * srcu_readers_active_idx(), summing up to CPU 1, and then that
> - * task C starts reading on CPU 0, so that its increment is not
> - * summed, but finishes reading on CPU 2, so that its decrement
> - * -is- summed. Then when task B completes its sum, it will
> - * incorrectly get zero, despite the fact that task A has been
> - * in its SRCU read-side critical section the whole time.
> + * If the locks are the same as the unlocks, then there must have
> + * been no readers on this index at some time in between. This does
> not
> + * mean that there are no more readers, as one could have read the
> + * current index but not have incremented the lock counter yet.
> *
> - * We therefore do a validation step should srcu_readers_active_idx()
> - * return zero.
> + * Note that there can be at most NR_CPUS worth of readers using the
> old
> + * index that haven't incremented ->lock_count[] yet. Therefore, the
> + * sum of the ->lock_count[]s cannot increment enough times to
> overflow
> + * and end up equal the sum of the ->unlock_count[]s, as long as
> there
> + * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this
> does
> + * mean that systems having more than a billion or so CPUs need to be
> + * 64-bit systems.) Therefore, the only way that the return values
> of
> + * the two calls to srcu_readers_(un)lock_idx() can be equal is if
> there
> + * are no active readers using this index.
> */
> - if (srcu_readers_active_idx(sp, idx) != 0)
> - return false;
> -
> - /*
> - * The remainder of this function is the validation step.
> - * The following smp_mb() D pairs with the smp_mb() C in
> - * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
> - * by srcu_readers_active_idx() above, then any destructive
> - * operation performed after the grace period will happen after
> - * the corresponding SRCU read-side critical section.
> - *
> - * Note that there can be at most NR_CPUS worth of readers using
> - * the old index, which is not enough to overflow even a 32-bit
> - * integer. (Yes, this does mean that systems having more than
> - * a billion or so CPUs need to be 64-bit systems.) Therefore,
> - * the sum of the ->seq[] counters cannot possibly overflow.
> - * Therefore, the only way that the return values of the two
> - * calls to srcu_readers_seq_idx() can be equal is if there were
> - * no increments of the corresponding rank of ->seq[] counts
> - * in the interim. But the missed-increment scenario laid out
> - * above includes an increment of the ->seq[] counter by
> - * the corresponding __srcu_read_lock(). Therefore, if this
> - * scenario occurs, the return values from the two calls to
> - * srcu_readers_seq_idx() will differ, and thus the validation
> - * step below suffices.
> - */
> - smp_mb(); /* D */
> -
> - return srcu_readers_seq_idx(sp, idx) == seq;
> + return srcu_readers_lock_idx(sp, idx) == unlocks;
> }
>
> /**
> @@ -266,8 +230,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
> unsigned long sum = 0;
>
> for_each_possible_cpu(cpu) {
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
> - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
> + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> +
> + sum += READ_ONCE(cpuc->lock_count[0]);
> + sum += READ_ONCE(cpuc->lock_count[1]);
> + sum -= READ_ONCE(cpuc->unlock_count[0]);
> + sum -= READ_ONCE(cpuc->unlock_count[1]);
> }
> return sum;
> }
> @@ -298,9 +266,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
> int idx;
>
> idx = READ_ONCE(sp->completed) & 0x1;
> - __this_cpu_inc(sp->per_cpu_ref->c[idx]);
> + __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
> smp_mb(); /* B */ /* Avoid leaking the critical section. */
> - __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
> return idx;
> }
> EXPORT_SYMBOL_GPL(__srcu_read_lock);
> @@ -314,7 +281,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
> void __srcu_read_unlock(struct srcu_struct *sp, int idx)
> {
> smp_mb(); /* C */ /* Avoid leaking the critical section. */
> - this_cpu_dec(sp->per_cpu_ref->c[idx]);
> + this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
> }
> EXPORT_SYMBOL_GPL(__srcu_read_unlock);
>
> @@ -349,7 +316,7 @@ static bool try_check_zero(struct srcu_struct *sp, int
> idx, int trycount)
> /*
> * Increment the ->completed counter so that future SRCU readers will
> - * use the other rank of the ->c[] and ->seq[] arrays. This allows
> + * use the other rank of the ->(un)lock_count[] arrays. This allows
> * us to wait for pre-existing readers in a starvation-free manner.
> */
> static void srcu_flip(struct srcu_struct *sp)
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts
2017-01-25 18:17 ` Lance Roy
@ 2017-01-25 21:03 ` Paul E. McKenney
0 siblings, 0 replies; 43+ messages in thread
From: Paul E. McKenney @ 2017-01-25 21:03 UTC (permalink / raw)
To: Lance Roy
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, peterz, rostedt, dhowells,
edumazet, dvhart, fweisbec, oleg, bobby.prani
On Wed, Jan 25, 2017 at 10:17:52AM -0800, Lance Roy wrote:
> Could you please use the new patch? The remark about ULONG_MAX - NR_CPUS is
> incorrect in this one.
Apologies -- I very carefully applied your patch, verified that it changed
only comments, and then very carefully forgot to rebase the rest of the
commits on top of it.
Fixed on -rcu branch dev.2017.01.25a.
Thanx, Paul
> Thanks,
> Lance
>
> On Tue, 24 Jan 2017 14:00:26 -0800
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
>
> > From: Lance Roy <ldr709@gmail.com>
> >
> > SRCU uses two per-cpu counters: a nesting counter to count the number of
> > active critical sections, and a sequence counter to ensure that the nesting
> > counters don't change while they are being added together in
> > srcu_readers_active_idx_check().
> >
> > This patch instead uses per-cpu lock and unlock counters. Because both
> > counters only increase and srcu_readers_active_idx_check() reads the unlock
> > counter before the lock counter, this achieves the same end without having
> > to increment two different counters in srcu_read_lock(). This also saves a
> > smp_mb() in srcu_readers_active_idx_check().
> >
> > A possible problem with this patch is that it can only handle
> > ULONG_MAX - NR_CPUS simultaneous readers, whereas the old version could
> > handle up to ULONG_MAX.
> >
> > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Signed-off-by: Lance Roy <ldr709@gmail.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > ---
> > include/linux/srcu.h | 10 ++--
> > kernel/rcu/rcutorture.c | 19 +++++++-
> > kernel/rcu/srcu.c | 123
> > ++++++++++++++++++------------------------------ 3 files changed, 67
> > insertions(+), 85 deletions(-)
> >
> > diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> > index dc8eb63c6568..a598cf3ac70c 100644
> > --- a/include/linux/srcu.h
> > +++ b/include/linux/srcu.h
> > @@ -33,9 +33,9 @@
> > #include <linux/rcupdate.h>
> > #include <linux/workqueue.h>
> >
> > -struct srcu_struct_array {
> > - unsigned long c[2];
> > - unsigned long seq[2];
> > +struct srcu_array {
> > + unsigned long lock_count[2];
> > + unsigned long unlock_count[2];
> > };
> >
> > struct rcu_batch {
> > @@ -46,7 +46,7 @@ struct rcu_batch {
> >
> > struct srcu_struct {
> > unsigned long completed;
> > - struct srcu_struct_array __percpu *per_cpu_ref;
> > + struct srcu_array __percpu *per_cpu_ref;
> > spinlock_t queue_lock; /* protect ->batch_queue, ->running */
> > bool running;
> > /* callbacks just queued */
> > @@ -118,7 +118,7 @@ void process_srcu(struct work_struct *work);
> > * See include/linux/percpu-defs.h for the rules on per-CPU variables.
> > */
> > #define __DEFINE_SRCU(name,
> > is_static) \
> > - static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\
> > + static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
> > is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
> > #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static
> > */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
> > diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> > index 87c51225ceec..d81345be730e 100644
> > --- a/kernel/rcu/rcutorture.c
> > +++ b/kernel/rcu/rcutorture.c
> > @@ -564,10 +564,25 @@ static void srcu_torture_stats(void)
> > pr_alert("%s%s per-CPU(idx=%d):",
> > torture_type, TORTURE_FLAG, idx);
> > for_each_possible_cpu(cpu) {
> > + unsigned long l0, l1;
> > + unsigned long u0, u1;
> > long c0, c1;
> > + struct srcu_array *counts =
> > per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu);
> > - c0 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[!idx];
> > - c1 = (long)per_cpu_ptr(srcu_ctlp->per_cpu_ref, cpu)->c[idx];
> > + u0 = counts->unlock_count[!idx];
> > + u1 = counts->unlock_count[idx];
> > +
> > + /*
> > + * Make sure that a lock is always counted if the
> > corresponding
> > + * unlock is counted.
> > + */
> > + smp_rmb();
> > +
> > + l0 = counts->lock_count[!idx];
> > + l1 = counts->lock_count[idx];
> > +
> > + c0 = l0 - u0;
> > + c1 = l1 - u1;
> > pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
> > }
> > pr_cont("\n");
> > diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> > index 9b9cdd549caa..ddabf5fbf562 100644
> > --- a/kernel/rcu/srcu.c
> > +++ b/kernel/rcu/srcu.c
> > @@ -106,7 +106,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
> > rcu_batch_init(&sp->batch_check1);
> > rcu_batch_init(&sp->batch_done);
> > INIT_DELAYED_WORK(&sp->work, process_srcu);
> > - sp->per_cpu_ref = alloc_percpu(struct srcu_struct_array);
> > + sp->per_cpu_ref = alloc_percpu(struct srcu_array);
> > return sp->per_cpu_ref ? 0 : -ENOMEM;
> > }
> >
> > @@ -141,114 +141,78 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
> > #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
> >
> > /*
> > - * Returns approximate total of the readers' ->seq[] values for the
> > + * Returns approximate total of the readers' ->lock_count[] values for the
> > * rank of per-CPU counters specified by idx.
> > */
> > -static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)
> > +static unsigned long srcu_readers_lock_idx(struct srcu_struct *sp, int idx)
> > {
> > int cpu;
> > unsigned long sum = 0;
> > - unsigned long t;
> >
> > for_each_possible_cpu(cpu) {
> > - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->seq[idx]);
> > - sum += t;
> > + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> > +
> > + sum += READ_ONCE(cpuc->lock_count[idx]);
> > }
> > return sum;
> > }
> >
> > /*
> > - * Returns approximate number of readers active on the specified rank
> > - * of the per-CPU ->c[] counters.
> > + * Returns approximate total of the readers' ->unlock_count[] values for the
> > + * rank of per-CPU counters specified by idx.
> > */
> > -static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)
> > +static unsigned long srcu_readers_unlock_idx(struct srcu_struct *sp, int idx)
> > {
> > int cpu;
> > unsigned long sum = 0;
> > - unsigned long t;
> >
> > for_each_possible_cpu(cpu) {
> > - t = READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[idx]);
> > - sum += t;
> > + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> > +
> > + sum += READ_ONCE(cpuc->unlock_count[idx]);
> > }
> > return sum;
> > }
> >
> > /*
> > * Return true if the number of pre-existing readers is determined to
> > - * be stably zero. An example unstable zero can occur if the call
> > - * to srcu_readers_active_idx() misses an __srcu_read_lock() increment,
> > - * but due to task migration, sees the corresponding __srcu_read_unlock()
> > - * decrement. This can happen because srcu_readers_active_idx() takes
> > - * time to sum the array, and might in fact be interrupted or preempted
> > - * partway through the summation.
> > + * be zero.
> > */
> > static bool srcu_readers_active_idx_check(struct srcu_struct *sp, int idx)
> > {
> > - unsigned long seq;
> > + unsigned long unlocks;
> >
> > - seq = srcu_readers_seq_idx(sp, idx);
> > + unlocks = srcu_readers_unlock_idx(sp, idx);
> >
> > /*
> > - * The following smp_mb() A pairs with the smp_mb() B located in
> > - * __srcu_read_lock(). This pairing ensures that if an
> > - * __srcu_read_lock() increments its counter after the summation
> > - * in srcu_readers_active_idx(), then the corresponding SRCU
> > read-side
> > - * critical section will see any changes made prior to the start
> > - * of the current SRCU grace period.
> > + * Make sure that a lock is always counted if the corresponding
> > unlock
> > + * is counted. Needs to be a smp_mb() as the read side may contain a
> > + * read from a variable that is written to before the
> > synchronize_srcu()
> > + * in the write side. In this case smp_mb()s A and B act like the
> > store
> > + * buffering pattern.
> > *
> > - * Also, if the above call to srcu_readers_seq_idx() saw the
> > - * increment of ->seq[], then the call to srcu_readers_active_idx()
> > - * must see the increment of ->c[].
> > + * This smp_mb() also pairs with smp_mb() C to prevent writes after
> > the
> > + * synchronize_srcu() from being executed before the grace period
> > ends. */
> > smp_mb(); /* A */
> >
> > /*
> > - * Note that srcu_readers_active_idx() can incorrectly return
> > - * zero even though there is a pre-existing reader throughout.
> > - * To see this, suppose that task A is in a very long SRCU
> > - * read-side critical section that started on CPU 0, and that
> > - * no other reader exists, so that the sum of the counters
> > - * is equal to one. Then suppose that task B starts executing
> > - * srcu_readers_active_idx(), summing up to CPU 1, and then that
> > - * task C starts reading on CPU 0, so that its increment is not
> > - * summed, but finishes reading on CPU 2, so that its decrement
> > - * -is- summed. Then when task B completes its sum, it will
> > - * incorrectly get zero, despite the fact that task A has been
> > - * in its SRCU read-side critical section the whole time.
> > + * If the locks are the same as the unlocks, then there must have
> > + * been no readers on this index at some time in between. This does
> > not
> > + * mean that there are no more readers, as one could have read the
> > + * current index but not have incremented the lock counter yet.
> > *
> > - * We therefore do a validation step should srcu_readers_active_idx()
> > - * return zero.
> > + * Note that there can be at most NR_CPUS worth of readers using the
> > old
> > + * index that haven't incremented ->lock_count[] yet. Therefore, the
> > + * sum of the ->lock_count[]s cannot increment enough times to
> > overflow
> > + * and end up equal the sum of the ->unlock_count[]s, as long as
> > there
> > + * are at most ULONG_MAX - NR_CPUS readers at a time. (Yes, this
> > does
> > + * mean that systems having more than a billion or so CPUs need to be
> > + * 64-bit systems.) Therefore, the only way that the return values
> > of
> > + * the two calls to srcu_readers_(un)lock_idx() can be equal is if
> > there
> > + * are no active readers using this index.
> > */
> > - if (srcu_readers_active_idx(sp, idx) != 0)
> > - return false;
> > -
> > - /*
> > - * The remainder of this function is the validation step.
> > - * The following smp_mb() D pairs with the smp_mb() C in
> > - * __srcu_read_unlock(). If the __srcu_read_unlock() was seen
> > - * by srcu_readers_active_idx() above, then any destructive
> > - * operation performed after the grace period will happen after
> > - * the corresponding SRCU read-side critical section.
> > - *
> > - * Note that there can be at most NR_CPUS worth of readers using
> > - * the old index, which is not enough to overflow even a 32-bit
> > - * integer. (Yes, this does mean that systems having more than
> > - * a billion or so CPUs need to be 64-bit systems.) Therefore,
> > - * the sum of the ->seq[] counters cannot possibly overflow.
> > - * Therefore, the only way that the return values of the two
> > - * calls to srcu_readers_seq_idx() can be equal is if there were
> > - * no increments of the corresponding rank of ->seq[] counts
> > - * in the interim. But the missed-increment scenario laid out
> > - * above includes an increment of the ->seq[] counter by
> > - * the corresponding __srcu_read_lock(). Therefore, if this
> > - * scenario occurs, the return values from the two calls to
> > - * srcu_readers_seq_idx() will differ, and thus the validation
> > - * step below suffices.
> > - */
> > - smp_mb(); /* D */
> > -
> > - return srcu_readers_seq_idx(sp, idx) == seq;
> > + return srcu_readers_lock_idx(sp, idx) == unlocks;
> > }
> >
> > /**
> > @@ -266,8 +230,12 @@ static bool srcu_readers_active(struct srcu_struct *sp)
> > unsigned long sum = 0;
> >
> > for_each_possible_cpu(cpu) {
> > - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[0]);
> > - sum += READ_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);
> > + struct srcu_array *cpuc = per_cpu_ptr(sp->per_cpu_ref, cpu);
> > +
> > + sum += READ_ONCE(cpuc->lock_count[0]);
> > + sum += READ_ONCE(cpuc->lock_count[1]);
> > + sum -= READ_ONCE(cpuc->unlock_count[0]);
> > + sum -= READ_ONCE(cpuc->unlock_count[1]);
> > }
> > return sum;
> > }
> > @@ -298,9 +266,8 @@ int __srcu_read_lock(struct srcu_struct *sp)
> > int idx;
> >
> > idx = READ_ONCE(sp->completed) & 0x1;
> > - __this_cpu_inc(sp->per_cpu_ref->c[idx]);
> > + __this_cpu_inc(sp->per_cpu_ref->lock_count[idx]);
> > smp_mb(); /* B */ /* Avoid leaking the critical section. */
> > - __this_cpu_inc(sp->per_cpu_ref->seq[idx]);
> > return idx;
> > }
> > EXPORT_SYMBOL_GPL(__srcu_read_lock);
> > @@ -314,7 +281,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock);
> > void __srcu_read_unlock(struct srcu_struct *sp, int idx)
> > {
> > smp_mb(); /* C */ /* Avoid leaking the critical section. */
> > - this_cpu_dec(sp->per_cpu_ref->c[idx]);
> > + this_cpu_inc(sp->per_cpu_ref->unlock_count[idx]);
> > }
> > EXPORT_SYMBOL_GPL(__srcu_read_unlock);
> >
> > @@ -349,7 +316,7 @@ static bool try_check_zero(struct srcu_struct *sp, int
> > idx, int trycount)
> > /*
> > * Increment the ->completed counter so that future SRCU readers will
> > - * use the other rank of the ->c[] and ->seq[] arrays. This allows
> > + * use the other rank of the ->(un)lock_count[] arrays. This allows
> > * us to wait for pre-existing readers in a starvation-free manner.
> > */
> > static void srcu_flip(struct srcu_struct *sp)
>
^ permalink raw reply [flat|nested] 43+ messages in thread
end of thread, other threads:[~2017-01-25 21:03 UTC | newest]
Thread overview: 43+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-14 9:19 [PATCH tip/core/rcu 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-14 9:19 ` [PATCH tip/core/rcu 1/3] srcu: More efficient reader counts Paul E. McKenney
2017-01-14 9:31 ` Ingo Molnar
2017-01-14 19:48 ` Paul E. McKenney
2017-01-14 9:20 ` [PATCH tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
2017-01-14 9:35 ` Ingo Molnar
2017-01-14 19:54 ` Paul E. McKenney
2017-01-14 21:41 ` Paul E. McKenney
2017-01-15 7:11 ` Ingo Molnar
2017-01-15 7:40 ` Paul E. McKenney
2017-01-15 7:57 ` Ingo Molnar
2017-01-15 9:24 ` Paul E. McKenney
2017-01-15 9:40 ` Ingo Molnar
2017-01-15 19:45 ` Paul E. McKenney
2017-01-16 6:56 ` Ingo Molnar
2017-01-23 8:12 ` Michael Ellerman
2017-01-24 2:45 ` Paul E. McKenney
2017-01-15 6:54 ` Ingo Molnar
2017-01-14 9:20 ` [PATCH tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-15 22:41 ` [PATCH tip/core/rcu v2 0/3] SRCU updates for 4.11 Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 1/3] srcu: Implement more-efficient reader counts Paul E. McKenney
2017-01-23 20:17 ` Lance Roy
2017-01-23 20:17 ` [PATCH] SRCU: More efficient " Lance Roy
2017-01-23 20:35 ` Paul E. McKenney
2017-01-23 21:33 ` Lance Roy
2017-01-23 21:35 ` [PATCH] srcu: Implement more-efficient " Lance Roy
2017-01-24 0:42 ` Paul E. McKenney
2017-01-24 0:53 ` Lance Roy
2017-01-24 1:57 ` Paul E. McKenney
2017-01-24 3:26 ` Lance Roy
2017-01-24 17:07 ` Paul E. McKenney
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 2/3] srcu: Force full grace-period ordering Paul E. McKenney
2017-01-23 8:38 ` Lance Roy
2017-01-23 19:12 ` Paul E. McKenney
2017-01-23 20:06 ` Lance Roy
2017-01-15 22:42 ` [PATCH v2 tip/core/rcu 3/3] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 0/4] SRCU updates for 4.11 Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 1/4] srcu: Implement more-efficient reader counts Paul E. McKenney
2017-01-25 18:17 ` Lance Roy
2017-01-25 21:03 ` Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 2/4] srcu: Force full grace-period ordering Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 3/4] rcutorture: Add CBMC-based formal verification for SRCU Paul E. McKenney
2017-01-24 22:00 ` [PATCH v3 tip/core/rcu 4/4] srcu: Reduce probability of SRCU ->unlock_count[] counter overflow Paul E. McKenney
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.