From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Mathieu Desnoyers <compudj@krystal.dyndns.org>
Cc: ltt-dev@lists.casi.polymtl.ca, linux-kernel@vger.kernel.org
Subject: Re: [ltt-dev] [RFC git tree] Userspace RCU (urcu) for Linux (repost)
Date: Tue, 10 Feb 2009 21:28:28 -0800 [thread overview]
Message-ID: <20090211052828.GQ6742@linux.vnet.ibm.com> (raw)
In-Reply-To: <20090211005701.GA550@Krystal>
On Tue, Feb 10, 2009 at 07:57:01PM -0500, Mathieu Desnoyers wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > On Tue, Feb 10, 2009 at 04:28:33PM -0500, Mathieu Desnoyers wrote:
> > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > > On Tue, Feb 10, 2009 at 02:17:31PM -0500, Mathieu Desnoyers wrote:
> > > > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > > > > On Mon, Feb 09, 2009 at 02:03:17AM -0500, Mathieu Desnoyers wrote:
> > > > > >
> > > > > > [ . . . ]
> > > > > >
> > > > > > > I just added modified rcutorture.h and api.h from your git tree
> > > > > > > specifically for an urcutorture program to the repository. Some results :
> > > > > > >
> > > > > > > 8-way x86_64
> > > > > > > E5405 @2 GHZ
> > > > > > >
> > > > > > > ./urcutorture 8 perf
> > > > > > > n_reads: 1937650000 n_updates: 3 nreaders: 8 nupdaters: 1 duration: 1
> > > > > > > ns/read: 4.12871 ns/update: 3.33333e+08
> > > > > > >
> > > > > > > ./urcutorture 8 uperf
> > > > > > > n_reads: 0 n_updates: 4413892 nreaders: 0 nupdaters: 8 duration: 1
> > > > > > > ns/read: nan ns/update: 1812.46
> > > > > > >
> > > > > > > n_reads: 98844204 n_updates: 10 n_mberror: 0
> > > > > > > rcu_stress_count: 98844171 33 0 0 0 0 0 0 0 0 0
> > > > > > >
> > > > > > > However, I've tried removing the second switch_qparity() call, and the
> > > > > > > rcutorture test did not detect anything wrong. I also did a variation
> > > > > > > which calls the "sched_yield" version of the urcu, "urcutorture-yield".
> > > > > >
> > > > > > My confusion -- I was testing my old approach where the memory barriers
> > > > > > are in rcu_read_lock() and rcu_read_unlock(). To force the failures in
> > > > > > your signal-handler-memory-barrier approach, I suspect that you are
> > > > > > going to need a bigger hammer. In this case, one such bigger hammer
> > > > > > would be:
> > > > > >
> > > > > > o Just before exit from the signal handler, do a
> > > > > > pthread_cond_wait() under a pthread_mutex().
> > > > > >
> > > > > > o In force_mb_all_threads(), refrain from sending a signal to self.
> > > > > >
> > > > > > Then it should be safe in force_mb_all_threads() to do a
> > > > > > pthread_cond_broadcast() under the same pthread_mutex().
> > > > > >
> > > > > > This should raise the probability of seeing the failure in the case
> > > > > > where there is a single switch_qparity().
> > > > > >
> > > > >
> > > > > I just did a mb() version of the urcu :
> > > > >
> > > > > (uncomment CFLAGS=+-DDEBUG_FULL_MB in the Makefile)
> > > > >
> > > > > Time per read : 48.4086 cycles
> > > > > (about 6-7 times slower, as expected)
> > > > >
> > > > > This will be useful especially to increase the chance to trigger races.
> > > > >
> > > > > I tried removing the second parity switch from the writer. The rcu
> > > > > torture test did not find the problem yet (maybe I am not using the
> > > > > correct parameters ? It does not run for more than 5 seconds).
> > > > >
> > > > > So I added a "-n" option to test_urcu, so it can make the usleep(1)
> > > > > between the writes optional. I also changed the yield for a usleep with
> > > > > random delay. I also now use a circular buffer rather than malloc so we
> > > > > are sure the memory is not quickly reused by the writer and stays longer
> > > > > in an invalid state.
> > > > >
> > > > > So what really make the problem appear quickly is to add a delay between
> > > > > the rcu_dereference and the assertion on the data validity in thr_reader.
> > > > >
> > > > > It now appears after just a few seconds when running
> > > > > ./test_urcu_yield 20 -r -n
> > > > > Compiled with CFLAGS=+-DDEBUG_FULL_MB
> > > > >
> > > > > It seem to be much harder to trigger with the signal-based version. It's
> > > > > expected, because the writer takes about 50 times longer to execute than
> > > > > with the -DDEBUG_FULL_MB version.
> > > > >
> > > > > So I'll let the ./test_urcu_yield NN -r -n run for a while on the
> > > > > correct version (with DEBUG_FULL_MB) and see what it gives.
> > > >
> > > > Hmmm... I had worse luck this time, took three 10-second tries to
> > > > see a failure:
> > > >
> > > > paulmck@paulmck-laptop:~/paper/perfbook/CodeSamples/defer$ ./rcu_nest32 1 stress
> > > > n_reads: 44682055 n_updates: 9609503 n_mberror: 0
> > > > rcu_stress_count: 44679377 2678 0 0 0 0 0 0 0 0 0
> > > > paulmck@paulmck-laptop:~/paper/perfbook/CodeSamples/defer$ !!
> > > > ./rcu_nest32 1 stress
> > > > n_reads: 42281884 n_updates: 9870129 n_mberror: 0
> > > > rcu_stress_count: 42277756 4128 0 0 0 0 0 0 0 0 0
> > > > paulmck@paulmck-laptop:~/paper/perfbook/CodeSamples/defer$ !!
> > > > ./rcu_nest32 1 stress
> > > > n_reads: 41384304 n_updates: 10040805 n_mberror: 0
> > > > rcu_stress_count: 41380075 4228 1 0 0 0 0 0 0 0 0
> > > > paulmck@paulmck-laptop:~/paper/perfbook/CodeSamples/defer$
> > > >
> > > > This is my prototype version, with read-side memory barriers, no
> > > > signals, and without your initialization-value speedup.
> > > >
> > >
> > > It would be interesting to re-sync our trees, or if you can point me to
> > > a current version of your prototype, I could review it.
> >
> > Look at:
> >
> > CodeSamples/defer/rcu_nest32.[hc]
> >
> > In the git archive:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git
>
> flip_counter_and_wait : yours do rcu_gp_ctr += RCU_GP_CTR_BOTTOM_BIT
> mine : rcu_gp_ctr ^= RCU_GP_CTR_BOTTOM_BIT.
Yep, this is before your optimization.
> Another major difference between our tree is the lack of smp_mb() at the
> end of flip_counter_and_wait() (in your tree).
>
> Your code does :
>
> smp_mb()
> switch parity
> smp_mb()
> wait for each thread ongoing old gp
> <<<<<<< ---- missing smp_mb.
> switch parity
> smp_mb()
> wait for each thread ongoing old gp
> smp_mb()
This should be OK -- or am I missing a failure scenario?
Keep in mind that I get failures only when omitting a counter
flip, not with the above code.
> I also wonder why you have a smp_mb() after spin_unlock() in your
> synchronize_rcu() -> if you follow the Linux kernel semantics for
> spinlocks, the smp_mb() should be implied. (but I have not looked at
> your spin_lock/unlock primitives yet).
Perhaps things have changed, but last I knew, spin_lock() and
spin_unlock() were only required to keep the critical section in, not
to keep things out of the critical section.
Thanx, Paul
> Mathieu
>
> > Thanx, Paul
> >
> > _______________________________________________
> > ltt-dev mailing list
> > ltt-dev@lists.casi.polymtl.ca
> > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
next prev parent reply other threads:[~2009-02-11 5:28 UTC|newest]
Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-02-06 3:05 [RFC git tree] Userspace RCU (urcu) for Linux Mathieu Desnoyers
2009-02-06 4:58 ` [RFC git tree] Userspace RCU (urcu) for Linux (repost) Mathieu Desnoyers
2009-02-06 13:06 ` Paul E. McKenney
2009-02-06 16:34 ` Paul E. McKenney
2009-02-07 15:10 ` Paul E. McKenney
2009-02-07 22:16 ` Paul E. McKenney
2009-02-08 0:19 ` Mathieu Desnoyers
2009-02-07 23:38 ` Mathieu Desnoyers
2009-02-08 0:44 ` Paul E. McKenney
2009-02-08 21:46 ` Mathieu Desnoyers
2009-02-08 22:36 ` Paul E. McKenney
2009-02-09 0:24 ` Paul E. McKenney
2009-02-09 0:54 ` Mathieu Desnoyers
2009-02-09 1:08 ` [ltt-dev] " Mathieu Desnoyers
2009-02-09 3:47 ` Paul E. McKenney
2009-02-09 3:42 ` Paul E. McKenney
2009-02-09 0:40 ` [ltt-dev] " Mathieu Desnoyers
2009-02-08 22:44 ` Mathieu Desnoyers
2009-02-09 4:11 ` Paul E. McKenney
2009-02-09 4:53 ` Mathieu Desnoyers
2009-02-09 5:17 ` [ltt-dev] " Mathieu Desnoyers
2009-02-09 7:03 ` Mathieu Desnoyers
2009-02-09 15:33 ` Paul E. McKenney
2009-02-10 19:17 ` Mathieu Desnoyers
2009-02-10 21:16 ` Paul E. McKenney
2009-02-10 21:28 ` Mathieu Desnoyers
2009-02-10 22:21 ` Paul E. McKenney
2009-02-10 22:58 ` Paul E. McKenney
2009-02-10 23:01 ` Paul E. McKenney
2009-02-11 0:57 ` Mathieu Desnoyers
2009-02-11 5:28 ` Paul E. McKenney [this message]
2009-02-11 6:35 ` Mathieu Desnoyers
2009-02-11 15:32 ` Paul E. McKenney
2009-02-11 18:52 ` Mathieu Desnoyers
2009-02-11 20:09 ` Paul E. McKenney
2009-02-11 21:42 ` Mathieu Desnoyers
2009-02-11 22:08 ` Mathieu Desnoyers
[not found] ` <20090212003549.GU6694@linux.vnet.ibm.com>
2009-02-12 2:33 ` Paul E. McKenney
2009-02-12 2:37 ` Paul E. McKenney
2009-02-12 4:10 ` Mathieu Desnoyers
2009-02-12 5:09 ` Paul E. McKenney
2009-02-12 5:47 ` Mathieu Desnoyers
2009-02-12 16:18 ` Paul E. McKenney
2009-02-12 18:40 ` Mathieu Desnoyers
2009-02-12 20:28 ` Paul E. McKenney
2009-02-12 21:27 ` Mathieu Desnoyers
2009-02-12 23:26 ` Paul E. McKenney
2009-02-13 13:12 ` Mathieu Desnoyers
2009-02-12 4:08 ` Mathieu Desnoyers
2009-02-12 5:01 ` Paul E. McKenney
2009-02-12 7:05 ` Mathieu Desnoyers
2009-02-12 16:46 ` Paul E. McKenney
2009-02-12 19:29 ` Mathieu Desnoyers
2009-02-12 20:02 ` Paul E. McKenney
2009-02-12 20:09 ` Mathieu Desnoyers
2009-02-12 20:35 ` Paul E. McKenney
2009-02-12 21:15 ` Mathieu Desnoyers
2009-02-12 20:13 ` Linus Torvalds
2009-02-12 20:39 ` Paul E. McKenney
2009-02-12 21:15 ` Linus Torvalds
2009-02-12 21:59 ` Paul E. McKenney
2009-02-13 13:50 ` Nick Piggin
2009-02-13 14:56 ` Paul E. McKenney
2009-02-13 15:10 ` Mathieu Desnoyers
2009-02-13 15:55 ` Mathieu Desnoyers
2009-02-13 16:18 ` Linus Torvalds
2009-02-13 17:33 ` Mathieu Desnoyers
2009-02-13 17:53 ` Linus Torvalds
2009-02-13 18:09 ` Linus Torvalds
2009-02-13 18:54 ` Mathieu Desnoyers
2009-02-13 19:36 ` Paul E. McKenney
2009-02-14 5:07 ` Mike Frysinger
2009-02-14 5:20 ` Paul E. McKenney
2009-02-14 5:46 ` Mike Frysinger
2009-02-14 15:06 ` Paul E. McKenney
2009-02-14 17:37 ` Mike Frysinger
2009-02-22 14:23 ` Pavel Machek
2009-02-22 18:28 ` Mike Frysinger
2009-02-14 6:42 ` Mathieu Desnoyers
2009-02-14 3:15 ` [Uclinux-dist-devel] " Mike Frysinger
2009-02-13 18:40 ` Mathieu Desnoyers
2009-02-13 16:05 ` Linus Torvalds
2009-02-14 3:11 ` [Uclinux-dist-devel] " Mike Frysinger
2009-02-14 4:58 ` Robin Getz
2009-02-12 19:38 ` Mathieu Desnoyers
2009-02-12 20:17 ` Paul E. McKenney
2009-02-12 21:53 ` Mathieu Desnoyers
2009-02-12 23:04 ` Paul E. McKenney
2009-02-13 12:49 ` Mathieu Desnoyers
2009-02-11 5:08 ` Lai Jiangshan
2009-02-11 8:58 ` Mathieu Desnoyers
2009-02-09 13:23 ` Paul E. McKenney
2009-02-09 17:28 ` Mathieu Desnoyers
2009-02-09 17:47 ` Paul E. McKenney
2009-02-09 18:13 ` Mathieu Desnoyers
2009-02-09 18:19 ` Mathieu Desnoyers
2009-02-09 18:37 ` Paul E. McKenney
2009-02-09 18:49 ` Paul E. McKenney
2009-02-09 19:05 ` Mathieu Desnoyers
2009-02-09 19:15 ` Mathieu Desnoyers
2009-02-09 19:35 ` Paul E. McKenney
2009-02-09 19:23 ` Paul E. McKenney
2009-02-09 13:16 ` Paul E. McKenney
2009-02-09 17:19 ` Bert Wesarg
2009-02-09 17:34 ` Paul E. McKenney
2009-02-09 17:35 ` Bert Wesarg
2009-02-09 17:40 ` Paul E. McKenney
2009-02-09 17:42 ` Mathieu Desnoyers
2009-02-09 18:00 ` Paul E. McKenney
2009-02-09 17:45 ` Bert Wesarg
2009-02-09 17:59 ` Paul E. McKenney
2009-02-07 22:56 ` Kyle Moffett
2009-02-07 23:50 ` Mathieu Desnoyers
2009-02-08 0:13 ` Paul E. McKenney
2009-02-06 8:55 ` [RFC git tree] Userspace RCU (urcu) for Linux Bert Wesarg
2009-02-06 11:36 ` Mathieu Desnoyers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20090211052828.GQ6742@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=compudj@krystal.dyndns.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ltt-dev@lists.casi.polymtl.ca \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).