* Re: rculfstack bug
[not found] <50754A57.1080104@cn.fujitsu.com>
@ 2012-10-10 11:42 ` Mathieu Desnoyers
[not found] ` <20121010114215.GA11307@Krystal>
1 sibling, 0 replies; 10+ messages in thread
From: Mathieu Desnoyers @ 2012-10-10 11:42 UTC (permalink / raw)
To: Lai Jiangshan; +Cc: lttng-dev, Paul E. McKenney
* Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> test code:
> ./tests/test_urcu_lfs 100 10 10
>
> bug produce rate > 60%
>
> {{{
> I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> But I just test it about 5 times
> }}}
>
> 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> RCU_MB (no time to test for other rcu type)
> test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
>
> I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
>
> Sorry, I tried, but I failed to find out the root cause currently.
I think I managed to narrow down the issue:
1) the master branch does not reproduce it, but commit
768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
time.
2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
moving to wfcqueue.
3) the bug always arise, for me, at the end of the 10 seconds.
However, it might be simply due to the fact that most of the memory
get freed at the end of program execution.
4) I've been able to get a backtrace, and it looks like we have some
call_rcu callback-invokation threads still working while
call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
is nicely waiting for the next thread to stop, and during that time,
two callback-invokation threads are invoking callbacks (and one of
them triggers the segfault).
So I expect that commit
commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Tue Sep 25 10:50:49 2012 -0500
call_rcu: use wfcqueue, eliminate false-sharing
Eliminate false-sharing between call_rcu (enqueuer) and worker threads
on the queue head and tail.
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Could have managed to fix the issue, or change the timing enough that it
does not reproduces. I'll continue investigating.
Thanks,
Mathieu
>
> *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x37ee676d63]
> /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> /lib64/libpthread.so.0[0x37eda06ccb]
> /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> ======= Memory map: ========
> 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> 00606000-00616000 rw-p 00000000 00:00 0
> 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> 37ed820000-37ed821000 rw-p 00000000 00:00 0
> 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010114215.GA11307@Krystal>
@ 2012-10-10 15:02 ` Paul E. McKenney
[not found] ` <20121010150207.GB2495@linux.vnet.ibm.com>
1 sibling, 0 replies; 10+ messages in thread
From: Paul E. McKenney @ 2012-10-10 15:02 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: lttng-dev
On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> > test code:
> > ./tests/test_urcu_lfs 100 10 10
> >
> > bug produce rate > 60%
> >
> > {{{
> > I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> > But I just test it about 5 times
> > }}}
> >
> > 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> > RCU_MB (no time to test for other rcu type)
> > test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> >
> > I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> >
> > Sorry, I tried, but I failed to find out the root cause currently.
>
> I think I managed to narrow down the issue:
>
> 1) the master branch does not reproduce it, but commit
> 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> time.
>
> 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> moving to wfcqueue.
>
> 3) the bug always arise, for me, at the end of the 10 seconds.
> However, it might be simply due to the fact that most of the memory
> get freed at the end of program execution.
>
> 4) I've been able to get a backtrace, and it looks like we have some
> call_rcu callback-invokation threads still working while
> call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> is nicely waiting for the next thread to stop, and during that time,
> two callback-invokation threads are invoking callbacks (and one of
> them triggers the segfault).
Do any of the callbacks reference __thread variables from some other
thread? If so, those threads must refrain from exiting until after
such callbacks complete.
Thanx, Paul
> So I expect that commit
>
> commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date: Tue Sep 25 10:50:49 2012 -0500
>
> call_rcu: use wfcqueue, eliminate false-sharing
>
> Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> on the queue head and tail.
>
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>
> Could have managed to fix the issue, or change the timing enough that it
> does not reproduces. I'll continue investigating.
>
> Thanks,
>
> Mathieu
>
>
> >
> > *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x37ee676d63]
> > /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> > /lib64/libpthread.so.0[0x37eda06ccb]
> > /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> > ======= Memory map: ========
> > 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > 00606000-00616000 rw-p 00000000 00:00 0
> > 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> > 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> > 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> > 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> > 37ed820000-37ed821000 rw-p 00000000 00:00 0
> > 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> > 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> > 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> > 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> > 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> > 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> > 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> > 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> > 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> > 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> > 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> > 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> > 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> > 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> > 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> > 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> > 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> > 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> > 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> > 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> > 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> > 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> > 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> > 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> > 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> > 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> > 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> > 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> > 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> > 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> > 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> > 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> > 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> > 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> > 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> > 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> > 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> > 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> > 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> > 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> > 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> > 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> > 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> > 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> > 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> > 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> > 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> > 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> > 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> > 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> > 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> > 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> > 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> > 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> > 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> > 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> > 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> > 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> > 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> > 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> > 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> > 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> > 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> > 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> > 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> > 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> > 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> > 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> > 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> > 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> > 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> > 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> > 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> > 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> > 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> > 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> > 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> > 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> > 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> > 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> > 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> > 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> > 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> > 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> > 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> > 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> > 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> > 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> > 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> > 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> > 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> > 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> > 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> > 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> > 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> > 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> > 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> > 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> > 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> > 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> > 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> > 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> > 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> > 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> > 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> > 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> > 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> > 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> > 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> > 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> > 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> > 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> > 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> > 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> > 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> > 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> > 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> > 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> > 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> > 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> > 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> > 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> > 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> > 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> > 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> > 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> > 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> > 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> > 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> > 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> > 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> > 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> > 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> > 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> > 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> > 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> > 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> > 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> > 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> > 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> > 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> > 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> > 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> > 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> > 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> > 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> > 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> > 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> > 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> > 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> > 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> > 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> > 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> > 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> > 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010150207.GB2495@linux.vnet.ibm.com>
@ 2012-10-10 15:07 ` Mathieu Desnoyers
[not found] ` <20121010150751.GB20761@Krystal>
1 sibling, 0 replies; 10+ messages in thread
From: Mathieu Desnoyers @ 2012-10-10 15:07 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: lttng-dev
* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> > * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> > > test code:
> > > ./tests/test_urcu_lfs 100 10 10
> > >
> > > bug produce rate > 60%
> > >
> > > {{{
> > > I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> > > But I just test it about 5 times
> > > }}}
> > >
> > > 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> > > RCU_MB (no time to test for other rcu type)
> > > test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> > >
> > > I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> > >
> > > Sorry, I tried, but I failed to find out the root cause currently.
> >
> > I think I managed to narrow down the issue:
> >
> > 1) the master branch does not reproduce it, but commit
> > 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> > time.
> >
> > 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> > current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> > moving to wfcqueue.
> >
> > 3) the bug always arise, for me, at the end of the 10 seconds.
> > However, it might be simply due to the fact that most of the memory
> > get freed at the end of program execution.
> >
> > 4) I've been able to get a backtrace, and it looks like we have some
> > call_rcu callback-invokation threads still working while
> > call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> > is nicely waiting for the next thread to stop, and during that time,
> > two callback-invokation threads are invoking callbacks (and one of
> > them triggers the segfault).
>
> Do any of the callbacks reference __thread variables from some other
> thread? If so, those threads must refrain from exiting until after
> such callbacks complete.
The callback is a simple caa_container_of + free, usual stuff, nothing
fancy.
Thanks,
Mathieu
>
> Thanx, Paul
>
> > So I expect that commit
> >
> > commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Date: Tue Sep 25 10:50:49 2012 -0500
> >
> > call_rcu: use wfcqueue, eliminate false-sharing
> >
> > Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> > on the queue head and tail.
> >
> > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> >
> > Could have managed to fix the issue, or change the timing enough that it
> > does not reproduces. I'll continue investigating.
> >
> > Thanks,
> >
> > Mathieu
> >
> >
> > >
> > > *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6[0x37ee676d63]
> > > /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> > > /lib64/libpthread.so.0[0x37eda06ccb]
> > > /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> > > ======= Memory map: ========
> > > 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > 00606000-00616000 rw-p 00000000 00:00 0
> > > 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> > > 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> > > 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> > > 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> > > 37ed820000-37ed821000 rw-p 00000000 00:00 0
> > > 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> > > 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> > > 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> > > 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> > > 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> > > 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> > > 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> > > 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> > > 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> > > 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> > > 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> > > 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> > > 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> > > 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> > > 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> > > 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> > > 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> > > 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> > > 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> > > 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> > > 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> > > 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> > > 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> > > 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> > > 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> > > 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> > > 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> > > 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> > > 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> > > 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> > > 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> > > 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> > > 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> > > 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> > > 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> > > 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> > > 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> > > 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> > > 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> > > 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> > > 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> > > 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> > > 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> > > 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> > > 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> > > 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> > > 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> > > 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> > > 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> > > 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> > > 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> > > 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> > > 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> > > 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> > > 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> > > 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> > > 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> > > 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> > > 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> > > 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> > > 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> > > 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> > > 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> > > 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> > > 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> > > 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> > > 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> > > 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> > > 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> > > 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> > > 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> > > 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> > > 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> > > 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> > > 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> > > 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> > > 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> > > 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> > > 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> > > 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> > > 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> > > 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> > > 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> > > 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> > > 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> > > 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> > > 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> > > 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> > > 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> > > 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> > > 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> > > 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> > > 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> > > 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> > > 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> > > 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> > > 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> > > 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> > > 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> > > 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> > > 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> > > 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> > > 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> > > 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> > > 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> > > 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> > > 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> > > 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> > > 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> > > 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> > > 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> > > 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> > > 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> > > 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> > > 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> > > 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> > > 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> > > 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> > > 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> > > 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> > > 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> > > 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> > > 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> > > 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> > > 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> > > 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> > > 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> > > 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> > > 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> > > 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> > > 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> > > 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> > > 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> > > 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> > > 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> > > 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> > > 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> > > 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> > > 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> > > 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> > > 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> > > 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> > > 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> > > 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> > > 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> > > 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> > > 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> > > 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> > > 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> > > 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> > > 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> > > 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> > > 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> > > 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> > > 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> > >
> > > _______________________________________________
> > > lttng-dev mailing list
> > > lttng-dev@lists.lttng.org
> > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant
> > EfficiOS Inc.
> > http://www.efficios.com
> >
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010150751.GB20761@Krystal>
@ 2012-10-10 17:53 ` Mathieu Desnoyers
[not found] ` <20121010175304.GA25511@Krystal>
1 sibling, 0 replies; 10+ messages in thread
From: Mathieu Desnoyers @ 2012-10-10 17:53 UTC (permalink / raw)
To: Paul E. McKenney, Lai Jiangshan; +Cc: lttng-dev
* Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> > > * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> > > > test code:
> > > > ./tests/test_urcu_lfs 100 10 10
> > > >
> > > > bug produce rate > 60%
> > > >
> > > > {{{
> > > > I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> > > > But I just test it about 5 times
> > > > }}}
> > > >
> > > > 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> > > > RCU_MB (no time to test for other rcu type)
> > > > test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> > > >
> > > > I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> > > >
> > > > Sorry, I tried, but I failed to find out the root cause currently.
> > >
> > > I think I managed to narrow down the issue:
> > >
> > > 1) the master branch does not reproduce it, but commit
> > > 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> > > time.
> > >
> > > 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> > > current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> > > moving to wfcqueue.
> > >
> > > 3) the bug always arise, for me, at the end of the 10 seconds.
> > > However, it might be simply due to the fact that most of the memory
> > > get freed at the end of program execution.
> > >
> > > 4) I've been able to get a backtrace, and it looks like we have some
> > > call_rcu callback-invokation threads still working while
> > > call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> > > is nicely waiting for the next thread to stop, and during that time,
> > > two callback-invokation threads are invoking callbacks (and one of
> > > them triggers the segfault).
> >
> > Do any of the callbacks reference __thread variables from some other
> > thread? If so, those threads must refrain from exiting until after
> > such callbacks complete.
>
> The callback is a simple caa_container_of + free, usual stuff, nothing
> fancy.
Here is the fix: the bug was in call rcu. It is not required for master,
because we fixed it while moving to wfcqueue.
We were erroneously writing to the head field of the default
call_rcu_data rather than tail.
I wonder if we should simply do a new release with call_rcu using
wfcqueue and tell people to upgrade, or if we should somehow create a
stable branch with this fix.
Thoughts ?
Thanks,
Mathieu
---
diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
index 13b24ff..b205229 100644
--- a/urcu-call-rcu-impl.h
+++ b/urcu-call-rcu-impl.h
@@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
/* Create default call rcu data if need be */
(void) get_default_call_rcu_data();
cbs_endprev = (struct cds_wfq_node **)
- uatomic_xchg(&default_call_rcu_data, cbs_tail);
- *cbs_endprev = cbs;
+ uatomic_xchg(&default_call_rcu_data->cbs.tail,
+ cbs_tail);
+ _CMM_STORE_SHARED(*cbs_endprev, cbs);
uatomic_add(&default_call_rcu_data->qlen,
uatomic_read(&crdp->qlen));
wake_call_rcu_thread(default_call_rcu_data);
>
> Thanks,
>
> Mathieu
>
> >
> > Thanx, Paul
> >
> > > So I expect that commit
> > >
> > > commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> > > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > Date: Tue Sep 25 10:50:49 2012 -0500
> > >
> > > call_rcu: use wfcqueue, eliminate false-sharing
> > >
> > > Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> > > on the queue head and tail.
> > >
> > > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > >
> > > Could have managed to fix the issue, or change the timing enough that it
> > > does not reproduces. I'll continue investigating.
> > >
> > > Thanks,
> > >
> > > Mathieu
> > >
> > >
> > > >
> > > > *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> > > > ======= Backtrace: =========
> > > > /lib64/libc.so.6[0x37ee676d63]
> > > > /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> > > > /lib64/libpthread.so.0[0x37eda06ccb]
> > > > /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> > > > ======= Memory map: ========
> > > > 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > 00606000-00616000 rw-p 00000000 00:00 0
> > > > 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> > > > 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> > > > 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> > > > 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> > > > 37ed820000-37ed821000 rw-p 00000000 00:00 0
> > > > 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> > > > 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> > > > 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> > > > 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> > > > 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> > > > 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> > > > 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> > > > 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> > > > 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> > > > 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> > > > 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> > > > 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> > > > 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> > > > 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> > > > 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> > > > 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> > > > 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> > > > 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> > > > 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> > > > 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> > > > 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> > > > 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> > > > 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> > > > 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> > > > 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> > > > 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> > > > 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> > > > 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> > > > 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> > > > 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> > > > 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> > > > 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> > > > 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> > > > 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> > > > 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> > > > 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> > > > 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> > > > 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> > > > 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> > > > 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> > > > 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> > > > 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> > > > 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> > > > 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> > > > 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> > > > 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> > > > 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> > > > 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> > > > 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> > > > 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> > > > 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> > > > 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> > > > 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> > > > 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> > > > 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> > > > 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> > > > 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> > > > 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> > > > 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> > > > 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> > > > 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> > > > 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> > > > 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> > > > 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> > > > 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> > > > 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> > > > 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> > > > 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> > > > 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> > > > 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> > > > 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> > > > 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> > > > 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> > > > 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> > > > 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> > > > 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> > > > 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> > > > 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> > > > 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> > > > 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> > > > 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> > > > 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> > > > 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> > > > 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> > > > 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> > > > 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> > > > 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> > > > 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> > > > 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> > > > 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> > > > 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> > > > 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> > > > 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> > > > 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> > > > 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> > > > 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> > > > 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> > > > 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> > > > 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> > > > 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> > > > 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> > > > 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> > > > 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> > > > 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> > > > 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> > > > 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> > > > 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> > > > 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> > > > 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> > > > 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> > > > 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> > > > 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> > > > 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> > > > 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> > > > 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> > > > 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> > > > 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> > > > 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> > > > 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> > > > 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> > > > 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> > > > 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> > > > 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> > > > 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> > > > 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> > > > 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> > > > 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> > > > 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> > > > 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> > > > 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> > > > 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> > > > 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> > > > 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> > > > 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> > > > 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> > > > 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> > > > 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> > > > 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> > > > 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> > > > 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> > > > 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> > > > 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> > > > 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> > > > 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> > > > 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> > > > 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> > > > 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> > > > 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> > > > 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> > > > 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> > > > 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> > > > 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> > > > 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> > > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> > > >
> > > > _______________________________________________
> > > > lttng-dev mailing list
> > > > lttng-dev@lists.lttng.org
> > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > >
> > > --
> > > Mathieu Desnoyers
> > > Operating System Efficiency R&D Consultant
> > > EfficiOS Inc.
> > > http://www.efficios.com
> > >
> >
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010175304.GA25511@Krystal>
@ 2012-10-10 19:50 ` Paul E. McKenney
[not found] ` <20121010195007.GG2495@linux.vnet.ibm.com>
1 sibling, 0 replies; 10+ messages in thread
From: Paul E. McKenney @ 2012-10-10 19:50 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: lttng-dev
On Wed, Oct 10, 2012 at 01:53:04PM -0400, Mathieu Desnoyers wrote:
> * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> > > > * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> > > > > test code:
> > > > > ./tests/test_urcu_lfs 100 10 10
> > > > >
> > > > > bug produce rate > 60%
> > > > >
> > > > > {{{
> > > > > I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> > > > > But I just test it about 5 times
> > > > > }}}
> > > > >
> > > > > 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> > > > > RCU_MB (no time to test for other rcu type)
> > > > > test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> > > > >
> > > > > I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> > > > >
> > > > > Sorry, I tried, but I failed to find out the root cause currently.
> > > >
> > > > I think I managed to narrow down the issue:
> > > >
> > > > 1) the master branch does not reproduce it, but commit
> > > > 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> > > > time.
> > > >
> > > > 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> > > > current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> > > > moving to wfcqueue.
> > > >
> > > > 3) the bug always arise, for me, at the end of the 10 seconds.
> > > > However, it might be simply due to the fact that most of the memory
> > > > get freed at the end of program execution.
> > > >
> > > > 4) I've been able to get a backtrace, and it looks like we have some
> > > > call_rcu callback-invokation threads still working while
> > > > call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> > > > is nicely waiting for the next thread to stop, and during that time,
> > > > two callback-invokation threads are invoking callbacks (and one of
> > > > them triggers the segfault).
> > >
> > > Do any of the callbacks reference __thread variables from some other
> > > thread? If so, those threads must refrain from exiting until after
> > > such callbacks complete.
> >
> > The callback is a simple caa_container_of + free, usual stuff, nothing
> > fancy.
>
> Here is the fix: the bug was in call rcu. It is not required for master,
> because we fixed it while moving to wfcqueue.
>
> We were erroneously writing to the head field of the default
> call_rcu_data rather than tail.
Ouch!!! I have no idea why that would have passed my testing. :-(
> I wonder if we should simply do a new release with call_rcu using
> wfcqueue and tell people to upgrade, or if we should somehow create a
> stable branch with this fix.
>
> Thoughts ?
Under what conditions does this bug appear? It is necessary to not just
use call_rcu(), but also to explicitly call call_rcu_data_free(), right?
My guess is that a stable branch would be good -- there will be other
bugs, after all. :-/
Thanx, Paul
> Thanks,
>
> Mathieu
>
> ---
> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
> index 13b24ff..b205229 100644
> --- a/urcu-call-rcu-impl.h
> +++ b/urcu-call-rcu-impl.h
> @@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
> /* Create default call rcu data if need be */
> (void) get_default_call_rcu_data();
> cbs_endprev = (struct cds_wfq_node **)
> - uatomic_xchg(&default_call_rcu_data, cbs_tail);
> - *cbs_endprev = cbs;
> + uatomic_xchg(&default_call_rcu_data->cbs.tail,
> + cbs_tail);
> + _CMM_STORE_SHARED(*cbs_endprev, cbs);
> uatomic_add(&default_call_rcu_data->qlen,
> uatomic_read(&crdp->qlen));
> wake_call_rcu_thread(default_call_rcu_data);
>
>
> >
> > Thanks,
> >
> > Mathieu
> >
> > >
> > > Thanx, Paul
> > >
> > > > So I expect that commit
> > > >
> > > > commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> > > > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > > Date: Tue Sep 25 10:50:49 2012 -0500
> > > >
> > > > call_rcu: use wfcqueue, eliminate false-sharing
> > > >
> > > > Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> > > > on the queue head and tail.
> > > >
> > > > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > >
> > > > Could have managed to fix the issue, or change the timing enough that it
> > > > does not reproduces. I'll continue investigating.
> > > >
> > > > Thanks,
> > > >
> > > > Mathieu
> > > >
> > > >
> > > > >
> > > > > *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> > > > > ======= Backtrace: =========
> > > > > /lib64/libc.so.6[0x37ee676d63]
> > > > > /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> > > > > /lib64/libpthread.so.0[0x37eda06ccb]
> > > > > /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> > > > > ======= Memory map: ========
> > > > > 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > > 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > > 00606000-00616000 rw-p 00000000 00:00 0
> > > > > 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> > > > > 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> > > > > 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> > > > > 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> > > > > 37ed820000-37ed821000 rw-p 00000000 00:00 0
> > > > > 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> > > > > 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> > > > > 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > > 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > > 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> > > > > 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> > > > > 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> > > > > 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> > > > > 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> > > > > 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> > > > > 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> > > > > 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> > > > > 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> > > > > 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> > > > > 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> > > > > 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> > > > > 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> > > > > 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> > > > > 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> > > > > 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> > > > > 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> > > > > 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> > > > > 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> > > > > 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> > > > > 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> > > > > 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> > > > > 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> > > > > 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> > > > > 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> > > > > 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> > > > > 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> > > > > 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> > > > > 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> > > > > 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> > > > > 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> > > > > 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> > > > > 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> > > > > 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> > > > > 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> > > > > 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> > > > > 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> > > > > 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> > > > > 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> > > > > 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> > > > > 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> > > > > 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> > > > > 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> > > > > 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> > > > > 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> > > > > 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> > > > > 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> > > > > 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> > > > > 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> > > > > 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> > > > > 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> > > > > 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> > > > > 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> > > > > 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> > > > > 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> > > > > 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> > > > > 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> > > > > 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> > > > > 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> > > > > 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> > > > > 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> > > > > 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> > > > > 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> > > > > 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> > > > > 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> > > > > 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> > > > > 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> > > > > 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> > > > > 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> > > > > 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> > > > > 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> > > > > 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> > > > > 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> > > > > 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> > > > > 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> > > > > 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> > > > > 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> > > > > 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> > > > > 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> > > > > 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> > > > > 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> > > > > 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> > > > > 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> > > > > 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> > > > > 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> > > > > 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> > > > > 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> > > > > 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> > > > > 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> > > > > 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> > > > > 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> > > > > 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> > > > > 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> > > > > 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> > > > > 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> > > > > 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> > > > > 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> > > > > 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> > > > > 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> > > > > 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> > > > > 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> > > > > 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> > > > > 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> > > > > 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> > > > > 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> > > > > 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> > > > > 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> > > > > 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> > > > > 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> > > > > 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> > > > > 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> > > > > 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> > > > > 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> > > > > 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> > > > > 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> > > > > 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> > > > > 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> > > > > 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> > > > > 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> > > > > 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> > > > > 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> > > > > 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> > > > > 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> > > > > 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> > > > > 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> > > > > 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> > > > > 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> > > > > 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> > > > > 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> > > > > 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> > > > > 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> > > > > 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> > > > > 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> > > > > 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> > > > > 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> > > > > 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> > > > > 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> > > > > 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> > > > > 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> > > > > 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> > > > > 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> > > > > 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> > > > > 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> > > > > 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> > > > > 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> > > > > 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> > > > > 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> > > > > 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> > > > > 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> > > > > 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> > > > > 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> > > > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> > > > >
> > > > > _______________________________________________
> > > > > lttng-dev mailing list
> > > > > lttng-dev@lists.lttng.org
> > > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > > >
> > > > --
> > > > Mathieu Desnoyers
> > > > Operating System Efficiency R&D Consultant
> > > > EfficiOS Inc.
> > > > http://www.efficios.com
> > > >
> > >
> > >
> > > _______________________________________________
> > > lttng-dev mailing list
> > > lttng-dev@lists.lttng.org
> > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant
> > EfficiOS Inc.
> > http://www.efficios.com
>
> --
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010195007.GG2495@linux.vnet.ibm.com>
@ 2012-10-10 20:49 ` Mathieu Desnoyers
2012-10-11 1:31 ` Lai Jiangshan
[not found] ` <50762155.3080804@cn.fujitsu.com>
2 siblings, 0 replies; 10+ messages in thread
From: Mathieu Desnoyers @ 2012-10-10 20:49 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: lttng-dev
* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> On Wed, Oct 10, 2012 at 01:53:04PM -0400, Mathieu Desnoyers wrote:
> > * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > > On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> > > > > * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> > > > > > test code:
> > > > > > ./tests/test_urcu_lfs 100 10 10
> > > > > >
> > > > > > bug produce rate > 60%
> > > > > >
> > > > > > {{{
> > > > > > I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> > > > > > But I just test it about 5 times
> > > > > > }}}
> > > > > >
> > > > > > 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> > > > > > RCU_MB (no time to test for other rcu type)
> > > > > > test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> > > > > >
> > > > > > I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> > > > > >
> > > > > > Sorry, I tried, but I failed to find out the root cause currently.
> > > > >
> > > > > I think I managed to narrow down the issue:
> > > > >
> > > > > 1) the master branch does not reproduce it, but commit
> > > > > 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> > > > > time.
> > > > >
> > > > > 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> > > > > current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> > > > > moving to wfcqueue.
> > > > >
> > > > > 3) the bug always arise, for me, at the end of the 10 seconds.
> > > > > However, it might be simply due to the fact that most of the memory
> > > > > get freed at the end of program execution.
> > > > >
> > > > > 4) I've been able to get a backtrace, and it looks like we have some
> > > > > call_rcu callback-invokation threads still working while
> > > > > call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> > > > > is nicely waiting for the next thread to stop, and during that time,
> > > > > two callback-invokation threads are invoking callbacks (and one of
> > > > > them triggers the segfault).
> > > >
> > > > Do any of the callbacks reference __thread variables from some other
> > > > thread? If so, those threads must refrain from exiting until after
> > > > such callbacks complete.
> > >
> > > The callback is a simple caa_container_of + free, usual stuff, nothing
> > > fancy.
> >
> > Here is the fix: the bug was in call rcu. It is not required for master,
> > because we fixed it while moving to wfcqueue.
> >
> > We were erroneously writing to the head field of the default
> > call_rcu_data rather than tail.
>
> Ouch!!! I have no idea why that would have passed my testing. :-(
Well, teardown is fairly unfrequent compared to call_rcu/cb execution.
>
> > I wonder if we should simply do a new release with call_rcu using
> > wfcqueue and tell people to upgrade, or if we should somehow create a
> > stable branch with this fix.
> >
> > Thoughts ?
>
> Under what conditions does this bug appear? It is necessary to not just
> use call_rcu(), but also to explicitly call call_rcu_data_free(), right?
The conditions for it to appear:
1) setup per-cpu callback-invokation threads,
2) use call_rcu
3) call call_rcu_data_free() while there are still some pending
callbacks that have not yet been executed by the callback-invokation
threads,
4) we then get corruption due to the "default" callback invokation
that walks through a corrupted queue.
My guess is that in all your test-cases, there were no callbacks left to
move to the default cb-invokation thread. Interestingly, this
test_urcu_lfs 100 10 10 test really stresses out reclaim.
>
> My guess is that a stable branch would be good -- there will be other
> bugs, after all. :-/
Yep, although we have a pretty good track record so far! And I must say
that this bug has been fixed in the master branch (with the move to
wfcqueue) before it's been discovered. ;-)
I will soon create stable-0.6 and stable-0.7 branches,
Thanks!
Mathieu
>
> Thanx, Paul
>
> > Thanks,
> >
> > Mathieu
> >
> > ---
> > diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
> > index 13b24ff..b205229 100644
> > --- a/urcu-call-rcu-impl.h
> > +++ b/urcu-call-rcu-impl.h
> > @@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
> > /* Create default call rcu data if need be */
> > (void) get_default_call_rcu_data();
> > cbs_endprev = (struct cds_wfq_node **)
> > - uatomic_xchg(&default_call_rcu_data, cbs_tail);
> > - *cbs_endprev = cbs;
> > + uatomic_xchg(&default_call_rcu_data->cbs.tail,
> > + cbs_tail);
> > + _CMM_STORE_SHARED(*cbs_endprev, cbs);
> > uatomic_add(&default_call_rcu_data->qlen,
> > uatomic_read(&crdp->qlen));
> > wake_call_rcu_thread(default_call_rcu_data);
> >
> >
> > >
> > > Thanks,
> > >
> > > Mathieu
> > >
> > > >
> > > > Thanx, Paul
> > > >
> > > > > So I expect that commit
> > > > >
> > > > > commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> > > > > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > > > Date: Tue Sep 25 10:50:49 2012 -0500
> > > > >
> > > > > call_rcu: use wfcqueue, eliminate false-sharing
> > > > >
> > > > > Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> > > > > on the queue head and tail.
> > > > >
> > > > > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > > >
> > > > > Could have managed to fix the issue, or change the timing enough that it
> > > > > does not reproduces. I'll continue investigating.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mathieu
> > > > >
> > > > >
> > > > > >
> > > > > > *** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
> > > > > > ======= Backtrace: =========
> > > > > > /lib64/libc.so.6[0x37ee676d63]
> > > > > > /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
> > > > > > /lib64/libpthread.so.0[0x37eda06ccb]
> > > > > > /lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
> > > > > > ======= Memory map: ========
> > > > > > 00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > > > 00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
> > > > > > 00606000-00616000 rw-p 00000000 00:00 0
> > > > > > 00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
> > > > > > 37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
> > > > > > 37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
> > > > > > 37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
> > > > > > 37ed820000-37ed821000 rw-p 00000000 00:00 0
> > > > > > 37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > > 37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > > 37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > > 37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
> > > > > > 37edc18000-37edc1c000 rw-p 00000000 00:00 0
> > > > > > 37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
> > > > > > 37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > > > 37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
> > > > > > 37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
> > > > > > 37ee996000-37ee99c000 rw-p 00000000 00:00 0
> > > > > > 37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > > 37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > > 37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
> > > > > > 7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
> > > > > > 7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
> > > > > > 7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
> > > > > > 7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
> > > > > > 7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
> > > > > > 7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
> > > > > > 7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
> > > > > > 7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
> > > > > > 7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
> > > > > > 7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
> > > > > > 7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
> > > > > > 7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
> > > > > > 7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
> > > > > > 7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
> > > > > > 7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
> > > > > > 7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
> > > > > > 7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
> > > > > > 7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
> > > > > > 7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
> > > > > > 7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
> > > > > > 7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
> > > > > > 7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
> > > > > > 7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
> > > > > > 7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
> > > > > > 7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
> > > > > > 7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
> > > > > > 7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
> > > > > > 7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
> > > > > > 7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
> > > > > > 7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
> > > > > > 7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
> > > > > > 7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
> > > > > > 7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
> > > > > > 7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
> > > > > > 7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
> > > > > > 7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
> > > > > > 7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
> > > > > > 7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
> > > > > > 7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
> > > > > > 7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
> > > > > > 7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
> > > > > > 7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
> > > > > > 7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
> > > > > > 7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
> > > > > > 7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
> > > > > > 7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
> > > > > > 7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
> > > > > > 7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
> > > > > > 7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
> > > > > > 7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
> > > > > > 7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
> > > > > > 7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
> > > > > > 7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
> > > > > > 7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
> > > > > > 7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
> > > > > > 7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
> > > > > > 7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
> > > > > > 7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
> > > > > > 7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
> > > > > > 7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
> > > > > > 7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
> > > > > > 7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
> > > > > > 7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
> > > > > > 7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
> > > > > > 7f20060c6000-7f2008000000 ---p 00000000 00:00 0
> > > > > > 7f200c000000-7f200c029000 rw-p 00000000 00:00 0
> > > > > > 7f200c029000-7f2010000000 ---p 00000000 00:00 0
> > > > > > 7f2014000000-7f2014029000 rw-p 00000000 00:00 0
> > > > > > 7f2014029000-7f2018000000 ---p 00000000 00:00 0
> > > > > > 7f201c000000-7f201c029000 rw-p 00000000 00:00 0
> > > > > > 7f201c029000-7f2020000000 ---p 00000000 00:00 0
> > > > > > 7f2024000000-7f2024029000 rw-p 00000000 00:00 0
> > > > > > 7f2024029000-7f2028000000 ---p 00000000 00:00 0
> > > > > > 7f202c000000-7f202c029000 rw-p 00000000 00:00 0
> > > > > > 7f202c029000-7f2030000000 ---p 00000000 00:00 0
> > > > > > 7f2034000000-7f2034029000 rw-p 00000000 00:00 0
> > > > > > 7f2034029000-7f2038000000 ---p 00000000 00:00 0
> > > > > > 7f203c000000-7f203c029000 rw-p 00000000 00:00 0
> > > > > > 7f203c029000-7f2040000000 ---p 00000000 00:00 0
> > > > > > 7f2044000000-7f2044029000 rw-p 00000000 00:00 0
> > > > > > 7f2044029000-7f2048000000 ---p 00000000 00:00 0
> > > > > > 7f204c000000-7f204c029000 rw-p 00000000 00:00 0
> > > > > > 7f204c029000-7f2050000000 ---p 00000000 00:00 0
> > > > > > 7f2054000000-7f2054029000 rw-p 00000000 00:00 0
> > > > > > 7f2054029000-7f2058000000 ---p 00000000 00:00 0
> > > > > > 7f205c000000-7f205c029000 rw-p 00000000 00:00 0
> > > > > > 7f205c029000-7f2060000000 ---p 00000000 00:00 0
> > > > > > 7f2064000000-7f2064029000 rw-p 00000000 00:00 0
> > > > > > 7f2064029000-7f2068000000 ---p 00000000 00:00 0
> > > > > > 7f206c000000-7f206c029000 rw-p 00000000 00:00 0
> > > > > > 7f206c029000-7f2070000000 ---p 00000000 00:00 0
> > > > > > 7f2074000000-7f2074029000 rw-p 00000000 00:00 0
> > > > > > 7f2074029000-7f2078000000 ---p 00000000 00:00 0
> > > > > > 7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
> > > > > > 7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
> > > > > > 7f2084000000-7f2084029000 rw-p 00000000 00:00 0
> > > > > > 7f2084029000-7f2088000000 ---p 00000000 00:00 0
> > > > > > 7f208c000000-7f208c029000 rw-p 00000000 00:00 0
> > > > > > 7f208c029000-7f2090000000 ---p 00000000 00:00 0
> > > > > > 7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
> > > > > > 7f20960c6000-7f2098000000 ---p 00000000 00:00 0
> > > > > > 7f209c000000-7f209c029000 rw-p 00000000 00:00 0
> > > > > > 7f209c029000-7f20a0000000 ---p 00000000 00:00 0
> > > > > > 7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
> > > > > > 7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
> > > > > > 7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
> > > > > > 7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
> > > > > > 7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
> > > > > > 7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
> > > > > > 7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
> > > > > > 7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
> > > > > > 7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
> > > > > > 7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
> > > > > > 7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
> > > > > > 7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
> > > > > > 7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
> > > > > > 7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
> > > > > > 7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
> > > > > > 7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
> > > > > > 7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
> > > > > > 7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
> > > > > > 7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
> > > > > > 7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
> > > > > > 7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
> > > > > > 7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
> > > > > > 7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
> > > > > > 7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
> > > > > > 7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
> > > > > > 7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
> > > > > > 7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
> > > > > > 7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
> > > > > > 7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
> > > > > > 7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
> > > > > > 7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
> > > > > > 7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
> > > > > > 7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
> > > > > > 7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > > 7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > > 7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
> > > > > > 7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
> > > > > > 7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > > 7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > > 7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
> > > > > > 7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
> > > > > > 7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
> > > > > > 7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
> > > > > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
> > > > > >
> > > > > > _______________________________________________
> > > > > > lttng-dev mailing list
> > > > > > lttng-dev@lists.lttng.org
> > > > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > > > >
> > > > > --
> > > > > Mathieu Desnoyers
> > > > > Operating System Efficiency R&D Consultant
> > > > > EfficiOS Inc.
> > > > > http://www.efficios.com
> > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > lttng-dev mailing list
> > > > lttng-dev@lists.lttng.org
> > > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > >
> > > --
> > > Mathieu Desnoyers
> > > Operating System Efficiency R&D Consultant
> > > EfficiOS Inc.
> > > http://www.efficios.com
> >
> > --
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant
> > EfficiOS Inc.
> > http://www.efficios.com
> >
>
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <20121010195007.GG2495@linux.vnet.ibm.com>
2012-10-10 20:49 ` Mathieu Desnoyers
@ 2012-10-11 1:31 ` Lai Jiangshan
[not found] ` <50762155.3080804@cn.fujitsu.com>
2 siblings, 0 replies; 10+ messages in thread
From: Lai Jiangshan @ 2012-10-11 1:31 UTC (permalink / raw)
To: paulmck; +Cc: lttng-dev, Mathieu Desnoyers
On 10/11/2012 03:50 AM, Paul E. McKenney wrote:
> On Wed, Oct 10, 2012 at 01:53:04PM -0400, Mathieu Desnoyers wrote:
>> * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
>>> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
>>>> On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
>>>>> * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
>>>>>> test code:
>>>>>> ./tests/test_urcu_lfs 100 10 10
>>>>>>
>>>>>> bug produce rate > 60%
>>>>>>
>>>>>> {{{
>>>>>> I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
>>>>>> But I just test it about 5 times
>>>>>> }}}
>>>>>>
>>>>>> 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
>>>>>> RCU_MB (no time to test for other rcu type)
>>>>>> test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
>>>>>>
>>>>>> I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
>>>>>>
>>>>>> Sorry, I tried, but I failed to find out the root cause currently.
>>>>>
>>>>> I think I managed to narrow down the issue:
>>>>>
>>>>> 1) the master branch does not reproduce it, but commit
>>>>> 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
>>>>> time.
>>>>>
>>>>> 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
>>>>> current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
>>>>> moving to wfcqueue.
>>>>>
>>>>> 3) the bug always arise, for me, at the end of the 10 seconds.
>>>>> However, it might be simply due to the fact that most of the memory
>>>>> get freed at the end of program execution.
>>>>>
>>>>> 4) I've been able to get a backtrace, and it looks like we have some
>>>>> call_rcu callback-invokation threads still working while
>>>>> call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
>>>>> is nicely waiting for the next thread to stop, and during that time,
>>>>> two callback-invokation threads are invoking callbacks (and one of
>>>>> them triggers the segfault).
>>>>
>>>> Do any of the callbacks reference __thread variables from some other
>>>> thread? If so, those threads must refrain from exiting until after
>>>> such callbacks complete.
>>>
>>> The callback is a simple caa_container_of + free, usual stuff, nothing
>>> fancy.
>>
>> Here is the fix: the bug was in call rcu. It is not required for master,
>> because we fixed it while moving to wfcqueue.
>>
>> We were erroneously writing to the head field of the default
>> call_rcu_data rather than tail.
>
> Ouch!!! I have no idea why that would have passed my testing. :-(
It's one of the reasons that I rewrite wfqueue and introduce delete_all()
(Mathieu uses splice instead) to replace open code of wfqueue in urcu-call-rcu-impl.h.
>
>> I wonder if we should simply do a new release with call_rcu using
>> wfcqueue and tell people to upgrade, or if we should somehow create a
>> stable branch with this fix.
>>
>> Thoughts ?
>
> Under what conditions does this bug appear? It is necessary to not just
> use call_rcu(), but also to explicitly call call_rcu_data_free(), right?
>
> My guess is that a stable branch would be good -- there will be other
> bugs, after all. :-/
>
> Thanx, Paul
>
>> Thanks,
>>
>> Mathieu
>>
>> ---
>> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
>> index 13b24ff..b205229 100644
>> --- a/urcu-call-rcu-impl.h
>> +++ b/urcu-call-rcu-impl.h
>> @@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
>> /* Create default call rcu data if need be */
>> (void) get_default_call_rcu_data();
>> cbs_endprev = (struct cds_wfq_node **)
>> - uatomic_xchg(&default_call_rcu_data, cbs_tail);
>> - *cbs_endprev = cbs;
>> + uatomic_xchg(&default_call_rcu_data->cbs.tail,
>> + cbs_tail);
>> + _CMM_STORE_SHARED(*cbs_endprev, cbs);
>> uatomic_add(&default_call_rcu_data->qlen,
>> uatomic_read(&crdp->qlen));
>> wake_call_rcu_thread(default_call_rcu_data);
>>
>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>>
>>>> Thanx, Paul
>>>>
>>>>> So I expect that commit
>>>>>
>>>>> commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
>>>>> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>>>> Date: Tue Sep 25 10:50:49 2012 -0500
>>>>>
>>>>> call_rcu: use wfcqueue, eliminate false-sharing
>>>>>
>>>>> Eliminate false-sharing between call_rcu (enqueuer) and worker threads
>>>>> on the queue head and tail.
>>>>>
I think the changelog of this commit is too short.
Thanks,
Lai
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <50762155.3080804@cn.fujitsu.com>
@ 2012-10-11 3:02 ` Paul E. McKenney
2012-10-11 15:27 ` Mathieu Desnoyers
1 sibling, 0 replies; 10+ messages in thread
From: Paul E. McKenney @ 2012-10-11 3:02 UTC (permalink / raw)
To: Lai Jiangshan; +Cc: lttng-dev, Mathieu Desnoyers
On Thu, Oct 11, 2012 at 09:31:01AM +0800, Lai Jiangshan wrote:
> On 10/11/2012 03:50 AM, Paul E. McKenney wrote:
> > On Wed, Oct 10, 2012 at 01:53:04PM -0400, Mathieu Desnoyers wrote:
> >> * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> >>> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> >>>> On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> >>>>> * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> >>>>>> test code:
> >>>>>> ./tests/test_urcu_lfs 100 10 10
> >>>>>>
> >>>>>> bug produce rate > 60%
> >>>>>>
> >>>>>> {{{
> >>>>>> I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> >>>>>> But I just test it about 5 times
> >>>>>> }}}
> >>>>>>
> >>>>>> 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> >>>>>> RCU_MB (no time to test for other rcu type)
> >>>>>> test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> >>>>>>
> >>>>>> I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> >>>>>>
> >>>>>> Sorry, I tried, but I failed to find out the root cause currently.
> >>>>>
> >>>>> I think I managed to narrow down the issue:
> >>>>>
> >>>>> 1) the master branch does not reproduce it, but commit
> >>>>> 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> >>>>> time.
> >>>>>
> >>>>> 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> >>>>> current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> >>>>> moving to wfcqueue.
> >>>>>
> >>>>> 3) the bug always arise, for me, at the end of the 10 seconds.
> >>>>> However, it might be simply due to the fact that most of the memory
> >>>>> get freed at the end of program execution.
> >>>>>
> >>>>> 4) I've been able to get a backtrace, and it looks like we have some
> >>>>> call_rcu callback-invokation threads still working while
> >>>>> call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> >>>>> is nicely waiting for the next thread to stop, and during that time,
> >>>>> two callback-invokation threads are invoking callbacks (and one of
> >>>>> them triggers the segfault).
> >>>>
> >>>> Do any of the callbacks reference __thread variables from some other
> >>>> thread? If so, those threads must refrain from exiting until after
> >>>> such callbacks complete.
> >>>
> >>> The callback is a simple caa_container_of + free, usual stuff, nothing
> >>> fancy.
> >>
> >> Here is the fix: the bug was in call rcu. It is not required for master,
> >> because we fixed it while moving to wfcqueue.
> >>
> >> We were erroneously writing to the head field of the default
> >> call_rcu_data rather than tail.
> >
> > Ouch!!! I have no idea why that would have passed my testing. :-(
>
> It's one of the reasons that I rewrite wfqueue and introduce delete_all()
> (Mathieu uses splice instead) to replace open code of wfqueue in urcu-call-rcu-impl.h.
Good catch!!!
Thanx, Paul
> >> I wonder if we should simply do a new release with call_rcu using
> >> wfcqueue and tell people to upgrade, or if we should somehow create a
> >> stable branch with this fix.
> >>
> >> Thoughts ?
> >
> > Under what conditions does this bug appear? It is necessary to not just
> > use call_rcu(), but also to explicitly call call_rcu_data_free(), right?
> >
> > My guess is that a stable branch would be good -- there will be other
> > bugs, after all. :-/
> >
> > Thanx, Paul
> >
> >> Thanks,
> >>
> >> Mathieu
> >>
> >> ---
> >> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
> >> index 13b24ff..b205229 100644
> >> --- a/urcu-call-rcu-impl.h
> >> +++ b/urcu-call-rcu-impl.h
> >> @@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
> >> /* Create default call rcu data if need be */
> >> (void) get_default_call_rcu_data();
> >> cbs_endprev = (struct cds_wfq_node **)
> >> - uatomic_xchg(&default_call_rcu_data, cbs_tail);
> >> - *cbs_endprev = cbs;
> >> + uatomic_xchg(&default_call_rcu_data->cbs.tail,
> >> + cbs_tail);
> >> + _CMM_STORE_SHARED(*cbs_endprev, cbs);
> >> uatomic_add(&default_call_rcu_data->qlen,
> >> uatomic_read(&crdp->qlen));
> >> wake_call_rcu_thread(default_call_rcu_data);
> >>
> >>
> >>>
> >>> Thanks,
> >>>
> >>> Mathieu
> >>>
> >>>>
> >>>> Thanx, Paul
> >>>>
> >>>>> So I expect that commit
> >>>>>
> >>>>> commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> >>>>> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> >>>>> Date: Tue Sep 25 10:50:49 2012 -0500
> >>>>>
> >>>>> call_rcu: use wfcqueue, eliminate false-sharing
> >>>>>
> >>>>> Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> >>>>> on the queue head and tail.
> >>>>>
>
> I think the changelog of this commit is too short.
>
> Thanks,
> Lai
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rculfstack bug
[not found] ` <50762155.3080804@cn.fujitsu.com>
2012-10-11 3:02 ` Paul E. McKenney
@ 2012-10-11 15:27 ` Mathieu Desnoyers
1 sibling, 0 replies; 10+ messages in thread
From: Mathieu Desnoyers @ 2012-10-11 15:27 UTC (permalink / raw)
To: Lai Jiangshan; +Cc: lttng-dev, paulmck
* Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> On 10/11/2012 03:50 AM, Paul E. McKenney wrote:
> > On Wed, Oct 10, 2012 at 01:53:04PM -0400, Mathieu Desnoyers wrote:
> >> * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> >>> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> >>>> On Wed, Oct 10, 2012 at 07:42:15AM -0400, Mathieu Desnoyers wrote:
> >>>>> * Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> >>>>>> test code:
> >>>>>> ./tests/test_urcu_lfs 100 10 10
> >>>>>>
> >>>>>> bug produce rate > 60%
> >>>>>>
> >>>>>> {{{
> >>>>>> I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
> >>>>>> But I just test it about 5 times
> >>>>>> }}}
> >>>>>>
> >>>>>> 4cores*1threads: Intel(R) Core(TM) i5 CPU 760
> >>>>>> RCU_MB (no time to test for other rcu type)
> >>>>>> test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
> >>>>>>
> >>>>>> I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
> >>>>>>
> >>>>>> Sorry, I tried, but I failed to find out the root cause currently.
> >>>>>
> >>>>> I think I managed to narrow down the issue:
> >>>>>
> >>>>> 1) the master branch does not reproduce it, but commit
> >>>>> 768fba83676f49eb73fd1d8ad452016a84c5ec2a repdroduces it about 50% of the
> >>>>> time.
> >>>>>
> >>>>> 2) the main change between 768fba83676f49eb73fd1d8ad452016a84c5ec2a and
> >>>>> current master (f94061a3df4c9eab9ac869a19e4228de54771fcb) is call_rcu
> >>>>> moving to wfcqueue.
> >>>>>
> >>>>> 3) the bug always arise, for me, at the end of the 10 seconds.
> >>>>> However, it might be simply due to the fact that most of the memory
> >>>>> get freed at the end of program execution.
> >>>>>
> >>>>> 4) I've been able to get a backtrace, and it looks like we have some
> >>>>> call_rcu callback-invokation threads still working while
> >>>>> call_rcu_data_free() is invoked. In the backtrace, call_rcu_data_free()
> >>>>> is nicely waiting for the next thread to stop, and during that time,
> >>>>> two callback-invokation threads are invoking callbacks (and one of
> >>>>> them triggers the segfault).
> >>>>
> >>>> Do any of the callbacks reference __thread variables from some other
> >>>> thread? If so, those threads must refrain from exiting until after
> >>>> such callbacks complete.
> >>>
> >>> The callback is a simple caa_container_of + free, usual stuff, nothing
> >>> fancy.
> >>
> >> Here is the fix: the bug was in call rcu. It is not required for master,
> >> because we fixed it while moving to wfcqueue.
> >>
> >> We were erroneously writing to the head field of the default
> >> call_rcu_data rather than tail.
> >
> > Ouch!!! I have no idea why that would have passed my testing. :-(
>
> It's one of the reasons that I rewrite wfqueue and introduce delete_all()
> (Mathieu uses splice instead) to replace open code of wfqueue in
> urcu-call-rcu-impl.h.
Yes, you did an excellent call on this one. We had the bug fixed in the
master branch before it was discovered, which is always nice :)
>
> >
> >> I wonder if we should simply do a new release with call_rcu using
> >> wfcqueue and tell people to upgrade, or if we should somehow create a
> >> stable branch with this fix.
> >>
> >> Thoughts ?
> >
> > Under what conditions does this bug appear? It is necessary to not just
> > use call_rcu(), but also to explicitly call call_rcu_data_free(), right?
> >
> > My guess is that a stable branch would be good -- there will be other
> > bugs, after all. :-/
> >
> > Thanx, Paul
> >
> >> Thanks,
> >>
> >> Mathieu
> >>
> >> ---
> >> diff --git a/urcu-call-rcu-impl.h b/urcu-call-rcu-impl.h
> >> index 13b24ff..b205229 100644
> >> --- a/urcu-call-rcu-impl.h
> >> +++ b/urcu-call-rcu-impl.h
> >> @@ -647,8 +647,9 @@ void call_rcu_data_free(struct call_rcu_data *crdp)
> >> /* Create default call rcu data if need be */
> >> (void) get_default_call_rcu_data();
> >> cbs_endprev = (struct cds_wfq_node **)
> >> - uatomic_xchg(&default_call_rcu_data, cbs_tail);
> >> - *cbs_endprev = cbs;
> >> + uatomic_xchg(&default_call_rcu_data->cbs.tail,
> >> + cbs_tail);
> >> + _CMM_STORE_SHARED(*cbs_endprev, cbs);
> >> uatomic_add(&default_call_rcu_data->qlen,
> >> uatomic_read(&crdp->qlen));
> >> wake_call_rcu_thread(default_call_rcu_data);
> >>
> >>
> >>>
> >>> Thanks,
> >>>
> >>> Mathieu
> >>>
> >>>>
> >>>> Thanx, Paul
> >>>>
> >>>>> So I expect that commit
> >>>>>
> >>>>> commit 5161f31e09ce33dd79afad8d08a2372fbf1c4fbe
> >>>>> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> >>>>> Date: Tue Sep 25 10:50:49 2012 -0500
> >>>>>
> >>>>> call_rcu: use wfcqueue, eliminate false-sharing
> >>>>>
> >>>>> Eliminate false-sharing between call_rcu (enqueuer) and worker threads
> >>>>> on the queue head and tail.
> >>>>>
>
> I think the changelog of this commit is too short.
Yes, unfortunately, it's already in the master branch, and rewriting
history is something mere mortals are not allowed to do. ;-)
There are a couple of patches that I have pending that did not receive
any negative feedback at this point. I will move things forwards
cautiously to ensure we have a consistent master branch, and create
stable branches for 0.6 and 0.7. Review of these commits will be welcome
before we do a 0.8 release (with the new wfcqueue). We'll be able to
proceed to further changes as separate commits.
Thanks!
Mathieu
>
> Thanks,
> Lai
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* rculfstack bug
@ 2012-10-10 10:13 Lai Jiangshan
0 siblings, 0 replies; 10+ messages in thread
From: Lai Jiangshan @ 2012-10-10 10:13 UTC (permalink / raw)
To: Mathieu Desnoyers, Paul E. McKenney, lttng-dev
test code:
./tests/test_urcu_lfs 100 10 10
bug produce rate > 60%
{{{
I didn't see any bug when "./tests/test_urcu_lfs 10 10 10" Or "./tests/test_urcu_lfs 100 100 10"
But I just test it about 5 times
}}}
4cores*1threads: Intel(R) Core(TM) i5 CPU 760
RCU_MB (no time to test for other rcu type)
test commit: 768fba83676f49eb73fd1d8ad452016a84c5ec2a
I didn't see any bug when "./tests/test_urcu_mb 10 100 10"
Sorry, I tried, but I failed to find out the root cause currently.
*** glibc detected *** /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs: double free or corruption (out): 0x00007f20955dfbb0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x37ee676d63]
/home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs[0x4024f5]
/lib64/libpthread.so.0[0x37eda06ccb]
/lib64/libc.so.6(clone+0x6d)[0x37ee6e0c2d]
======= Memory map: ========
00400000-00405000 r-xp 00000000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
00605000-00606000 rw-p 00005000 08:08 6031723 /home/laijs/work/userspace-rcu/tests/.libs/lt-test_urcu_lfs
00606000-00616000 rw-p 00000000 00:00 0
00e9c000-03482000 rw-p 00000000 00:00 0 [heap]
37ed600000-37ed61f000 r-xp 00000000 08:01 1507421 /lib64/ld-2.13.so
37ed81e000-37ed81f000 r--p 0001e000 08:01 1507421 /lib64/ld-2.13.so
37ed81f000-37ed820000 rw-p 0001f000 08:01 1507421 /lib64/ld-2.13.so
37ed820000-37ed821000 rw-p 00000000 00:00 0
37eda00000-37eda17000 r-xp 00000000 08:01 1507427 /lib64/libpthread-2.13.so
37eda17000-37edc16000 ---p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
37edc16000-37edc17000 r--p 00016000 08:01 1507427 /lib64/libpthread-2.13.so
37edc17000-37edc18000 rw-p 00017000 08:01 1507427 /lib64/libpthread-2.13.so
37edc18000-37edc1c000 rw-p 00000000 00:00 0
37ee600000-37ee791000 r-xp 00000000 08:01 1507423 /lib64/libc-2.13.so
37ee791000-37ee991000 ---p 00191000 08:01 1507423 /lib64/libc-2.13.so
37ee991000-37ee995000 r--p 00191000 08:01 1507423 /lib64/libc-2.13.so
37ee995000-37ee996000 rw-p 00195000 08:01 1507423 /lib64/libc-2.13.so
37ee996000-37ee99c000 rw-p 00000000 00:00 0
37f0e00000-37f0e15000 r-xp 00000000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
37f0e15000-37f1014000 ---p 00015000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
37f1014000-37f1015000 rw-p 00014000 08:01 1507437 /lib64/libgcc_s-4.5.1-20100924.so.1
7f1ee4000000-7f1ee4029000 rw-p 00000000 00:00 0
7f1ee4029000-7f1ee8000000 ---p 00000000 00:00 0
7f1eec000000-7f1eee039000 rw-p 00000000 00:00 0
7f1eee039000-7f1ef0000000 ---p 00000000 00:00 0
7f1ef4000000-7f1ef4029000 rw-p 00000000 00:00 0
7f1ef4029000-7f1ef8000000 ---p 00000000 00:00 0
7f1efc000000-7f1efc029000 rw-p 00000000 00:00 0
7f1efc029000-7f1f00000000 ---p 00000000 00:00 0
7f1f04000000-7f1f060b8000 rw-p 00000000 00:00 0
7f1f060b8000-7f1f08000000 ---p 00000000 00:00 0
7f1f0c000000-7f1f0c029000 rw-p 00000000 00:00 0
7f1f0c029000-7f1f10000000 ---p 00000000 00:00 0
7f1f14000000-7f1f14029000 rw-p 00000000 00:00 0
7f1f14029000-7f1f18000000 ---p 00000000 00:00 0
7f1f1c000000-7f1f1c029000 rw-p 00000000 00:00 0
7f1f1c029000-7f1f20000000 ---p 00000000 00:00 0
7f1f24000000-7f1f24029000 rw-p 00000000 00:00 0
7f1f24029000-7f1f28000000 ---p 00000000 00:00 0
7f1f2c000000-7f1f2c029000 rw-p 00000000 00:00 0
7f1f2c029000-7f1f30000000 ---p 00000000 00:00 0
7f1f34000000-7f1f34029000 rw-p 00000000 00:00 0
7f1f34029000-7f1f38000000 ---p 00000000 00:00 0
7f1f3c000000-7f1f3c029000 rw-p 00000000 00:00 0
7f1f3c029000-7f1f40000000 ---p 00000000 00:00 0
7f1f44000000-7f1f44029000 rw-p 00000000 00:00 0
7f1f44029000-7f1f48000000 ---p 00000000 00:00 0
7f1f4c000000-7f1f4c029000 rw-p 00000000 00:00 0
7f1f4c029000-7f1f50000000 ---p 00000000 00:00 0
7f1f54000000-7f1f54029000 rw-p 00000000 00:00 0
7f1f54029000-7f1f58000000 ---p 00000000 00:00 0
7f1f5c000000-7f1f5c029000 rw-p 00000000 00:00 0
7f1f5c029000-7f1f60000000 ---p 00000000 00:00 0
7f1f64000000-7f1f64029000 rw-p 00000000 00:00 0
7f1f64029000-7f1f68000000 ---p 00000000 00:00 0
7f1f6c000000-7f1f6c029000 rw-p 00000000 00:00 0
7f1f6c029000-7f1f70000000 ---p 00000000 00:00 0
7f1f74000000-7f1f74029000 rw-p 00000000 00:00 0
7f1f74029000-7f1f78000000 ---p 00000000 00:00 0
7f1f7c000000-7f1f7c029000 rw-p 00000000 00:00 0
7f1f7c029000-7f1f80000000 ---p 00000000 00:00 0
7f1f84000000-7f1f84029000 rw-p 00000000 00:00 0
7f1f84029000-7f1f88000000 ---p 00000000 00:00 0
7f1f8c000000-7f1f8c029000 rw-p 00000000 00:00 0
7f1f8c029000-7f1f90000000 ---p 00000000 00:00 0
7f1f94000000-7f1f94029000 rw-p 00000000 00:00 0
7f1f94029000-7f1f98000000 ---p 00000000 00:00 0
7f1f9c000000-7f1f9c029000 rw-p 00000000 00:00 0
7f1f9c029000-7f1fa0000000 ---p 00000000 00:00 0
7f1fa4000000-7f1fa60ac000 rw-p 00000000 00:00 0
7f1fa60ac000-7f1fa8000000 ---p 00000000 00:00 0
7f1fac000000-7f1fac029000 rw-p 00000000 00:00 0
7f1fac029000-7f1fb0000000 ---p 00000000 00:00 0
7f1fb4000000-7f1fb4029000 rw-p 00000000 00:00 0
7f1fb4029000-7f1fb8000000 ---p 00000000 00:00 0
7f1fbc000000-7f1fbc029000 rw-p 00000000 00:00 0
7f1fbc029000-7f1fc0000000 ---p 00000000 00:00 0
7f1fc4000000-7f1fc4029000 rw-p 00000000 00:00 0
7f1fc4029000-7f1fc8000000 ---p 00000000 00:00 0
7f1fcc000000-7f1fce0a1000 rw-p 00000000 00:00 0
7f1fce0a1000-7f1fd0000000 ---p 00000000 00:00 0
7f1fd4000000-7f1fd4029000 rw-p 00000000 00:00 0
7f1fd4029000-7f1fd8000000 ---p 00000000 00:00 0
7f1fdc000000-7f1fde06b000 rw-p 00000000 00:00 0
7f1fde06b000-7f1fe0000000 ---p 00000000 00:00 0
7f1fe4000000-7f1fe4029000 rw-p 00000000 00:00 0
7f1fe4029000-7f1fe8000000 ---p 00000000 00:00 0
7f1fec000000-7f1fede38000 rw-p 00000000 00:00 0
7f1fede38000-7f1ff0000000 ---p 00000000 00:00 0
7f1ff4000000-7f1ff4029000 rw-p 00000000 00:00 0
7f1ff4029000-7f1ff8000000 ---p 00000000 00:00 0
7f1ffc000000-7f1ffc029000 rw-p 00000000 00:00 0
7f1ffc029000-7f2000000000 ---p 00000000 00:00 0
7f2004000000-7f20060c6000 rw-p 00000000 00:00 0
7f20060c6000-7f2008000000 ---p 00000000 00:00 0
7f200c000000-7f200c029000 rw-p 00000000 00:00 0
7f200c029000-7f2010000000 ---p 00000000 00:00 0
7f2014000000-7f2014029000 rw-p 00000000 00:00 0
7f2014029000-7f2018000000 ---p 00000000 00:00 0
7f201c000000-7f201c029000 rw-p 00000000 00:00 0
7f201c029000-7f2020000000 ---p 00000000 00:00 0
7f2024000000-7f2024029000 rw-p 00000000 00:00 0
7f2024029000-7f2028000000 ---p 00000000 00:00 0
7f202c000000-7f202c029000 rw-p 00000000 00:00 0
7f202c029000-7f2030000000 ---p 00000000 00:00 0
7f2034000000-7f2034029000 rw-p 00000000 00:00 0
7f2034029000-7f2038000000 ---p 00000000 00:00 0
7f203c000000-7f203c029000 rw-p 00000000 00:00 0
7f203c029000-7f2040000000 ---p 00000000 00:00 0
7f2044000000-7f2044029000 rw-p 00000000 00:00 0
7f2044029000-7f2048000000 ---p 00000000 00:00 0
7f204c000000-7f204c029000 rw-p 00000000 00:00 0
7f204c029000-7f2050000000 ---p 00000000 00:00 0
7f2054000000-7f2054029000 rw-p 00000000 00:00 0
7f2054029000-7f2058000000 ---p 00000000 00:00 0
7f205c000000-7f205c029000 rw-p 00000000 00:00 0
7f205c029000-7f2060000000 ---p 00000000 00:00 0
7f2064000000-7f2064029000 rw-p 00000000 00:00 0
7f2064029000-7f2068000000 ---p 00000000 00:00 0
7f206c000000-7f206c029000 rw-p 00000000 00:00 0
7f206c029000-7f2070000000 ---p 00000000 00:00 0
7f2074000000-7f2074029000 rw-p 00000000 00:00 0
7f2074029000-7f2078000000 ---p 00000000 00:00 0
7f207c000000-7f207e0bc000 rw-p 00000000 00:00 0
7f207e0bc000-7f2080000000 ---p 00000000 00:00 0
7f2084000000-7f2084029000 rw-p 00000000 00:00 0
7f2084029000-7f2088000000 ---p 00000000 00:00 0
7f208c000000-7f208c029000 rw-p 00000000 00:00 0
7f208c029000-7f2090000000 ---p 00000000 00:00 0
7f2094000000-7f20960c6000 rw-p 00000000 00:00 0
7f20960c6000-7f2098000000 ---p 00000000 00:00 0
7f209c000000-7f209c029000 rw-p 00000000 00:00 0
7f209c029000-7f20a0000000 ---p 00000000 00:00 0
7f20a4000000-7f20a4029000 rw-p 00000000 00:00 0
7f20a4029000-7f20a8000000 ---p 00000000 00:00 0
7f20ac000000-7f20ac029000 rw-p 00000000 00:00 0
7f20ac029000-7f20b0000000 ---p 00000000 00:00 0
7f20b4000000-7f20b4029000 rw-p 00000000 00:00 0
7f20b4029000-7f20b8000000 ---p 00000000 00:00 0
7f20bc000000-7f20bc029000 rw-p 00000000 00:00 0
7f20bc029000-7f20c0000000 ---p 00000000 00:00 0
7f20c4000000-7f20c4029000 rw-p 00000000 00:00 0
7f20c4029000-7f20c8000000 ---p 00000000 00:00 0
7f20c8ffa000-7f20c8ffb000 ---p 00000000 00:00 0
7f20c8ffb000-7f20c97fb000 rw-p 00000000 00:00 0 [stack:10274]
7f20c97fb000-7f20c97fc000 ---p 00000000 00:00 0
7f20c97fc000-7f20c9ffc000 rw-p 00000000 00:00 0
7f20c9ffc000-7f20c9ffd000 ---p 00000000 00:00 0
7f20c9ffd000-7f20ca7fd000 rw-p 00000000 00:00 0
7f20ca7fd000-7f20ca7fe000 ---p 00000000 00:00 0
7f20ca7fe000-7f20caffe000 rw-p 00000000 00:00 0
7f20cc000000-7f20cc029000 rw-p 00000000 00:00 0
7f20cc029000-7f20d0000000 ---p 00000000 00:00 0
7f20d4000000-7f20d4029000 rw-p 00000000 00:00 0
7f20d4029000-7f20d8000000 ---p 00000000 00:00 0
7f20dc000000-7f20dc029000 rw-p 00000000 00:00 0
7f20dc029000-7f20e0000000 ---p 00000000 00:00 0
7f210d9dd000-7f210d9de000 ---p 00000000 00:00 0
7f210d9de000-7f210e1de000 rw-p 00000000 00:00 0 [stack:10160]
7f210e1de000-7f210e1df000 ---p 00000000 00:00 0
7f210e1df000-7f210e9df000 rw-p 00000000 00:00 0 [stack:10159]
7f210e9df000-7f210e9e0000 ---p 00000000 00:00 0
7f210e9e0000-7f210f1e0000 rw-p 00000000 00:00 0
7f210f1e0000-7f210f1e1000 ---p 00000000 00:00 0
7f210f1e1000-7f210f9e4000 rw-p 00000000 00:00 0
7f210fa00000-7f210fa01000 rw-p 00000000 00:00 0
7f210fa01000-7f210fa02000 r-xp 00000000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
7f210fa02000-7f210fc02000 ---p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
7f210fc02000-7f210fc03000 rw-p 00001000 08:08 6029369 /home/laijs/work/userspace-rcu/.libs/liburcu-common.so.1.0.0
7f210fc03000-7f210fc04000 rw-p 00000000 00:00 0
7f210fc04000-7f210fc0a000 r-xp 00000000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
7f210fc0a000-7f210fe09000 ---p 00006000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
7f210fe09000-7f210fe0a000 rw-p 00005000 08:08 6029586 /home/laijs/work/userspace-rcu/.libs/liburcu-cds.so.1.0.0
7f210fe0a000-7f210fe0b000 rw-p 00000000 00:00 0
7fff7c648000-7fff7c669000 rw-p 00000000 00:00 0 [stack]
7fff7c715000-7fff7c716000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2012-10-11 15:27 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <50754A57.1080104@cn.fujitsu.com>
2012-10-10 11:42 ` rculfstack bug Mathieu Desnoyers
[not found] ` <20121010114215.GA11307@Krystal>
2012-10-10 15:02 ` Paul E. McKenney
[not found] ` <20121010150207.GB2495@linux.vnet.ibm.com>
2012-10-10 15:07 ` Mathieu Desnoyers
[not found] ` <20121010150751.GB20761@Krystal>
2012-10-10 17:53 ` Mathieu Desnoyers
[not found] ` <20121010175304.GA25511@Krystal>
2012-10-10 19:50 ` Paul E. McKenney
[not found] ` <20121010195007.GG2495@linux.vnet.ibm.com>
2012-10-10 20:49 ` Mathieu Desnoyers
2012-10-11 1:31 ` Lai Jiangshan
[not found] ` <50762155.3080804@cn.fujitsu.com>
2012-10-11 3:02 ` Paul E. McKenney
2012-10-11 15:27 ` Mathieu Desnoyers
2012-10-10 10:13 Lai Jiangshan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.